A target detection method under unstable visible-infrared image quality

Abstract

The present invention relates to a target detection method under the condition of unstable visible light-infrared image quality. The present invention first reconstructs the unstable target information of visible light and infrared images to obtain an incremental information image; fuses the incremental information image with the original visible light and infrared images to obtain visible light and infrared images containing incremental information; inputs the visible light and infrared images into a feature extraction network to extract global features of the visible light and infrared images; inputs the global features into a kernel similarity measurement module to obtain similarity weights of a balanced optimization network; multiplies the similarity weights with the global features to obtain features of balanced optimization; inputs the features into a feature fusion module to obtain a fused feature map; and inputs the feature map into a detector for target detection. The present invention can effectively mine the incremental information of visible light and infrared images, and enhance the detection robustness of target detectors for visible light and infrared images with unstable information quality.

Description

Technical Field

[0001] The invention relates to a target detection method under unstable visible light-infrared image quality, and belongs to the technical field of image processing based on deep learning. Background Art

[0002] Using visible light and infrared images as multimodal data to train target detection networks is an effective way to enhance the robustness of single-modal target detection algorithms in complex environments. However, affected by factors such as the data acquisition environment and sensor characteristics, the quality of visible light and infrared image information is not stable, and the model tends to optimize a certain modality information. In response to this problem, relevant researchers at home and abroad have conducted multi-faceted research, mainly including the following three technical routes: based on data enhancement methods, based on generative model methods, and based on multimodal joint representation learning methods.

[0003] Based on data augmentation method: This method mainly randomly removes a certain modality input during the training process to simulate the phenomenon that a certain modality data is missing in the multimodal dataset, thereby improving the recognition performance of the model under missing modality conditions. Based on generative model method: This method mainly directly predicts and reconstructs the missing modality information based on the input modality information. In 2017, Tran et al. proposed the Cascaded Residual Autoencoder (CRA). By using the residual connection mechanism on the autoencoder structure, damaged data can be obtained and restored through the estimation function; in 2021, Zhao et al. constructed a Missing Modality Imagination Network (MMIN) based on CRA and cycle consistency learning strategy, which can "reconstruct" missing modality features based on the given available modality information representation; in 2021, Ma et al. constructed a feature reconstruction network based on Bayesian meta-learning, which can reconstruct missing modality features based on available modality information. Similar methods include: Conditional Variational Autoencoder (CVAE), Conditional Multimodal Autoencoder (CMAE) and Multimodal Variational Autoencoder (MVAE). Based on multimodal joint representation learning method: This method mainly represents multimodal data by mining and fusing multimodal complementary information. In 2019, Han et al. proposed a joint training model consisting of two mode-specific encoders and a shared classifier, which uses audio and visual information as a joint representation to improve the performance of unimodal sentiment analysis; in the same year, Pham et al. proposed a model based on sequential translation to learn the joint representation between the source modality and multiple target modalities, using the hidden vector in the source modality encoder as a joint representation to improve the sentiment recognition performance of the source modality. In response to the modal collapse problem, in 2022, Javaloy et al. analyzed the cause of it and believed that it was caused by gradient conflicts in some parameters during the model optimization process. In order to solve this problem, they proposed a multimodal VAE training strategy. By utilizing the existing multi-task learning solution, the network has a fair optimization process that is not biased towards a certain modality. Experimental results show that this method can more accurately fit the marginal, joint and conditional distributions of all modal data.

[0004] In current related research, only single modal data is used, or the data of each modal is simply added together to assist the model in reconstructing the uncertain information of the modalities, resulting in poor information quality obtained by the model reconstruction and the inability to introduce effective incremental information; secondly, there is a lack of a method to guide the model to optimize the information of each modality in a simple and intuitive way, and the model cannot complete the task of equally optimizing the information of each modality. Therefore, how to design a visible light and infrared image incremental information mining and generation model from the aspects of model network structure and loss optimization, and alleviate the unstable quality of visible light and infrared image data and the modal collapse of the current model has become an urgent problem to be solved in the current cross-modal detection algorithm. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a target detection method under unstable visible light-infrared image quality, which is used to reconstruct the unstable information of visible light and infrared images, guide the model fairness to optimize the visible light and infrared image information, and ultimately alleviate the impact of the unstable information of visible light and infrared images and modal collapse on the positive fairness optimization of the model.

[0006] The technical solution of the present invention is: a target detection method under unstable visible light-infrared image quality, the specific steps are:

[0007] Step 1: Input the visible light and infrared images into the diffusion model, reconstruct the unstable target information of the visible light and infrared images, and obtain the incremental information image;

[0008] Step 2: Fuse the incremental information image obtained in Step 1 with the original visible light and infrared images to obtain visible light and infrared images containing incremental information;

[0009] Step 3: Input the visible light and infrared images obtained in Step 2 into the feature extraction network to extract the global features of the visible light and infrared images;

[0010] Step 4: Input the global features obtained in Step 3 into the kernel similarity measurement module to obtain the similarity weight of the balanced optimization network;

[0011] Step 5: Multiply the similarity weight obtained in Step 4 by the global feature obtained in Step 3 to obtain the feature of balance optimization;

[0012] Step 6: Input the features obtained in Step 5 into the feature fusion module to fuse the complementary information of visible light and infrared image features to obtain a fused feature map;

[0013] Step7: Input the feature map obtained in Step6 into the detector for target detection.

[0014] Furthermore, the method of reconstructing the unstable target information of the visible light and infrared images described in Step 1 is to first fuse the visible light and infrared images, and then convert the fused image into a normal distribution, specifically:

[0015]

[0016] Where rgb_images and thermal_images represent visible light and infrared images respectively, Φ(·) represents the operation of converting data into normal distribution. represents the sampling operation of the quasi-diffusion process, I c Represents the obtained incremental information image.

[0017] Furthermore, the fusion method described in Step 2 is to add the incremental information image to the original visible light and infrared images pixel by pixel, specifically:

[0018]

[0019]

[0020] In the formula, Represents the element-by-element addition operation, I r Represents a visible light image containing incremental information, I t Represents an infrared image containing incremental information.

[0021] Furthermore, the feature extraction network described in Step 3 is a backbone network based on the SSD algorithm. The backbone network of the SSD algorithm is copied into two branches, one branch extracts visible light image features, and the other branch extracts infrared image features, specifically:

[0022] F r =SSD(I r ) (4)

[0023] F t =SSD(I t ) (5)

[0024] In the formula, SSD represents the backbone network of the SSD algorithm, and F r and F t Represent the extracted visible light and infrared image feature maps respectively.

[0025] Furthermore, the kernel similarity measurement module described in Step 4 designs a convolutional layer and a fully connected layer to map the visible light and infrared image features into two low-dimensional feature vectors, and then uses a Gaussian kernel function to calculate the similarity of the two low-dimensional feature vectors based on the two low-dimensional feature vectors to obtain the similarity weight, which is specifically:

[0026]

[0027] In the formula, and They represent the low-dimensional vectors of visible light and infrared image features respectively, σ represents the parameters of the Gaussian kernel function, and W represents the similarity weight.

[0028] Furthermore, the multiplication operation described in Step 5 is to multiply the visible light and infrared image features by the similarity weights in a global operation, specifically:

[0029]

[0030]

[0031] In the formula, Represents a global multiplication operation on the feature map, F′ r and F′ t Represent the visible light and infrared image features with balanced optimization, respectively.

[0032] Furthermore, the feature fusion module in Step 6 adds a channel attention mechanism, specifically:

[0033]

[0034]

[0035] Where CAM(·) represents the channel attention mechanism, and Represents the fused visible light and infrared image features.

[0036] The calculation formula of the channel attention mechanism is as follows:

[0037]

[0038]

[0039] Where H and W represent the height and width of the feature map, respectively, and σ represents the hyperbolic tangent activation function.

[0040] Furthermore, the Step 7 is specifically as follows:

[0041] cls,conf,box=d_m(F′ r_cam ,F′ t_cam ) (13)

[0042] Where d_m(·) represents the detector, cls, conf, box represent the category, confidence and coordinates of the bounding box of the target in the predicted image respectively.

[0043] The target detection refers to detecting the target in the image according to the specific task requirements. The predicted result is the bounding box corresponding to the target in the input image. The output results include classification and regression.

[0044] The beneficial effects of the present invention are as follows: the present invention utilizes image processing technology based on deep learning, takes SSD as the backbone network, introduces diffusion model and kernel similarity measurement module, mines the intrinsic information of visible light and infrared images, guides the modeling of unstable information of visible light and infrared images; and measures the similarity of high-dimensional semantic information of visible light and infrared image features with multi-kernel function, guides the model to optimize the information of visible light and infrared images in a fair manner. Finally, the influence of unstable information and modal collapse of visible light and infrared images on the forward fairness optimization of the model is alleviated. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of the present invention;

[0046] Figure 2 It is a network framework diagram of the present invention;

[0047] Figure 3 is the kernel similarity measurement module of the present invention;

[0048] Figure 4 is the feature fusion module of the present invention;

[0049] Figure 5 2 is an example diagram of the detection results of the present invention. DETAILED DESCRIPTION

[0050] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] Embodiment 1: The embodiment of the present invention adopts an improved SSD algorithm, and trains the network by combining visible light and infrared images, so as to achieve the goal that the algorithm still has the ability to detect targets with high precision when the quality of visible light and infrared images is unstable.

[0052] In order to effectively reconstruct the unstable information of visible light and infrared images and make the model balance and optimize the information of each modality, the embodiment of the present invention adopts the SSD algorithm as the backbone network and makes improvements to the network. Based on the SSD algorithm backbone network, a diffusion model is added to reconstruct the unstable information of visible light and infrared images, and the incremental information of visible light and infrared images is mined. In addition, a kernel similarity measurement module is added, and a kernel function is used to measure the similarity of visible light and infrared images, and each modality information is balanced and optimized based on the degree of similarity. Inspired by the attention mechanism, channel attention is added to the feature fusion module to effectively fuse important feature information between different modalities. The detection flow chart is shown below. Figure 1 As shown, the algorithm network framework diagram is as follows Figure 2 The algorithm network is built and trained based on the Tensorflow framework. The training data set uses the KUSTDroneFail data set. The Adam optimization method is used during the training process. The initial learning rate is 0.0001, and the learning rate is reduced by 10 every 6 rounds. -1 , the training duration is 30 rounds.

[0053] Step 1: Input the visible light and infrared images into the diffusion model, reconstruct the unstable target information of the visible light and infrared images, and obtain the incremental information image; the method of reconstructing the unstable information of the visible light and infrared images is to first fuse the visible light and infrared images, and then convert the fused image into a normal distribution, specifically:

[0054]

[0055] Where rgb_images and thermal_images represent visible light and infrared images, respectively, with a size of 512*640*3, and Φ(·) represents the operation of converting data into a normal distribution. represents the sampling operation of the quasi-diffusion process, I c Represents the obtained incremental information image.

[0056] Step 2: Fusing the incremental information image obtained in Step 1 with the original visible light and infrared images to obtain visible light and infrared images containing incremental information; the fusion method is to add the incremental information image to the original visible light and infrared images pixel by pixel, specifically:

[0057]

[0058]

[0059] In the formula, Represents the element-by-element addition operation, I r Represents a visible light image containing incremental information, I t Represents the increase

[0060] The infrared image with quantitative information has a size of 512*640*3.

[0061] Step 3: Input the visible light and infrared images obtained in Step 2 into the feature extraction network to extract the global features of the visible light and infrared images; the feature extraction network is a backbone network based on the SSD algorithm, and the backbone network of the SSD algorithm is copied into two branches, one branch extracts the visible light image features, and the other branch extracts the infrared image features, specifically:

[0062] F r =SSD(I r ) (4)

[0063] F t =SSD(I t ) (5)

[0064] In the formula, SSD represents the backbone network of the SSD algorithm, and F r and F t They represent the extracted visible light and infrared image feature maps, respectively, and their sizes are 128*160*256.

[0065] Step 4: Input the global features obtained in Step 3 into the kernel similarity measurement module to obtain the similarity weight of the balanced optimization network; the kernel similarity measurement module is to design convolutional layers and fully connected layers, map the visible light and infrared image features into two low-dimensional feature vectors, and then use the Gaussian kernel function to calculate the similarity of the two low-dimensional feature vectors based on the two low-dimensional feature vectors to obtain the similarity weight. The kernel similarity measurement module is as follows: Figure 3 As shown in the figure, the specific calculation method of similarity weight is:

[0066]

[0067] In the formula, and They represent the low-dimensional vectors of visible light and infrared image features respectively, σ represents the parameters of the Gaussian kernel function, and W represents the similarity weight, W∈(0,1).

[0068] Step 5: Multiply the similarity weight obtained in Step 4 by the global feature obtained in Step 3 to obtain a balanced optimized feature; the multiplication operation is to multiply the visible light and infrared image features by the similarity weight in the global operation, specifically:

[0069]

[0070]

[0071] In the formula, Represents a global multiplication operation on the feature map, F′ r and F′ t They represent the visible light and infrared image features with optimized balance, and their sizes are 128*160*256.

[0072] Step 6: Input the features obtained in Step 5 into the feature fusion module to fuse the complementary information of the visible light and infrared image features to obtain a fused feature map; the feature fusion module adds a channel attention mechanism, first performing an element-by-element subtraction operation on the visible light and infrared image features. The features obtained after the subtraction represent the difference features between the visible light and infrared image features, and the difference features are regarded as complementary information. The difference features are then input into the channel attention mechanism to obtain the channel weights. Finally, the channel weights are multiplied by the corresponding visible light and infrared image features to amplify the difference features, and are added to the features of the other modality to obtain the final fused features. Therefore, the features obtained after fusion contain the difference features from the other modality, thereby achieving the purpose of fusing complementary information. The feature fusion module is as follows: Figure 4 As shown, the specific calculation formula is as follows:

[0073]

[0074]

[0075] In the formula, CAM(·) represents the channel attention mechanism, F′ r_cam and F′ t_cam Represents the fused visible light and infrared image features.

[0076] The calculation formula of the channel attention mechanism is as follows:

[0077]

[0078]

[0079] Where H and W represent the height and width of the feature map, respectively, and σ represents the hyperbolic tangent activation function.

[0080] Step 7: Input the feature map obtained in step 6 into the detector for target detection, specifically:

[0081]

[0082] Where d_m(·) represents the detector, cls, conf, box represent the category, confidence and coordinates of the bounding box of the target in the predicted image respectively.

[0083] Finally, the loss function of the network is divided into two parts: the class loss Lcls and regression box loss L reg , where the regression box loss L reg Use smoothL1loss, category loss L cls Use Focal loss. The final objective function is:

[0084] L=L cls +L reg (14)

[0085] The convergence state is reached by optimizing L. Finally, the trained model is used to detect targets in visible light and infrared images. Experimental verification shows that the missed detection rates of the embodiment of the present invention and the current popular algorithm on the full set, daytime, and nighttime subsets of the KUSTDroneFail dataset are 19.62%, 15.96%, and 30.41%, respectively, and the detection accuracy is 86.61%, 89.44%, and 78.14%, respectively. The indicators achieved have reached the optimal level. Figure 5 It can be observed from the example diagram of the detection results shown that, compared with the comparison algorithm, the embodiment of the present invention can accurately detect the target in the test image after Gaussian noise, salt and pepper noise, and blur processing without omission, which strongly verifies that the embodiment of the present invention can alleviate the problem of unstable quality of visible light and infrared images caused by Gaussian noise and salt and pepper noise; it verifies that the embodiment of the present invention can alleviate the problem of target feature blurring caused by factors such as high-speed target motion; it verifies that the embodiment of the present invention can alleviate the problem of lack of effective information in visible light images caused by poor lighting conditions at night.

[0086] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A target detection method under unstable visible light-infrared image quality, characterized by: Step 1: Input the visible light and infrared images into the diffusion model, reconstruct the unstable target information of the visible light and infrared images, and obtain the incremental information image; Step 2: Fuse the incremental information image obtained in Step 1 with the original visible light and infrared images to obtain visible light and infrared images containing incremental information; Step 3: Input the visible light and infrared images obtained in Step 2 into the feature extraction network to extract the global features of the visible light and infrared images; Step 4: Input the features obtained in Step 3 into the kernel similarity measurement module to obtain the similarity weight of the balanced optimization network; Step 5: Multiply the weight obtained in Step 4 by the feature obtained in Step 3 to obtain the feature of balance optimization; Step 6: Input the features obtained in Step 5 into the feature fusion module to fuse the complementary information of visible light and infrared image features to obtain a fused feature map; Step 7: Input the feature map obtained in Step 6 into the detector for target detection; The method of reconstructing the unstable target information of the visible light and infrared images described in Step 1 is to first fuse the visible light and infrared images, and then convert the fused image into a normal distribution, specifically: Where rgb_images and thermal_images represent visible light and infrared images respectively, Φ(·) represents the operation of converting data into normal distribution. represents the sampling operation of the quasi-diffusion process, I c Represents the obtained incremental information image; The kernel similarity measurement module described in Step 4 is to design convolutional layers and fully connected layers to map the visible light and infrared image features into two low-dimensional feature vectors. Then, based on the two low-dimensional feature vectors, the Gaussian kernel function is used to calculate the similarity of the two low-dimensional feature vectors to obtain the similarity weight, which is specifically: In the formula, and They represent the low-dimensional vectors of visible light and infrared image features respectively, σ represents the parameters of the Gaussian kernel function, and W represents the similarity weight.

2. The target detection method under unstable visible light-infrared image quality according to claim 1, characterized in that: The fusion method described in Step 2 is to add the incremental information image to the original visible light and infrared images pixel by pixel, specifically: In the formula, Represents the element-by-element addition operation, I r Represents a visible light image containing incremental information, I t Represents an infrared image containing incremental information.

3. The target detection method under unstable visible light-infrared image quality according to claim 2, characterized in that: The feature extraction network described in Step 3 is a backbone network based on the SSD algorithm. The backbone network of the SSD algorithm is copied into two branches, one branch extracts visible light image features, and the other branch extracts infrared image features. Specifically: F r =SSD(I r ) (4) F t =SSD(I t ) (5) In the formula, SSD represents the backbone network of the SSD algorithm, and F r and F t Represent the extracted visible light and infrared image feature maps respectively.

4. The target detection method under unstable visible light-infrared image quality according to claim 3 is characterized in that: The multiplication operation described in Step 5 is to multiply the visible light and infrared image features with the similarity weights in a global operation, specifically: In the formula, Represents a global multiplication operation on the feature map, F r ' and F t 'represent the visible light and infrared image features with balanced optimization respectively.

5. The target detection method under unstable visible light-infrared image quality according to claim 4 is characterized in that: The feature fusion module in Step 6 adds a channel attention mechanism, specifically: In the formula, CAM(·) represents the channel attention mechanism, F r ' _cam and F t ' _cam Represents the fused visible light and infrared image features; The calculation formula of the channel attention mechanism is as follows: Where H and W represent the height and width of the feature map, respectively, and σ represents the hyperbolic tangent activation function.

6. The target detection method under unstable visible light-infrared image quality according to claim 5 is characterized in that: The Step 7 is specifically as follows: cls,conf,box=d_m(F r ' _cam ,F t ' _cam ) (13) Where d_m(·) represents the detector, cls, conf, box represent the category, confidence and coordinates of the bounding box of the target in the predicted image respectively.

7. The target detection method under unstable visible light-infrared image quality according to claim 1 or 6, characterized in that: The target detection refers to detecting the target in the image according to the specific task requirements. The predicted result is the bounding box corresponding to the target in the input image. The output results include classification and regression.

Images