A target detection method based on weak alignment of visible-infrared images

Abstract

The present invention relates to a target detection method under weak alignment of visible light and infrared images, and belongs to the field of image processing technology based on deep learning. Visible light and infrared images are input into a feature extraction network to extract global features of the visible light and infrared images; the visible light images are input into an illumination perception module to extract illumination information; the visible light and infrared image features are input into a multi-scale spatial alignment module for feature alignment to obtain visible light and infrared image features aligned in the same potential physical space; the visible light and infrared image features are input into a feature fusion module to fuse target complementary information of the visible light and infrared image features to obtain a fused feature map; the illumination information and feature map are input into a detector for target detection. The present invention can effectively align the features of each dimension of the visible light and infrared images at the feature level, and enhance the detection robustness of the target detector for visible light and infrared images with misaligned information in each dimension.

Description

Technical Field

[0001] The present invention relates to a target detection method under weak alignment of visible light and infrared images, and belongs to the technical field of image processing based on deep learning. Background Art

[0002] Training object detection networks with multimodal data, including visible and infrared images, is an effective way to enhance the robustness of single-modal object detection algorithms in complex environments. However, due to inaccurate spatiotemporal synchronization between data acquisition sensors, stereo disparity, and errors in the registration of visible and infrared images, the positions of the same target in visible and infrared images do not completely overlap. This misalignment complicates the model's ability to mine the complementary information between visible and infrared images.

[0003] In response to this problem, relevant researchers at home and abroad have conducted multi-faceted research: A common solution to the problem of image misalignment is image registration. Although image registration technology is mature, it mainly performs low-level transformations on images. Its main disadvantages are that the image preprocessing time is long and the detector based on convolutional neural network cannot be trained in an end-to-end manner. Due to the development of deep learning technology, the feature detection and feature description processes of image registration can be solved using deep learning methods, making the acquired features universal and the image conversion process more robust. In addition, it is not limited to the feature extraction and feature description processes. Some scholars have also used the strategy of aligning region proposal boxes based on region proposal networks to align features of different modalities and alleviate the negative impact of weakly aligned images on detection results. Different from the above-mentioned region proposal network-based alignment strategy, Zhou et al. designed a deformable convolution module in the network to regress target feature deviations; Kim et al. used a multi-label strategy to encourage the model to learn more distinctive features about targets in visible light and infrared images; Shacht et al. proposed a learnable deformation alignment method to correct the position offset of targets between modalities in real time. The deformation alignment process consists of three different transformation layers, which perform feature alignment in a coarse-to-fine manner; in 2022, Tu et al. proposed a novel deep correlation network, which considered the correlation between modalities from the spatial, feature, and semantic levels, and designed a modality alignment module to robustly learn the correlation between modalities by implicitly learning spatial affine transformations and dynamically generating intermediate representations; in the same year, Yuan et al. designed a translation-scale-rotation alignment module to predict the deviation relationship of targets in visible light and infrared images, and calibrated the visible light and infrared image feature maps through the deviation relationship to alleviate the problem of poor model detection effect caused by weak alignment of visible light and infrared images.

[0004] Most current research is based on the two-stage Faster-RCNN strategy, which first uses a two-branch network to generate region proposals and then aligns the region proposals to achieve the goal of aligning visible light and infrared features. However, this type of alignment algorithm has slow inference speed and high computational load, which is not conducive to practical application. In addition, the above research does not fully consider the inconsistencies in the size, position, angle, and distortion of objects in visible light and infrared images, and exploring only a single problem is limited. Therefore, how to align information in various dimensions to provide the model with visible light and infrared image features that are consistent in physical space and alleviate the impact on the model's object detection performance caused by the misalignment of multi-dimensional information in visible light and infrared images is an urgent problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a target detection method under weak alignment of visible light and infrared images, which is used to align the various dimensional information of visible light and infrared image features, alleviate the impact of multi-scale misalignment of visible light and infrared image targets on the effective matching and fusion of multi-dimensional information, and improve the detection effect of the model on misaligned visible light and infrared images.

[0006] The technical solution of the present invention is: a target detection method under weak alignment of visible light and infrared images, the specific steps are:

[0007] Step 1: Input the visible light and infrared images into the feature extraction network to extract the global features of the visible light and infrared images;

[0008] Step 2: Input the visible light image into the illumination perception module to extract illumination information;

[0009] Step 3: Input the visible light and infrared image features obtained in Step 1 into the multi-scale spatial alignment module for feature alignment to obtain the visible light and infrared image features aligned in the same latent physical space;

[0010] Step 4: Input the visible light and infrared image features obtained in Step 3 into the feature fusion module to fuse the target complementary information of the visible light and infrared image features to obtain the fused feature map;

[0011] Step 5: Input the illumination information obtained in Step 2 and the feature map obtained in Step 4 into the detector for target detection.

[0012] Furthermore, the feature extraction network described in Step 1 is a backbone network based on the SSD algorithm. The backbone network of the SSD algorithm is copied into two branches, one branch extracts visible light image features, and the other branch extracts infrared image features. Specifically:

[0013] F r=SSD(rgb_images) (1)

[0014] F t =SSD(thermal_images) (2)

[0015] In the formula, SSD represents the backbone network of the SSD algorithm, rgb_images and ithermal_images represent visible light and infrared images respectively, and F r and F t Represent the extracted visible light and infrared image features respectively.

[0016] Furthermore, the illumination information in Step 2 refers to the two weights extracted by the illumination perception module, where the illumination perception module consists of a convolutional layer and a fully connected layer, specifically:

[0017] W s ,W l =l_m(rgb_images) (3)

[0018] Where, l_m represents the light perception module, W s and W l Represent the two extracted weights respectively.

[0019] Furthermore, the multi-scale spatial alignment module described in Step 3 consists of graph model construction, graph attention network, graph data conversion, and spatial transformation operations. The corresponding graph model is constructed by dividing the feature map into feature blocks, and then the graph model information is updated using the graph attention network. Finally, the graph data is converted into a matrix form, and the infrared image features are corrected using spatial transformation operations. The specific method of graph model construction is as follows:

[0020]

[0021]

[0022] Where w', h' and c represent the feature block f ri and f ti The width, height, and number of channels of F are selected to ensure that the feature map is completely separable by selecting appropriate w' and h' values. R_r and F R_t Represents a collection of feature blocks.

[0023] Then, the obtained feature block is converted into the graph model node feature form, which can be expressed as follows:

[0024]

[0025]

[0026] In the formula, flatten(·) represents the operation of flattening the feature block into a vector. and Represented by the feature block f ri and f ti The flattened vector, V r and V t They represent the set of all graph nodes in the corresponding graph model, 1×h'w'c represents the number of features of the graph nodes, E r and E t Respectively represent the set of edges between graph nodes in the corresponding graph model, and G t (V t ,E t ) represent the feature maps F r and F t Constructed graph model.

[0027] The updated graph model information is as follows:

[0028]

[0029]

[0030] Where GAT(·) represents the graph attention network, G' r (V' r ,E' r ) and G' t (V' t ,E' t ) represent the visible light and infrared image feature map models obtained after being updated by the graph attention network.

[0031] The conversion of graph data into matrix form is to first merge all node features of the graph model into vectors, and then convert the vectors into matrix form, as shown in the following formula:

[0032] M r =Trans_Add(V r ') (10)

[0033] M t =Trans_Add(V t ') (11)

[0034] Where Trans_Add(·) represents the operation of merging all node features of the graph model and converting them into matrix operations, M r and M t They represent the deviation matrices of the visible light and infrared images finally modeled.

[0035] The spatial variation operation is as follows:

[0036]

[0037] Where, F t ' represents the infrared image features obtained by the spatial transformation and correction operation.

[0038] Furthermore, the feature fusion module in Step 4 adds a channel attention mechanism, specifically:

[0039]

[0040]

[0041] Where CAM(·) represents the channel attention mechanism, represents the element-by-element addition operation, Represents a global multiplication operation on the feature map, F' r_cam and F' t_cam Represents the fused visible and infrared image features.

[0042] The calculation formula of the channel attention mechanism is as follows:

[0043]

[0044]

[0045] Where H and W represent the height and width of the feature map, respectively, and σ represents the hyperbolic tangent activation function.

[0046] Furthermore, the Step 5 is specifically as follows:

[0047] cls,conf,box=d_m(W s ,W l ,F' r_cam ,F' t_cam ) (17)

[0048] Where d_m(·) represents the detector, cls, conf, and box represent the category, confidence, and coordinates of the bounding box of the target in the predicted image, respectively.

[0049] Inputting the visible light image into the illumination perception module means trimming the visible light image to a size of 56*56 and then inputting the visible light image into the illumination perception module to extract illumination information.

[0050] The target detection refers to the detection of targets in an image according to specific task requirements. The predicted result is the bounding box corresponding to the target in the input image, and the output results include classification and regression.

[0051] The beneficial effects of the present invention are as follows: the present invention utilizes deep learning-based image processing technology, with SSD as the backbone network, and introduces a multi-scale spatial alignment module to divide the visible light and infrared image feature maps into feature blocks of various sizes. Multiple graph models are constructed based on this, and then a graph attention network is used to model the deviation relationship between visible light and infrared images. Based on this deviation relationship, the visible light and infrared image features are aligned in a unified latent physical space. Ultimately, the impact of multi-scale target misalignment between visible light and infrared images on multi-dimensional information matching and fusion is alleviated, and the model's detection effect on misaligned visible light and infrared images is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the present invention;

[0053] Figure 2 It is a network framework diagram of the present invention;

[0054] Figure 3 It is a multi-scale spatial alignment module diagram of the present invention;

[0055] Figure 4 It is a diagram of a bias module of the present invention;

[0056] Figure 5 It is a feature fusion module diagram of the present invention;

[0057] Figure 6 2 is an example diagram of the detection results of the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] Example 1: The embodiment of the present invention adopts an improved SSD algorithm and trains the network by combining visible light and infrared images to achieve the goal of the algorithm still having the ability to detect targets with high precision when the visible light and infrared images are weakly aligned.

[0060] In order to effectively align the features of visible light and infrared images, the embodiment of the present invention adopts the SSD algorithm as the backbone network and makes improvements to the network. Based on the SSD algorithm backbone network, a multi-scale spatial alignment module is added to model the deviation relationship between visible light and infrared images, and the features of visible light and infrared images are aligned in the same potential physical space. The feature map obtained after the alignment process is then input into the dense feature fusion module to fuse the complementary information of visible light and infrared image features. The detection flow chart is shown below. Figure 1As shown, the algorithm network framework diagram is as follows Figure 2 The algorithm network is built and trained based on the Tensorflow framework. The training dataset uses the CVC-14 pedestrian detection dataset. The Adam optimization method is used during the training process. The initial learning rate is 0.0001, and the learning rate is reduced by 10 every 6 rounds. -1 , the training time is 30 rounds.

[0061] Step 1: First, the visible light and infrared images are input into the feature extraction module respectively to extract the global features of the visible light and infrared images; the feature extraction module is based on the backbone network of the SSD algorithm, and the backbone network of the SSD algorithm is copied into two branches, one branch extracts the visible light image features, and the other branch extracts the infrared image features. Specifically:

[0062] F r =SSD(rgb_images) (1)

[0063] F t =SSD(thermal_images) (2)

[0064] In the formula, SSD represents the backbone network of the SSD algorithm, rgb_images and ithermal_images represent visible light and infrared images respectively, with a size of 512*640*3, and F r and F t Represents the extracted visible light and infrared image features respectively, and its size is 128*160*256.

[0065] Step 2: Lighting information is a priori condition that helps the detector make correct detection results. Therefore, the network introduces a light perception module, inputs the visible light image into the light perception module, extracts the light information, and makes detections based on the light information in the subsequent detection process. The light information refers to the two weights extracted by the light perception module, specifically:

[0066] W s ,W l =l_m(rgb_images) (3)

[0067] Where, l_m represents the light perception module, W s and W l Represent the two extracted weights, specifically the weights of good lighting conditions and poor lighting conditions, and W s and W l The relationship is: 1 = W s +W l .

[0068] Step 3: Input the visible light and infrared image features obtained in step 1 into the multi-scale spatial alignment module for feature alignment, and obtain the visible light and infrared image features aligned in the same potential physical space. The multi-scale spatial alignment module consists of graph model construction, graph attention network, graph data conversion and spatial transformation operation. The corresponding graph model is constructed by dividing the feature graph into feature blocks, and then the graph attention network is used to update the graph model information. Finally, the graph data is converted into a matrix form, and the infrared image features are corrected using the spatial transformation operation. The structure diagram of the multi-scale spatial alignment module is shown in the figure. Figure 3 and Figure 4 As shown. The specific method of constructing the graph model is as follows:

[0069]

[0070]

[0071] Where w' s 、h' s , c and n s Represents the width, height, channel and number of feature blocks at scale s, respectively. R_r_s and F R_t_s Represents the visible light and infrared image feature block sets at scale s, respectively, and f r_s_i and f t_s_i Represents the feature block set F R_r_s and F R_t_s The embodiment of the present invention converts the feature map F into three different scales of 4×10, 8×20 and 16×40. r and F t Divide.

[0072] Then, the obtained feature block is converted into the graph model node feature form, which can be expressed as follows:

[0073]

[0074]

[0075] In the formula, flatten(·) represents the operation of flattening the feature block into a vector. and Respectively represented by f r_s_i and f t_s_i The node feature vector obtained by conversion; V r_s and V t_s They represent the set of all graph nodes in the corresponding graph model at scale s; E r_s and E t_s Represents the set of edges between graph nodes in the corresponding graph model; G r_s (Vr_s ,E r_s ) and G t_s (V t_s ,E t_s ) represent the feature maps F r and F t A graphical model constructed at scale s.

[0076] The updated graph model information is as follows:

[0077]

[0078]

[0079] Where GAT(·) represents the graph attention network, G' r_s (V' r_s ,E' r_s ) and G' t_s (V' t_s ,E' t_s ) represent the visible light and infrared image feature map models obtained after being updated by the graph attention network.

[0080] The conversion of graph data into matrix form is to first merge all node features of the graph model into vectors, and then convert the vectors into matrix form, as shown in the following formula:

[0081] M r_s =Trans_Add(V' r_s ) (10)

[0082] M t_s =Trans_Add(V' t_s ) (11)

[0083] Where Trans_Add(·) represents the operation of merging all node features of the graph model and converting them into matrix operations, M r_s and M t_s They represent the deviation matrices of the visible light and infrared images finally modeled.

[0084] The spatial variation operation is as follows:

[0085]

[0086] Where F' t_s Represents the warp deviation matrix Infrared image features obtained by transformation operation.

[0087] In order to more robustly align target features of different scales, the embodiment of the present invention combines the infrared feature maps obtained after the above three different offset matrix transformations: F' t_1 、F't_2 、F' t_3 Specifically, first Perform full connection operations to generate corresponding weights W1, W2, and W3, which are used to measure the contribution of different deviation matrices to the aligned infrared feature map, as shown in the following formula:

[0088] F' t =Add(W1F' t_1 ,W2F' t_2 ,W3F' t_3 ) (13)

[0089] Where Add(·) represents the addition operation, F t ' represents the visible light feature map that has been aligned.

[0090] Step 4: Input the visible light and infrared image features obtained in step 3 into the feature fusion module to fuse the target complementary information of the visible light and infrared image features to obtain the fused feature map. The structure diagram of the feature fusion module is shown in the figure. Figure 5 As shown; the feature fusion module adds a channel attention mechanism. First, the visible light and infrared image features are subtracted element by element. The features obtained after subtraction represent the difference features of the visible light and infrared image features, and the difference features are regarded as complementary information. The difference features are then input into the channel attention mechanism to obtain the channel weights. Finally, the channel weights are multiplied by the corresponding visible light and infrared image features to amplify the difference features, and are added to the features of the other modality to obtain the final fused features. Therefore, the features obtained after fusion contain the difference features from the other modality, thereby achieving the purpose of fusing complementary information. The specific calculation formula is as follows:

[0091]

[0092]

[0093] Where CAM(·) represents the channel attention mechanism, represents the element-by-element addition operation, Represents a global multiplication operation on the feature map, F' r_cam and F' t_cam Represents the fused visible and infrared image features.

[0094] The calculation formula of the channel attention mechanism is as follows:

[0095]

[0096]

[0097] Where H and W represent the height and width of the feature map, respectively, and σ represents the hyperbolic tangent activation function.

[0098] Step 5: Input the illumination information obtained in step 2 and the feature map obtained in step 4 into the detector for target detection, specifically:

[0099] cls,conf,box=d_m(W s ,W l ,F' r_cam ,F' t_cam ) (17)

[0100] Where d_m(·) represents the detector, cls, conf, and box represent the category, confidence, and coordinates of the bounding box of the target in the predicted image, respectively.

[0101] Finally, the loss function of the network is divided into four parts: illumination loss L i , category loss L cls and regression box loss L reg , where the light loss L i Use cross entropy loss, category loss L cls Use Focal loss, regression box loss L reg Use smoothL1loss. The final objective function is:

[0102] L=L i +L cls +L reg (18)

[0103] By optimizing L to reach a convergence state, the trained model is finally used to detect targets in visible light and infrared images. Experimental verification shows that the missed detection rates of the embodiment of the present invention on the entire CVC-14 pedestrian detection dataset, daytime subsets, and nighttime subsets are 22.59%, 25.14%, and 17.23%, respectively, and the missed detection rates achieved are the lowest. Figure 6 It can be observed from the example diagram of the detection results shown that, compared with the comparison algorithm, the embodiment of the present invention has better detection performance and can detect targets accurately and without omission. Even if the embodiment of the present invention and the comparison algorithm both detect targets in the same test image, it is obvious that the target bounding box regressed by the embodiment of the present invention is closer to the target. The reason is that the comparison algorithm fuses the misaligned visible light and infrared image features, and the features obtained will produce ghosting near the target feature area, making the target area larger, resulting in the detected target bounding box being larger. However, the embodiment of the present invention first aligns the visible light and infrared image features so that the visible light and infrared image target information are spatially consistent. The features obtained after the fusion avoid the ghosting problem. Therefore, the target bounding box detected by the embodiment of the present invention can more accurately surround the target.

[0104] The above describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A target detection method using weak alignment of visible light and infrared images, characterized by: Step 1: Input the visible light and infrared images into the feature extraction network to extract the global features of the visible light and infrared images; Step 2: Input the visible light image into the illumination perception module to extract illumination information; Step 3: Input the visible light and infrared image features obtained in Step 1 into the multi-scale spatial alignment module for feature alignment to obtain the visible light and infrared image features aligned in the same latent physical space; Step 4: Input the visible light and infrared image features obtained in Step 3 into the feature fusion module to fuse the target complementary information of the visible light and infrared image features to obtain the fused feature map; Step 5: Input the illumination information obtained in Step 2 and the feature map obtained in Step 4 into the detector for target detection; The multi-scale spatial alignment module described in Step 3 consists of graph model construction, graph attention network, graph data conversion, and spatial transformation operations. The corresponding graph model is constructed by dividing the feature map into feature blocks, and then the graph model information is updated using the graph attention network. Finally, the graph data is converted into a matrix form, and the infrared image features are corrected using spatial transformation operations. The specific method of graph model construction is as follows: Where w', h' and c represent the feature block f ri and f ti Width, height, number of channels, F R_r and F R_t Represents a set of feature blocks; Then, the obtained feature block is converted into the graph model node feature form as follows: In the formula, flatten(·) represents the operation of flattening the feature block into a vector. and Represented by the feature block f ri and f ti The flattened vector is and They represent the set of all graph nodes in the corresponding graph model, 1×h'w'c represents the number of features of the graph nodes, ε r and ε t Respectively represent the set of edges between graph nodes in the corresponding graph model, and Represented by the feature map F r and F t Constructed graphical model; The updated graph model information is as follows: Where GAT(·) represents the graph attention network, and They represent the visible light and infrared image feature map models obtained after being updated by the graph attention network; The conversion of graph data into matrix form is to first merge all node features of the graph model into vectors, and then convert the vectors into matrix form, as shown in the following formula: Where Trans_Add(·) represents the operation of merging all node features of the graph model and converting them into matrix operations, M r and M t Represent the deviation matrices of the visible light and infrared images obtained by the final modeling; The spatial transformation operation is as follows: Where, F t ' represents the infrared image features corrected by spatial transformation operation.

2. The target detection method under weak alignment of visible light and infrared images according to claim 1, characterized in that: The feature extraction network described in Step 1 is the backbone network based on the SSD algorithm. The backbone network of the SSD algorithm is copied into two branches, one branch extracts visible light image features, and the other branch extracts infrared image features. Specifically: F r =SSD(rgb_images) (1) F t =SSD(thermal_images) (2) In the formula, SSD represents the backbone network of the SSD algorithm, rgb_images and ithermal_images represent visible light and infrared images respectively, and F r and F t Represent the extracted visible light and infrared image features respectively.

3. The target detection method under weak alignment of visible light and infrared images according to claim 1, characterized in that: The illumination information in Step 2 refers to the two weights extracted by the illumination perception module, where the illumination perception module consists of a convolutional layer and a fully connected layer, specifically: W s ,W l =l_m(rgb_images) (3) Where, l_m represents the light perception module, W s and W l Represent the two extracted weights respectively.

4. The target detection method under weak alignment of visible light and infrared images according to claim 1, characterized in that: The feature fusion module in Step 4 adds a channel attention mechanism, specifically: Where CAM(·) represents the channel attention mechanism, represents the element-by-element addition operation, Represents a global multiplication operation on the feature map, F r ' _cam and F t ' _cam Respectively represent the fused visible light and infrared image features; The calculation formula of the channel attention mechanism is as follows: Where H and W represent the height and width of the feature map, respectively, and σ represents the hyperbolic tangent activation function.

5. The target detection method under weak alignment of visible light and infrared images according to claim 1, characterized in that: The Step 5 is specifically as follows: cls,conf,box=d_m(W s ,W l ,F’ r_cam ,F’ t_cam ) (17) Where d_m(·) represents the detector, cls, conf, and box represent the category, confidence, and coordinates of the bounding box of the target in the predicted image, respectively.

6. The target detection method under weak alignment of visible light and infrared images according to claim 1 or 3, characterized in that: Inputting the visible light image into the illumination perception module means trimming the visible light image to a size of 56*56 and then inputting the visible light image into the illumination perception module to extract illumination information.

7. The target detection method using weak alignment of visible light and infrared images according to claim 1, characterized in that: The target detection refers to the detection of targets in an image according to specific task requirements. The predicted result is the bounding box corresponding to the target in the input image, and the output results include classification and regression.

Images