Infrared and visible light target detection method based on differential feature perception in adversarial learning

Abstract

The present invention discloses an infrared and visible light target detection method based on differential feature perception in adversarial learning, comprising: extracting differential infrared features and visible light features respectively using a non-shared feature extraction network based on a fused divergence loss function composed of KL and JS divergences; re-extracting the extracted infrared features and visible light features using an IR-Attention module and an RGB-Attention module respectively, paying more attention to the effective information in the infrared feature map and the visible light feature map; fusing the re-extracted differential dual-mode features using an F-Attention module, retaining the commonalities of the dual-mode features while pursuing more differential information; and regressing and classifying the fused dual-mode features using an RPN to complete target detection. The advantages of the present invention are: it can cause the feature extraction network to produce adversarial behavior, extract as much unique dual-mode information as possible, and the feature fusion network based on multiple attention mechanisms can comprehensively fuse complementary differential dual-mode features, thereby improving the detection accuracy of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision target detection, and in particular to an infrared and visible light target detection method based on differential feature perception in adversarial learning. Background Art

[0002] Images captured by a single sensor or under a single shooting setup can only describe the imaging scene from a limited perspective, which limits the algorithm's understanding of semantics. Some studies have enhanced feature mapping by combining multimodal complementary images obtained from different sensors or different shooting setups, and this has been proven to be effective. In all multimodal target detection scenarios, infrared and visible light images are the two most commonly used signals. Infrared images capture the thermal radiation emitted by objects and can effectively highlight salient objects, but lack texture details. In contrast, visible light images typically contain rich detail information but are easily affected by complex backgrounds and lose objects. These complementary features help the algorithm learn to obtain results that contain rich texture details and accurate semantic information. Due to the excellent detection performance of infrared and visible light target detection, they have been widely used in many fields such as video surveillance, vehicle detection, pedestrian detection, and autonomous driving.

[0003] A large body of research has been conducted on infrared and visible light object detection. However, these methods fail to fully exploit the unique information of the two modalities, resulting in incomplete complementary feature extraction and insufficient fusion. Specifically, most algorithms use a shared feature extraction network to extract infrared and visible light features, neglecting the unique infrared structural information and visible light texture and color information. Furthermore, some work considers the complementary information between infrared and visible light features while ignoring the connection between feature extraction and feature fusion, employing only simple fusion strategies, resulting in the loss of the extracted differential unique features.

[0004] References

[0005] [1]N.Dalal and B.Triggs, "Histograms of oriented gradients for human detection," in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), vol.1.Ieee, 2005, pp.886–893.

[0006] [2]K.Park,S.Kim,and K.Sohn,“Unified multi-spectral pedestriandetectionbased on probabilistic fusion networks,”Pattern Recognition,vol.80,pp.143–155,2018.

[0007] [3]X.Xiao,B.Wang,L.Miao,L.Li,Z.Zhou,J.Ma,and D.Dong,“Infraredandvisible image object detection via focused feature enhancement andcascaded semanticextension,”Remote Sensing,vol.13,no.13,p.2538,2021.

[0008] [4]C.Li,D.Song,R.Tong,and M.Tang,“Illumination-aware faster rcnnforrobust multispectral pedestrian detection,”Pattern Recognition,vol.85,pp.161–171,2019.

[0009] [5]P.Viola and M.Jones,“Rapid object detection using a boostedcascade ofsimple features,”in Proceedings of the 2001 IEEE computer societyconference oncomputer vision and pattern recognition.CVPR 2001,vol.1.Ieee,2001,pp.I–I.

[0010] [6]P.Viola and M.J.Jones,“Robust real-time face detection,”Internationaljournal of computer vision,vol.57,no.2,pp.137–154,2004.

[0011] [7]P.Felzenszwalb,D.McAllester,and D.Ramanan,“A discriminativelytrained,multiscale,deformable part model,”in 2008 IEEE conference on computervision andpattern recognition.Ieee,2008,pp.1–8.

[0012] [8]A.Bochkovskiy,C.-Y.Wang,and H.-Y.M.Liao,“Yolov4:Optimal speedandaccuracy of object detection,”arXiv preprint arXiv:2004.10934,2020.

[0013] [9]S.Ren,K.He,R.Girshick,and J.Sun,“Faster r-cnn:Towards real-timeobject detection with region proposal networks,”Advances in neuralinformation processing systems,vol.28,2015.

[0014]

[10] W.Liu,D.Anguelov,D.Erhan,C.Szegedy,S.Reed,C.-Y.Fu,and A.C.Berg,“Ssd:Single shot multibox detector,”in European conference on computervision.Springer,2016,pp.21–37.

[0015]

[11] S.Liu and Z.Liu,“Multi-channel cnn-based object detection forenhanced situation awareness,”arXiv preprint arXiv:1712.00075,2017.

[0016]

[12] Z.Zhou, B.Wang, S.Li, and M.Dong, “Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters,” Information Fusion, vol.30, pp.15–26, 2016.

[0017]

[13] J. Liu, S. Zhang, S. Wang, and DNMetaxas, "Multispectral deep neural networks for pedestrian detection," arXiv preprint arXiv:1611.02644, 2016. Summary of the Invention

[0018] In response to the defects of the prior art, the present invention provides an infrared and visible light target detection method based on differential feature perception in adversarial learning.

[0019] In order to achieve the above object of the invention, the technical solution adopted by the present invention is as follows:

[0020] A method for infrared and visible light target detection based on differential feature perception in adversarial learning, the method comprising the following steps:

[0021] Step 1: Based on the fusion divergence loss function composed of KL divergence and JS divergence, a non-shared feature extraction network is used to extract differential infrared features and visible light features respectively;

[0022] Step 2: Use the IR-Attention module and RGB-Attention module to re-extract the infrared features and visible light features extracted in step 1, respectively, paying more attention to the effective information in the infrared feature map and the visible light feature map;

[0023] Step 3: Use the F-Attention module to fuse the extracted differential bimodal features, retaining the commonalities of the bimodal features while pursuing more differential information;

[0024] Step 4: Use RPN to regress and classify the fused dual-mode features to complete target detection.

[0025] Furthermore, in step 1, a fusion divergence loss function consisting of KL divergence and JS divergence is applied to measure the similarity between infrared features and visible light features.

[0026] The similarity between infrared and visible light features is negatively correlated with the KL and JS divergence values. KL divergence is expressed as:

[0027]

[0028] JS divergence is expressed as:

[0029]

[0030] The fusion divergence loss function defined based on KL divergence and JS divergence is expressed as:

[0031]

[0032] Where F1 and F2 are the features output by the last convolutional layer of each ResNet in the two backbone networks; f1 and f2 represent the pixel intensities at each position in F1 and F2, respectively, which are calculated by the softmax function; N represents the number of features in F1 or F2, with the maximum value being N. Furthermore, in step 2, the IR-Attention module performs attention mechanisms in two directions, including a channel attention module and a spatial attention module.

[0033] In the channel attention module, the connection between channel direction information is used to generate the corresponding weights. First, global average pooling is used to Squeeze to obtain Then, F1 c Feed it into two fully connected layers to obtain the weight vector. Finally, reshape the weight vector into R C×H×W The channel attention module is expressed as:

[0034] M c =Reshape(FC1(FC0(GAP(F1)))) (4)

[0035] Among them, FC is the fully connected layer and GAP is the global average pooling operation.

[0036] In the spatial attention module, the spatial attention module is expressed as:

[0037] M s =Reshape(Conv1(Conv3(DConv(GAP(F1))))) (5)

[0038] Where, Conv i is a convolutional layer with kernel i×i, and DConv is a dilated convolution. After obtaining the weights of the channel attention module and the spatial attention module respectively;

[0039] Combining the channel attention module and the spatial attention module, we get the final weight M(F1) of the input feature F1, which is expressed as:

[0040] M(F1)=σ(M c +M s ) (6)

[0041] Where σ is the sigmoid function, and the infrared feature map processed by the IR-Attention module is expressed as:

[0042] F1'=F1+F1*M(F1) (7)

[0043] Furthermore, the RGB-Attention module in step 2 is used to generate contextually relevant visible light features. These visible light features F2 are then fed into three self-attention modules: the channel attention module, the height attention module, and the width attention module. These modules generate features with correlations in the channel, height, and width dimensions, respectively. The channel attention module is described in detail below. The height attention module and the width attention module have similar structures to the channel attention module.

[0044] Furthermore, firstly, the input feature map Perform three linear operations respectively to obtain three feature maps of query, key and value, which are represented by E, F and G respectively. Next, reshape E into Reshape F into After transposing Get the channel attention matrix:

[0045]

[0046] Where T ji represents the influence of the i-th channel on the j-th channel, and this correlation is positively correlated with the value of T. At the same time, G is reshaped into And perform matrix multiplication on T and G, reshaping the result into This is the sub-output feature map of the middle part. This embodiment defines the sub-output as

[0047]

[0048] Use global average pooling F sq (·) The original feature map Compress to This operation compresses the size of the feature map while retaining the complete information of the feature map. M can be expressed as:

[0049]

[0050] Where H and W are the height and width of the original feature map respectively. Next, M is fed into two fully connected layers to obtain the activation value N is represented by:

[0051] N=F ex (M,ω)=σ(ω2δ(ω1M)) (11)

[0052] Where σ(·) and δ(·) are the sigmoid function and ReLU function respectively. are the weights of the first and second fully connected layers respectively, r is the number of neurons in the hidden layers of the two fully connected layers, and r is set to 16. The sub-output feature map of this part Expressed as:

[0053]

[0054] Among them F scale (·,·) represents the multiplication of corresponding elements.

[0055] Finally, the two sub-output feature maps are obtained and Linearly combined with the original feature map F2, the total output feature map of the channel attention module is obtained

[0056]

[0057] The initial values ​​of α and β are both 0, and the weights are finally determined during the learning process. Similarly, this embodiment can obtain the height feature map F 2,H and width feature map F 2,W Finally, this embodiment combines the three feature maps by pixel-by-pixel addition to obtain the output of RGB-Attention, that is, the visible light feature F'2 with contextual connection:

[0058] F'2=F 2,C +F 2,H +F 2,W (14)

[0059] Furthermore, in step 3, F-Attention is used to fuse the differential infrared features and visible light features obtained in step 2. The formula for using F-Attention to fuse different infrared and visible light features is as follows:

[0060] F=F1'*Conv1(Conv1(F'))+F2'Conv1(Conv1(F'))+Conv1(Conv1(F')) (15)

[0061] Finally, the tensor output by F-Attention is reconstructed to obtain the final fusion feature.

[0062] Furthermore, in step 4, RPN is used to perform regression of the prediction box and classification of the target.

[0063] During the training process, multiple loss functions are used to assist the model in error analysis, namely, the loss L based on the fusion divergence div , classification loss and regression loss. The total loss is the weighted sum of the three losses, expressed as:

[0064]

[0065] Among them, L cls and L reg are the loss functions for classification and regression respectively. i and are the predicted probability of the object contained by the i-th anchor and its true value respectively. i Contains the four position parameters of the i-th bounding box. Contains the four position parameters of the ground truth box corresponding to the i-th bounding box. cls and N reg are the mini-batch size and the number of anchors, respectively. λ is the weight of the regression loss function.

[0066] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the above-mentioned infrared and visible light target detection method based on differential feature perception in adversarial learning is implemented.

[0067] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the infrared and visible light target detection method based on differential feature perception in adversarial learning is implemented.

[0068] Compared with the prior art, the advantages of the present invention are:

[0069] The ability to extract complementary differential infrared and visible light features as completely as possible allows for comprehensive fusion of the extracted infrared and visible light differential features, significantly improving target detection accuracy. This is of great significance for improving the performance of infrared and visible light target detection models.

[0070] In both the ICV and CVC 14 datasets, the AP of the proposed method is higher than the existing technology. Compared with the suboptimal model, the mAP of the proposed method on the two datasets is improved by 2.1% and 3.9% respectively.

[0071] The improvement of the present invention is more significant in the categories of cars, buses and trucks, and it shows stronger robustness in detecting large objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a general schematic diagram of the infrared and visible light target detection method according to an embodiment of the present invention;

[0073] Figure 2 This is a diagram of a multi-residual block structure according to an embodiment of the present invention;

[0074] Figure 3 This is a structural diagram of the attention-based differential feature fusion (ADFF) module according to an embodiment of the present invention;

[0075] Figure 4 This is a structural diagram of the RGB-Attention module according to an embodiment of the present invention;

[0076] Figure 5 This is a visualization diagram of the detection results on the ICV and CVC 14 datasets according to an embodiment of the present invention;

[0077] Figure 6 Graph showing the ablation experiment results of the divergence loss according to an embodiment of the present invention;

[0078] Figure 7 This is a diagram showing the ablation experiment results of the attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0080] A method for infrared and visible light target detection based on differential feature perception in adversarial learning, the method comprising the following steps:

[0081] Step 1: Based on the fusion divergence composed of KL and JS divergence, a non-shared feature extraction network is used to extract differential infrared features and visible light features respectively;

[0082] Step 2: Use the IR-Attention module and RGB-Attention module to re-extract the infrared features and visible light features extracted in step 1, respectively, paying more attention to the effective information in the infrared feature map and the visible light feature map;

[0083] Step 3: Use the F-Attention module to fuse the extracted differential bimodal features, retaining the commonalities of the bimodal features while pursuing more differential information;

[0084] Step 4: Use RPN to regress and classify the fused dual-mode features to complete target detection.

[0085] The following combination Figure 1 The present invention will be further described.

[0086] 1. Overall Solution for Infrared and Visible Light Target Detection Based on Differential Feature Perception in Adversarial Learning

[0087] The proposed target detection method is as follows Figure 1 As shown. First, this embodiment designs a divergence-based adversarial feature extraction (AFED) module, which uses non-shared residual blocks to extract infrared features and visible light features respectively; and uses a fused divergence loss function composed of KL and JS divergences to calculate the output feature value of each residual block. The loss-supervised dual-mode feature extraction network generates adversarial and learns to extract more differential dual-mode features. Secondly, this embodiment proposes an attention-based differential feature fusion (ADFF) module, which consists of three parts: IR-Attention, RGB-Attention, and F-Attention. ADFF takes the extracted multi-scale differential bi-modal features as input to achieve more comprehensive differential feature fusion. Finally, the fused multi-scale features are combined into RPN to complete the detection.

[0088] 2. Divergence-based adversarial feature extraction module

[0089] AFED uses two non-parameter-sharing multiple residual networks with the same structure as the backbone network. Its structure is as follows: Figure 2 As shown in Figure 2, each residual network consists of three residual blocks. By connecting the residual blocks, shallow detail features and deep semantic features are residually connected, and features containing more information can be obtained.

[0090] In addition, in order to extract complementary infrared differential features and visible light features, this embodiment proposes a fusion divergence loss function, which is composed of KL divergence and JS divergence. This embodiment uses it to measure the similarity between infrared features and visible light features. The similarity between infrared and visible light features is negatively correlated with the KL and JS divergence values. KL divergence can be expressed as:

[0091]

[0092] JS divergence can be expressed as:

[0093]

[0094] The fusion divergence loss function defined based on KL divergence and JS divergence can be expressed as:

[0095]

[0096] Where F1 and F2 are the features output by the last convolutional layer of each ResNet in the two backbone networks; f1 and f2 represent the pixel intensity at each position in F1 and F2, respectively, which are calculated by the softmax function; N represents the number of features in F1 or F2, and the maximum value is N.

[0097] Use IR-Attention to focus on the effective information in infrared features. The specific structure is as follows Figure 3 As shown in Figure 2, the IR-Attention module performs attention mechanisms in two directions: channel attention module and spatial attention module.

[0098] In the channel attention module, this embodiment uses the connection between channel direction information to generate corresponding weights. First, this embodiment uses global average pooling to Squeeze to obtain Then, F1 c Then, the weight vector is fed into two fully connected layers to obtain the weight vector. Finally, this embodiment reshapes the weight vector into R C×H×W The channel attention module can be expressed as:

[0099] M c =Reshape(FC1(FC0(GAP(F1)))) (4)

[0100] Among them, FC is the fully connected layer and GAP is the global average pooling operation.

[0101] In the spatial attention module, in order to process the contextual information of different spatial positions, the convolution operation requires a larger receptive field. The spatial attention module can be expressed as:

[0102] M s =Reshape(Conv1(Conv3(DConv(GAP(F1))))) (5)

[0103] Where, Conv i is a convolutional layer with a kernel of i×i, and DConv is a dilated convolution. After obtaining the weights of the channel attention module and the spatial attention module respectively, this embodiment combines the two to obtain the final weight M(F1) of the input feature F1, which can be expressed as:

[0104] M(F1)=σ(M c +M s ) (6)

[0105] Where σ is the sigmoid function, and the infrared feature map processed by the IR-Attention module can be expressed as:

[0106] F1'=F1+F1*M(F1) (7)

[0107] RGB-Attention is proposed to generate visible light features with contextual connections, such as Figure 3 As shown in Figure 2, the visible light feature F2 is input into three self-attention modules: the channel attention module, the height attention module, and the width attention module. Correlated features in the channel, height, and width dimensions can be obtained, respectively. This embodiment introduces the channel attention module in detail. The height attention module and the width attention module have similar structures to the channel attention module.

[0108] like Figure 4 As shown in the middle part, this embodiment first inputs the feature map Performing three linear operations respectively can obtain three feature maps of query, key and value, which are represented by E, F and G respectively. Next, reshape E into Reshape F into After transposing The channel attention matrix can be obtained:

[0109]

[0110] Where T ji represents the influence of the i-th channel on the j-th channel, and this correlation is positively correlated with the value of T. At the same time, G is reshaped into And perform matrix multiplication on T and G, reshaping the result into This is the sub-output feature map of the middle part. This embodiment defines the sub-output as

[0111]

[0112] like Figure 4 As shown in the lower part, this embodiment uses global average pooling F sq (·) The original feature map Compress to This operation compresses the size of the feature map while retaining the complete information of the feature map. M can be expressed as:

[0113]

[0114] Where H and W are the height and width of the original feature map respectively. Next, this embodiment sends M to two fully connected layers to obtain the activation value N can be expressed as:

[0115] N=F ex(M,ω)=σ(ω2δ(ω1M)) (11)

[0116] Where σ(·) and δ(·) are sigmoid function and ReLU function respectively. are the weights of the first and second fully connected layers respectively, r is the number of neurons in the hidden layers of the two fully connected layers, and r is set to 16. Sub-output feature map It can be expressed as:

[0117]

[0118] Among them F scale (·,·) represents the multiplication of corresponding elements.

[0119] Finally, this embodiment obtains two sub-output feature maps and Linearly combined with the original feature map F2, the total output feature map of the channel attention module is obtained

[0120]

[0121] The initial values ​​of α and β are both 0, and the weights are finally determined during the learning process. Similarly, this embodiment can obtain the height feature map F 2,H and width feature map F 2,W Finally, this embodiment combines the three feature maps by pixel-by-pixel addition to obtain the output of RGB-Attention, that is, the visible light feature F2' with contextual connection:

[0122] F2'=F 2,C +F 2,H +F 2,W (14)

[0123] F-Attention is used to fuse different infrared and visible light features to retain as much effective single-mode unique information as possible while retaining the commonalities of dual-mode features, such as Figure 3 shown.

[0124] F=F1'*Conv1(Conv1(F'))+F2'Conv1(Conv1(F'))+Conv1(Conv1(F')) (15)

[0125] Finally, the tensor output by F-Attention is reconstructed to obtain the final fusion feature.

[0126] 4. Supervised Training

[0127] During the training process, this embodiment uses multiple loss functions to assist the model in error analysis, namely, the loss L based on the fusion divergence div , classification loss and regression loss. The total loss is the weighted sum of the three losses and can be expressed as:

[0128]

[0129] Among them, L cls and L reg are the loss functions for classification and regression respectively. i and are the predicted probability of the object contained by the i-th anchor and its true value respectively. i Contains the four position parameters of the i-th bounding box. Contains the four position parameters of the ground truth box corresponding to the i-th bounding box. cls and N reg are the mini-batch size and the number of anchors, respectively. λ is the weight of the regression loss function.

[0130] V. Analysis and Discussion

[0131] To verify the effectiveness of the anti-interference method proposed in the paper, this example evaluates the proposed network on two datasets: the ICV dataset and the CVC 14 dataset. The mean average precision (mAP) is used to evaluate the performance of the target detection algorithm, and the proposed network is trained using the PyTorch framework. The model is trained using Adam optimization, with the initial learning rate of each layer set to 1e-4 and the regularization weight decay to 1e-5. The backbone network is initialized using parameters pre-trained on the ImageNet dataset

[20] , and the remaining parameters are randomly initialized.

[0132] In this example, the proposed DFANet is compared with several advanced infrared and visible light image target detection algorithms on two datasets. Among them, the traditional methods include VJ+PSO[5],[6], HOG+SVM[1], DPM[7], and the deep learning-based methods include YOLOv4[8], Faster-RCNN[9], SSD

[10] , MCDetection

[11] , FusionDetection

[12] , TwoFusion

[13] , TripleFusion[2], IAF R-CNN[4], and IVOD[3]. As shown in Tables 1 and 2, the AP of DFANet in each category is higher than that of other methods. Compared with the second-best model (IVOD), the mAP of the proposed model on the two datasets is improved by 2.1% and 3.9% respectively, and the improvement is more significant in the categories of cars, buses, and trucks. From the results, it can be seen that the proposed DFANet shows stronger robustness in detecting large targets. This example attributes this result to the complementary infrared and visible light features that enhance target detection. The divergence-based loss function forces the feature extraction network to compete, extracting differential infrared features containing structural information and visible light features containing texture and color information. Furthermore, the attention-based fusion module preserves the differences between the differential bimodal features, ensuring that the fused features contain more comprehensive bimodal information. This approach forces the network to more effectively utilize the complementary infrared and visible light features, improving detection performance.

[0133] Table 1 Detection accuracy of ICV dataset

[0134]

[0135] Table 2 Detection accuracy of CVC 14 dataset

[0136]

[0137] Figure 5 This example randomly selected six different traffic scenarios to visualize the results of other deep learning detection algorithms and the algorithm proposed in this example on the ICV and CVC 14 datasets. The results show that the algorithm proposed in this example can better detect targets and provide more accurate and comprehensive target recognition.

[0138] An ablation experiment of the entire module was conducted. First, this embodiment removes the divergence module and the attention module so that the entire network only contains the backbone and other required architectures, and records it as the initial network. In order to evaluate the effectiveness of each module of the network, this embodiment conducted various experiments, as shown in Table 3. While keeping the training strategy and processing flow unchanged, this embodiment gradually adds the divergence module and the attention module to the initial model. It can be seen that with the addition of each module, the detection mAP has been improved; the addition of the divergence module has improved by 5.8% and 7.4% on the ICV and CVC 14 datasets respectively, and the addition of the attention module has improved by 4.4% and 5.9% on the two datasets respectively.

[0139] The ablation experiment of the divergence module was conducted. First, this embodiment removes the divergence module from the proposed DFANet and uses it as the base network. Next, this embodiment adds JS divergence, KL divergence and KL+JS divergence to the base network to assist in the extraction of complementary features, such as Figure 6 It can be seen that each module contributes to the improvement of detection accuracy, and the method of combining KL divergence and JS divergence adopted in this embodiment is the most effective method to improve detection accuracy.

[0140] The ablation experiment of the attention module was conducted. First, this embodiment removes the attention module from the proposed DFANet and uses it as the base network. Next, this embodiment adds IR-Attention, RGB-Attention, F-Attention, IR+RGB-Attention, and IR+RGB+F-Attention to the base network respectively, as shown in the following figure. Figure 7 As shown in the figure, it can be seen that adding a single attention module or a combination of multiple attention modules contributes to the performance of the model, and the effectiveness of the module has been effectively verified through experiments.

[0141] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of an infrared and visible light target detection method based on differential feature perception in adversarial learning, including the following steps:

[0142] Step 1: Based on the fusion divergence composed of KL and JS divergence, a non-shared feature extraction network is used to extract differential infrared features and visible light features respectively;

[0143] Step 2: Use the IR-Attention module and RGB-Attention module to re-extract the infrared features and visible light features extracted in step 1, respectively, paying more attention to the effective information in the infrared feature map and the visible light feature map;

[0144] Step 3: Use the F-Attention module to fuse the extracted differential bimodal features, retaining the commonalities of the bimodal features while pursuing more differential information;

[0145] Step 4: Use RPN to regress and classify the fused dual-mode features to complete target detection.

[0146] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0147] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the above-mentioned embodiment related to an infrared and visible light target detection method based on differential feature perception in adversarial learning. The processor may load and execute the following steps:

[0148] Step 1: Based on the fusion divergence composed of KL and JS divergence, a non-shared feature extraction network is used to extract differential infrared features and visible light features respectively;

[0149] Step 2: Use the IR-Attention module and RGB-Attention module to re-extract the infrared features and visible light features extracted in step 1, respectively, paying more attention to the effective information in the infrared feature map and the visible light feature map;

[0150] Step 3: Use the F-Attention module to fuse the extracted differential bimodal features, retaining the commonalities of the bimodal features while pursuing more differential information;

[0151] Step 4: Use RPN to regress and classify the fused dual-mode features to complete target detection.

[0152] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0156] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A method for infrared and visible light target detection based on differential feature perception in adversarial learning, characterized in that: The following steps are involved: Step 1: Based on the fusion divergence loss function composed of KL and JS divergence, a non-shared feature extraction network is used to extract differential infrared features and visible light features respectively; A fusion divergence loss function consisting of KL divergence and JS divergence is applied to measure the similarity between infrared features and visible light features; The similarity between infrared and visible light features is negatively correlated with the KL and JS divergence values; the KL divergence is expressed as: JS divergence is expressed as: The fusion divergence loss function defined based on KL divergence and JS divergence is expressed as: Where F1 and F2 are the features output by the last convolutional layer of each ResNet of the two backbone networks respectively; f1 and f2 represent the pixel intensity at each position in F1 and F2 respectively, which are calculated by the softmax function; N represents the number of features in F1 or F2, and the maximum value is N; Step 2: Use the IR-Attention module and RGB-Attention module to re-extract the infrared features and visible light features extracted in step 1, respectively, paying more attention to the effective information in the infrared feature map and the visible light feature map; The specific sub-steps of step 2 are as follows: First, input feature map Perform three linear operations respectively to obtain three feature maps of query, key and value, which are represented by E, F and G respectively. Next, reshape E into Reshape F into After transposing Get the channel attention matrix: Where T ji Indicates the influence of the i-th channel on the j-th channel. This correlation is positively correlated with the value of T. At the same time, G is reshaped into And perform matrix multiplication on T and G, reshaping the result into This is the sub-output feature map of the middle part; define this sub-output as Use global average pooling F sq (·) The original feature map Compressed to This operation compresses the size of the feature map while retaining the complete information of the feature map; M is expressed as: Where H and W are the height and width of the original feature map respectively; Next, M is sent to two fully connected layers to obtain the activation value N is represented by: N=F ex (M,ω)=σ(ω2δ(ω1M)) (11) Where σ(·) and δ(·) are the sigmoid function and ReLU function respectively; are the weights of the first and second fully connected layers respectively, r is the number of neurons in the hidden layers of the two fully connected layers, and r is set to 16; the sub-output feature map of this part Expressed as: Among them F scale (·,·) indicates the multiplication of corresponding elements; Finally, the two sub-output feature maps are obtained and Linearly combined with the original feature map F2, the total output feature map of the channel attention module is obtained The initial values ​​of α and β are both 0, and the weights are finally determined during the learning process to obtain the height feature map F 2,H and width feature map F 2,W Finally, the three feature maps are combined by pixel-by-pixel addition to obtain the output of RGB-Attention, which is the visible light feature F'2 with contextual connection: F'2=F 2,C +F 2,H +F 2,W (14) Step 3: Use the F-Attention module to fuse the extracted differential bimodal features, retaining the commonalities of the bimodal features while pursuing more differential information; Step 4: Use RPN to regress and classify the fused dual-mode features to complete target detection.

2. The infrared and visible light target detection method based on differential feature perception in adversarial learning according to claim 1 is characterized by: In step 2, the IR-Attention module performs attention mechanisms in two directions, including: a channel attention module and a spatial attention module; In the channel attention module, the connection between channel direction information is used to generate the corresponding weights; first, global average pooling is used to pool the input Squeeze to obtain Then, F1 c Feed it into two fully connected layers to obtain the weight vector; finally, reshape the weight vector into R C×H×W ; The channel attention module is expressed as: M c =Reshape(FC1(FC0(GAP(F1)))) (4) In the formula, FC is the fully connected layer, GAP is the global average pooling operation; In the spatial attention module, the spatial attention module is expressed as: M s =Reshape(Conv1(Conv3(DConv(GAP(F1))))) (5) Where, Conv i is a convolutional layer with a kernel of i×i, and DConv is a dilated convolution; after obtaining the weights of the channel attention module and the spatial attention module respectively; Combining the channel attention module and the spatial attention module, we get the final weight M(F1) of the input feature F1, which is expressed as: M(F1)=σ(M c +M s ) (6) Where σ is the sigmoid function, and the infrared feature map processed by the IR-Attention module is expressed as: F1′=F1+F1*M(F1) (7).

3. The infrared and visible light target detection method based on differential feature perception in adversarial learning according to claim 2, characterized in that: In step 2, the RGB-Attention module is used to generate contextually relevant visible light features. The visible light features F2 are then fed into three self-attention modules: the channel attention module, the height attention module, and the width attention module. These modules obtain features that are correlated in the channel, height, and width dimensions, respectively. The channel attention module is described in detail. The height attention module and the width attention module have similar structures to the channel attention module.

4. The infrared and visible light target detection method based on differential feature perception in adversarial learning according to claim 3 is characterized by: In step 3, F-Attention is used to fuse the differential infrared features and visible light features obtained in step 2; Use F-Attention to fuse different infrared and visible light features. The formula is as follows: F=F1'*Conv1(Conv1(F'))+F'2Conv1(Conv1(F'))+Conv1(Conv1(F)') (15) Finally, the tensor output by F-Attention is reconstructed to obtain the final fusion feature.

5. The infrared and visible light target detection method based on differential feature perception in adversarial learning according to claim 4 is characterized in that: In step 4, RPN is used to regress the prediction box and classify the target; During the training process, multiple loss functions are used to assist the model in error analysis, namely, the loss L based on the fusion divergence div , classification loss and regression loss; the total loss is the weighted sum of the three losses, expressed as: Among them, L cls and L reg are the loss functions for classification and regression respectively; p i and are the predicted probability of the object contained by the i-th anchor and its true value; t i Contains the four position parameters of the i-th bounding box; Contains the four position parameters of the ground truth box corresponding to the i-th bounding box; N cls and N reg are the mini-batch size and the number of anchors respectively; λ is the weight of the regression loss function.

6. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the infrared and visible light target detection method based on differential feature perception in adversarial learning according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements the infrared and visible light target detection method based on differential feature perception in adversarial learning as described in any one of claims 1 to 5.

Images