Target detection method and system based on infrared-visible light information fusion

Abstract

The application provides a target detection method based on infrared-visible light information fusion, visible light and infrared features are trained through the ISF-Net network, common features and difference features of two modes can be effectively learned, a feature separation loss is used to supervise the training process, divergence in the common features of two modes is minimized, divergence in the difference features is maximized, then the common features and the difference features are enhanced by using a channel attention branch and a spatial attention branch, difference between the common features and the difference features is highlighted, feature expression capability of the fusion features is effectively improved, and the model has high detection precision. The application also provides a target detection system based on infrared-visible light information fusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to a target detection method and system based on infrared-visible light information fusion. BACKGROUND

[0002] Two-dimensional target detection based on vision is widely used in automatic driving, remote sensing, video monitoring and other fields. However, due to the insufficient capacity of visible light sensors, the performance of the vision-based method usually declines sharply in extreme cases such as insufficient light and object occlusion, and is sensitive to environmental factors such as rain, fog and weak light. Therefore, in recent years, researchers have begun to integrate multi-modal information of different sensors to enhance the detection capability, among which infrared sensors have shown excellent performance due to their good adaptability to harsh environments and good complementarity with visible light sensors.

[0003] The different modal information obtained by the visible light sensor and the infrared sensor needs to be fused, and the fusion strategy can be divided into three paradigms: pixel-level fusion, decision-level fusion and feature-level fusion. Pixel-level fusion fuses two original images of visible light and infrared sensors together to generate a fused image for target detection, while decision-level fusion considers fusing two detection results generated on two single modal features. Since the inter-modal correlation is not explored in depth, these two paradigms can only achieve limited performance improvement. In comparison, feature-level fusion uses a multi-modal feature fusion module to deeply explore the complex correlation between modalities, thereby obtaining better performance. However, due to the insufficient capacity of the sensor, there is a lot of incomplete information in the extracted visual and infrared features, which will significantly affect the subsequent feature fusion, and the commonality and specificity between multi-modal images are not fully utilized, resulting in unsatisfactory results.

[0004] Therefore, it is necessary to provide a target detection method and system based on infrared-visible light information fusion to solve the above problems. SUMMARY

[0005] The application provides a target detection method and system based on infrared-visible light information fusion, which constructs an ISF-Net network, the ISF-Net network includes a feature interaction module, a feature separation loss module and a multi-level feature fusion module, the feature interaction module enhances the feature representation by combining the complementary information between visible light and infrared images, the feature separation loss module promotes the approximation of common features of two modalities and the separation of difference features between modalities, and the multi-level feature fusion module is used for fusion of common features and difference features. The application makes full use of the commonality and specificity of visible light and infrared features, and effectively improves the performance of the detector.

[0006] To solve the above technical problems, the application is implemented as follows:

[0007] The application provides a target detection method based on infrared-visible light information fusion, comprising the following steps:

[0008] S1: constructing a data set comprising visible light image-infrared image pairs;

[0009] S2: constructing a feature extraction network comprising two branches with the same structure, inputting the visible light-infrared image pair into the feature extraction network, and extracting visible light features F rgb and infrared features F ir from the two branches respectively;

[0010] S3: constructing an ISF-Net network, inputting the visible light features F rgb and infrared features F ir into the ISF-Net network for training, wherein the ISF-Net network comprises a feature interaction module and a multi-level feature fusion module, the visible light features F rgb and infrared features F ir perform information interaction in the feature interaction module to generate common features and difference features, and then the generated common features and difference features are input into the multi-level feature fusion module for element-level feature fusion to output final features, and a loss function is used to supervise the training process to promote the approximation of the common features of the two modalities and the separation of the difference features between the modalities;

[0011] S4: inputting the final output features into a detection head to obtain a detection result.

[0012] Preferably, the feature extraction network is a CSPDarknet53 feature extraction network in YOLO v5.

[0013] Preferably, the feature interaction module comprises a first block and a second block connected in sequence, the first block is provided with a CM-MSA module, the second block is provided with a DI-MSA module, and the execution process of the feature interaction module comprises the following steps:

[0014] S311: using a sliding window to divide and recombine the input visible light features F rgb and infrared features F ir into a plurality of non-overlapping local window features X rgb and X ir respectively;

[0015] S312: inputting X rgb , X ir into the first block, and using the CM-MSA module to divide and recombine X rgb , X irX is mapped to the corresponding query space Q, key space K and value space V, respectively, and the attention score is calculated after the query Q, key K and value of X are interchanged rgb The query Q rgb , key K rgb and value of X ir The query Q ir , key K ir are interchanged, and the attention score is calculated, focusing on the interaction of the complementary information of the visible light modality and the infrared modality, outputting enhanced visible light features Z rgb and infrared features Z ir ;

[0016] S313: input Z rgb , Z ir into the second block, and use the DI-MSA module to map Z rgn to the query space Q rgb1 , key space K rgb1 and value space V rgb1 , map Z ir to the query space Q ir1 , key space K ir1 and value space V ir1 , and calculate the attention score after interchanging the Q rgb , K rgb1 of Z rgb1 and the Q ir , K ir1 of Z ir1 , focusing on the common area of the visible light modality and the infrared modality under the guidance of the same position encoding, outputting visible light features T rgb and infrared features T ir containing unique information of each modality.

[0017] Preferably, the processing process of the CM-MSA module includes a mapping process and an interactive self-attention process, and the mapping process is represented as:

[0018]

[0019]

[0020] In the formula, all represent learnable weight matrices.

[0021] The interactive self-attention process is represented as:

[0022]

[0023]

[0024] In the formula, d k represents the dimension of the key K, K rgb and K irare the same dimension; δ represents a softmax function; and T represents a matrix transposition operation.

[0025] Preferably, the processing procedure of the DI-MSA module comprises a mapping procedure and an interactive self-attention procedure, and the mapping procedure is represented as:

[0026]

[0027]

[0028] wherein, all represent learnable weight matrices;

[0029] The interactive self-attention procedure is represented as:

[0030]

[0031]

[0032] wherein, represents a learnable relative position encoding.

[0033] Preferably, the loss function is represented as:

[0034]

[0035] wherein, represents a total loss, represents a feature interaction loss; represents a bounding box regression loss;

[0036] represents an object classification loss; represents an object confidence loss;

[0037] wherein:

[0038]

[0039] wherein, D JS (·) represents a JS divergence calculation; j represents the jth layer of features in the three layers of features; and a represents an interval for accelerating loss convergence.

[0040]

[0041]

[0042] wherein, D KL (·) represents a KL (Kullback-Leibler) divergence calculation.

[0043] Preferably, the multi-level feature fusion module comprises a channel attention branch and a spatial attention branch arranged in parallel, and the common features and the difference features are enhanced by using the channel attention branch and the spatial attention branch.

[0044] Preferably, the processing process of the channel attention branch comprises the following steps:

[0045] S321: Strengthen the common features and the difference features of the visible light features and the infrared features by using the addition and subtraction operation, wherein the process of the addition and subtraction operation is represented as:

[0046] C = T rgb + T ir , D rgb = F rgb -T ir , D ir = F ir -T rgb ;

[0047] In the formula, C represents the enhanced common features of the visible light features and the infrared features, D rgb represents the enhanced difference features of the visible light features and the infrared features, and D ir represents the enhanced difference features of the infrared features and the visible light features.

[0048] S322: Use the average pool and the maximum pool operation to aggregate the spatial information of the feature map, generate two different spatial context descriptors, integrate and forward the two descriptors to the multi-layer perceptron network, and generate the channel attention map wherein, represents the channel attention map obtained by enhancing the common features; and are the channel attention maps obtained by the visible light and infrared filtered disparity features, respectively;

[0049] S323: Ensure the distinction of the two channel attention maps by using the concat and softmax operations, which is represented as:

[0050]

[0051]

[0052] In the formula, σ represents the sigmoid function, δ represents the softmax function, and [·] represents the concat operation. In addition, MLP(·) represents the multi-layer perceptron network operation, and the multi-layer perceptron network comprises one multi-layer perceptron and one hidden layer, and the activation size of the hidden layer is set to R C / r×1×1 , wherein r represents the reduction rate;

[0053] S324: for enhancing strong T rgb and T ir , and will be used to enhance F rgb and F ir , add enhanced common features and difference features, complete the feature enhancement of the channel level, and the process of enhancement is represented as:

[0054]

[0055]

[0056] wherein, ⊙ represents the multiplication of elements; CH rgb represents the channel level light enhancement feature; CH ir represents the channel level infrared enhancement feature.

[0057] Preferably, the processing process of the channel attention branch includes the following steps, and the processing process of the spatial attention branch includes the following steps:

[0058] S331: using the same processing method as the channel attention branch to obtain C, D rgb and D ir ;

[0059] S332: using a 1x1 convolution kernel to generate a two-dimensional map, aggregating the spatial information of the feature map, and the operation process is represented as:

[0060]

[0061]

[0062] wherein, wherein represents the spatial attention map obtained by enhancing the common features, and respectively represent the spatial attention maps obtained by the visible light and infrared filtered parallax features;

[0063] S333: using for enhancing strong T rgb and T ir , and will be used to enhance F rgb and F ir , add enhanced common features and difference features, complete the feature enhancement of the channel level, and the process of enhancement is represented as:

[0064]

[0065]

[0066] SP rgb represents the spatial level visible light enhanced feature; SP ir represents the spatial level infrared enhanced feature.

[0067] The channel level enhanced feature and the spatial level enhanced feature are summed based on elements to obtain a final output feature, denoted as:

[0068] W = CH rgb + CH ir + SP rgb + SP ir ;

[0069] In the formula, W represents the final output feature.

[0070] The application also provides a target detection system based on infrared-visible light information fusion, comprising:

[0071] A data set: the data set comprises a visible light image-infrared image pair.

[0072] A feature extraction network: the feature extraction network comprises two branches with the same structure, the visible light-infrared image pair is input into the feature extraction network, and the visible light feature F rgb and the infrared feature F ir are extracted by the two branches respectively.

[0073] An ISF-Net network: the ISF-Net network comprises a feature interaction module and a multi-level feature fusion module, the visible light feature F rgb and the infrared feature F ir perform information interaction in the feature interaction module to generate common features and difference features, and then the generated common features and difference features are sent into the multi-level feature fusion module to perform element level feature fusion, output a final feature, and use a loss function to supervise the training process to promote the approximation of the common features of the two modalities and the separation of the difference features between the modalities.

[0074] A detection head: receiving the final output feature to obtain a detection result.

[0075] The application has the following beneficial effects:

[0076] Through the ISF-Net network set, the visible light and infrared features are trained, the common features and difference features of the two modalities can be effectively learned, the feature separation loss is used to supervise the training process, the divergence in the common features of the two modalities is minimized, the divergence in the difference features is maximized, then the common features and the difference features are enhanced by using the channel attention branch and the spatial attention branch, the difference between the common features and the difference features is highlighted, the feature expression ability of the fused features is effectively improved, and the model has high detection precision. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 A flowchart of a target detection method based on infrared-visible light information fusion provided by the present application is shown;

[0078] Figure 2 A structural diagram of an ISF-Net network provided by the present application is shown;

[0079] Figure 3 A structural diagram of a feature interaction module provided by the present application is shown;

[0080] Figure 4 A use effect diagram of the feature interaction module provided by the present application is shown;

[0081] Figure 5 A structural diagram of a multi-level feature fusion module provided by the present application is shown;

[0082] Figure 6 A test result diagram of the target detection method based on infrared-visible light information fusion provided by the present application on a VEDAI dataset is shown;

[0083] Figure 7 A test result diagram of the target detection method based on infrared-visible light information fusion provided by the present application on a FLIR dataset is shown;

[0084] Figure 8 A test result diagram of the target detection method based on infrared-visible light information fusion provided by the present application on a LLVIP dataset is shown. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0086] Please refer to Figures 1-8The application provides a target detection method based on infrared-visible light information fusion, comprising the following steps:

[0087] S1: constructing a data set comprising visible light image-infrared image pairs.

[0088] S2: constructing a feature extraction network comprising two branches with the same structure, inputting the visible light-infrared image pair into the feature extraction network, and extracting visible light features F rgb and infrared features F ir from the two branches respectively.

[0089] S3: constructing an ISF-Net network, inputting the visible light features F rgb and infrared features F ir into the ISF-Net network for training, wherein the ISF-Net network comprises a feature interaction module and a multi-level feature fusion module, the visible light features F rgb and infrared features F ir perform information interaction in the feature interaction module to generate common features and difference features, and then the generated common features and difference features are input into the multi-level feature fusion module for element-level feature fusion to output final features, and a loss function is used to supervise the training process to promote the approximation of the common features of the two modalities and the separation of the difference features between the modalities.

[0090] S4: inputting the final output features into a detection head to obtain a detection result.

[0091] In the data set, the visible light image is represented as X rgb ∈R H×W×C , and the infrared image is represented as X ir ∈R H×W×C , wherein R represents a three-dimensional matrix, W represents the width, H represents the height, and C represents the number of channels.

[0092] The feature extraction network is a CSPDarknet53 feature extraction network in YOLO v5, which belongs to a conventional network architecture in the field, and the feature extraction process also adopts conventional technologies in the field, and thus will not be described here. Wherein, F rgb ∈R H×W×C ; F ir ∈R H×W×C .

[0093] The feature interaction module comprises a first block and a second block connected in sequence, the first block is provided with a CM-MSA module (cross-modal multi-head self-attention module), the second block is provided with a DI-MSA module (detailed information interaction multi-head self-attention module), and the execution process of the feature interaction module comprises the following steps:

[0094] S311: Using a sliding window, the input light characteristics F can be displayed. rgb and infrared signature F ir The features X are divided and recombined into multiple non-overlapping local windows. rgb and X ir .

[0095] The window size is M×M, and the recombined feature X rgb X ir All dimensions are: The quantity is:

[0096] S312: X rgb X ir Input the first block, and use the CM-MSA module to input X rgb X ir Mapping X to the corresponding query space Q, key space K, and value space V respectively, rgb Query Q rgb Key K rgb With X ir Query Q ir Key K ir After the swap, the attention score is calculated, focusing on the interaction of complementary information between the visible light mode and the infrared mode, and the enhanced visible light feature Z is output. rgb and infrared signature Z ir .

[0097] The processing procedures for visible light features and infrared features in the first block are the same. Taking visible light features as an example, the execution process of step S312 is as follows:

[0098]

[0099]

[0100] In the formula, LN(·) represents the intermediate processing characteristics of the first block; LN(·) represents normalization processing; CM-MSA(·) represents the interactive operation of the CM-MSA module; Z rgb The output features of the first block are represented by MLP(·). MLP(·) represents multilayer perceptron processing.

[0101] The processing of the CM-MSA module includes a mapping process and an interactive self-attention process. The mapping process is expressed as follows:

[0102]

[0103]

[0104] wherein, both represent learnable weight matrices.

[0105] The interaction self-attention process is represented as:

[0106]

[0107]

[0108] wherein, d k represents the dimension of the key K, K rgb and K ir have the same dimension; δ represents the softmax function; and T represents the matrix transposition operation.

[0109] The query Q rgb , the key K rgb and the value V ir of the visible light feature are interchanged with the query Q ir , the key K rgb and the value V ir of the infrared feature, so that the visible light mode and the infrared mode interact with each other in the respective unique information attention areas.

[0110] S313: input Z rgb , Z ir into the second block, map Z rgn to the query space Q rgb1 , the key space K rgb1 and the value space V rgb1 using the DI-MSA module, map Z ir to the query space Q ir1 , the key space K ir1 and the value space V ir1 , and calculate the attention score after interchanging Q rgb1 , K rgb1 of Z ir and Q ir1 , K ir1 of Z rgb , so as to focus on the common area of the visible light mode and the infrared mode under the guidance of the same position coding, and output the visible light feature T ir and the infrared feature T rgb containing unique information of each mode.

[0111] The processing process of the visible light feature and the infrared feature in the second block is the same. Taking the visible light feature as an example, the execution process of the step S313 is represented as:

[0112]

[0113]

[0114] wherein, represents the intermediate processing feature of the second block; DI-MSA(·) represents the interaction operation of the DI-MSA module; T rgb represents the output feature of the second block.

[0115] The processing process of the DI-MSA module includes a mapping process and an interaction self-attention process, and the mapping process is represented as:

[0116]

[0117]

[0118] In the formula, all represent learnable weight matrices.

[0119] The interaction self-attention process is represented as:

[0120]

[0121]

[0122] In the formula, represents a learnable relative position encoding.

[0123] The loss function is represented as:

[0124]

[0125] In the formula, represents the total loss, represents the feature interaction loss; represents the bounding box regression loss;

[0126] represents the object classification loss; represents the object confidence loss; wherein, is the complete IoU loss, and are both binary cross-entropy losses;

[0127] wherein:

[0128]

[0129] In the formula, D JS (·) represents the JS divergence calculation; j represents the jth layer feature in the three-layer feature; and α represents an interval for accelerating loss convergence.

[0130]

[0131]

[0132] In the formula, D KL (·) indicates the calculation of KL (Kullback-Leibler) divergence.

[0133] RGB and IR images possess both unique and shared features in object detection. During training, to significantly distinguish between the common and dissimilar features of RGB and IR modalities, it is desirable to maximize the distributional similarity between common features and minimize the distributional similarity between dissimilar features. Therefore, it is necessary to minimize D. JS (T rgb |T ir ) and maximizing D JS (F rgb |F ir ). In loss Driven by this, the commonalities between features in each layer become smaller and the differences become larger, thus enabling a sufficiently accurate representation of the commonalities and differences between RGB and IR features.

[0134] The multi-level feature fusion module includes a channel attention branch and a spatial attention branch, which are arranged in parallel. The common and dissimilar features are enhanced using these two branches. Preferably, the channel attention branch and the spatial attention branch are arranged in parallel.

[0135] The processing of the channel attention branch includes the following steps:

[0136] S321: Enhance the common and differential features of visible light and infrared features through addition and subtraction operations. The process of addition and subtraction operations is represented as follows:

[0137] C = T rgb +T ir D rgb =F rgb -T ir D ir =F ir -T rgb ;

[0138] In the formula, C represents the enhanced common feature that can combine optical and infrared features, and D... rgb This indicates the enhancement of the difference between optical and infrared features; D ir This indicates the difference between enhanced infrared features and optical features.

[0139] When performing subtraction, a cross subtraction method is used to ensure that there is some information exchange between modes, thereby filtering out noise in the single-modal information.

[0140] S322: using average pooling and max pooling operations to aggregate the spatial information of the feature map, generating two different spatial context descriptors, integrating and forwarding the two descriptor sets to a multi-layer perceptron network to generate a channel attention map wherein, represents the channel attention map obtained by enhancing the common features; and respectively represent the channel attention maps obtained by visible light and infrared filtered disparity features;

[0141] S323: ensure the distinction of the two channel attention maps through concat and softmax operations, denoted as:

[0142]

[0143]

[0144] In the formula, σ represents the sigmoid function; δ represents the softmax function; [·] represents the concat operation. In addition, MLP(·) represents a multi-layer perceptron network operation, which includes a multi-layer perceptron and a hidden layer, and the activation size of the hidden layer is set to R C / r×1×1 , wherein r represents the reduction rate;

[0145] S324: the is used to enhance the strong T rgb and T ir , and will be used to enhance F rgb and F ir , add enhanced common features and difference features, complete the feature enhancement at the channel level, and the enhancement process is represented as:

[0146]

[0147]

[0148] In the formula, ⊙ represents the multiplication of elements; CH rgb represents the channel-level visible light enhanced feature; CH ir represents the channel-level infrared enhanced feature.

[0149] The processing process of the spatial attention branch includes the following steps:

[0150] S331: obtain C, D rgb and D ir in the same way as the channel attention branch;

[0151] S332: a two-dimensional map is generated using a 1*1 convolution kernel, the spatial information of the feature map is aggregated, and the operation process is represented as:

[0152]

[0153]

[0154] wherein, wherein represents a spatial attention map obtained by enhancing the common feature, and respectively represent spatial attention maps obtained by visible light and infrared filtered parallax features;

[0155] S333: the is used to enhance the strong T rgb and T ir , and will be used to enhance the F rgb and F ir , add the enhanced common feature and the difference feature, complete the spatial level feature enhancement, and the enhanced process is represented as:

[0156]

[0157]

[0158] wherein, SP rgb represents the spatial level visible light enhanced feature; SP ir represents the spatial level infrared enhanced feature.

[0159] The channel level enhanced feature and the spatial level enhanced feature are summed based on elements to obtain the final output feature, represented as:

[0160] W=CH rgb +CH ir +SP rgb +SP ir ;

[0161] wherein, W represents the final output feature.

[0162] The application also provides a target detection system based on infrared-visible light information fusion, comprising:

[0163] Data set: the data set includes a visible light image-infrared image pair;

[0164] Feature extraction network: the feature extraction network includes two branches with the same structure, the visible-infrared image pair is input into the feature extraction network, and the visible feature F rgb and the infrared feature Fir ;

[0165] ISF-Net network: the ISF-Net network comprises a feature interaction module and a multi-level feature fusion module, the visible light feature F rgb and the infrared feature F ir In the feature interaction module, information interaction is performed to generate common features and difference features, and then the generated common features and difference features are sent to the multi-level feature fusion module for element-level feature fusion to output final features, and a loss function is used to supervise the training process to promote the approximation of the common features of the two modalities and the separation of the difference features between the modalities.

[0166] Detection head: receiving the final output features to obtain a detection result.

[0167] In order to verify the effectiveness of the IFS-Net network provided in the present application, an infrared feature extraction branch is added in YOLO v5, only using addition as the feature fusion method, a dual-channel detection network is constructed as a base detector, and a comparison test is performed between the base detector and the target detection system based on infrared-visible light information fusion provided in the present application. In order to ensure the fairness of the experiment, the same parameters are set for the two models during training. The device used for training is a 1080Ti desktop computer, and the SGD optimizer is used. The initial learning rate is 0.001. The momentum is 0.937, and the weight decay is 0.0005. The test data set includes VEDAI, FLIR and LLVIP three data sets.

[0168] Table 1 shows the performance test results of the ISFNet network of different components on the FLIR data set.

[0169] Table 1

[0170]

[0171]

[0172] In Table 1, in the network without using the feature interaction module, the convolution is used to replace the role of the feature interaction module to ensure the integrity of the network architecture. The feature interaction module has strong information interaction capability and can effectively utilize the complementary information between the visible-infrared modalities. As can be seen from the comparison in Table 1, in the case of using the feature interaction module, the evaluation index of mAP is increased by 0.5%; in the network using the feature separation loss L fs , the evaluation index of mAP is increased by 0.4%, indicating that the feature separation loss L fsThe proposed network makes a positive contribution; the multi-level feature fusion module distinguishes the different features and common features of the two modules, enhances the channel features and spatial features, and as can be seen from Table 1, the mAP evaluation index increases by 0.6% when the multi-level feature fusion module is used. In the case of using all components, the detection performance of the ISFNet network of the present application is improved by 5.3% compared with the basic detector, and the detection performance is also improved compared with the network using part of the components, which shows the effectiveness of each module in the ISFNet network provided by the present application and the necessity of module combination.

[0173] Table 2 shows the influence of the interval a in the feature separation loss on the performance of the detector, and the tested a is 0.05, 0.075 and 0.1 respectively. The feature separation loss proposed in this paper has good convergence regardless of different a. From the test results, it can be seen that when the boundary value is set to 0.05, the performance of the detector is the best.

[0174] Table 2

[0175] α 0.05 0.075 0.1 mAP 43.5％ 43.4％ 43.4％

[0176] Table 3 shows the influence of the arrangement of the channel attention branch and the spatial attention branch in the multi-level feature fusion module on the test results. Specifically, we tried three attention structures: "parallel arrangement", "spatial first channel" and "channel first space". Finally, it can be concluded that the "parallel arrangement" design has the best precision, because the parallel arrangement of the spatial branch and the channel branch can obtain more feature expression details, and the element addition provides better feature expression capability.

[0177] Table 3

[0178] arrangement mAP ISF-Net parallel 43.5％ ISF-Net space before channel 43.3％ ISF-Net channel before space 43.3％

[0179] Tables 4-6 show the detection results of the existing network and the ISF-Net network of the present application on three data sets.

[0180] Table 4

[0181]

[0182]

[0183] Table 5

[0184]

[0185] Table 6

[0186]

[0187]

[0188] From Figures 4-6 It can be seen that, due to the information integration of visible light (RGB) features and infrared (IR) features, the performance of the multi-modal detection method is better than that of the single-modal detection method; among all the multi-modal detection networks, the ISF-Net network provided in the application has the best accuracy.

[0189] The embodiments of the application are described above in conjunction with the drawings, but the application is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the application without departing from the scope of the application and the scope protected by the claims.

Claims

1. A target detection method based on infrared-visible light information fusion, characterized in that, Comprising the following steps: S1: constructing a data set, wherein the data set comprises visible light image-infrared image pairs; S2: a feature extraction network is constructed, the feature extraction network includes two branches with the same structure, a visible light-infrared image pair is input to the feature extraction network, and visible light features F rgb and infrared features F ir are respectively extracted by the two branches; S3: constructing an ISF-Net network, sending the visible light feature F rgb and the infrared feature F ir into the ISF-Net network for training, the ISF-Net network comprising a feature interaction module and a multi-level feature fusion module, the visible light feature F rgb and the infrared feature F ir performing information interaction in the feature interaction module to generate common features and difference features, then sending the generated common features and difference features into the multi-level feature fusion module for element-level feature fusion to output final features, and adopting a loss function to supervise the training process and promote the approximation of the common features of the two modalities and the separation of the difference features between the modalities; S4: feeding the final output feature into a detection head to obtain a detection result; The feature interaction module comprises a first block and a second block connected in sequence, the first block is provided with a CM-MSA module, and the second block is provided with a DI-MSA module. The multi-level feature fusion module comprises a channel attention branch and a spatial attention branch arranged in parallel, and the common feature and the difference feature are enhanced by using the channel attention branch and the spatial attention branch.

2. The target detection method based on infrared-visible light information fusion according to claim 1, characterized in that, The feature extraction network is a CSPDarknet53 feature extraction network in YOLO v5. 3.The target detection method based on infrared-visible light information fusion according to claim 1, characterized in that, The execution process of the feature interaction module comprises the following steps: S311: using a sliding window to divide the inputted optical feature F rgb and infrared feature F ir into multiple local window features X rgb and X ir respectively, which are non-overlapping. S312: X rgb X ir Input the first block, and use the CM-MSA module to input X rgb X ir Mapping X to the corresponding query space Q, key space K, and value space V respectively, rgb Query Q rgb Key K rgb With X ir Query Q ir Key K ir After the swap, the attention score is calculated, focusing on the interaction of complementary information between the visible light mode and the infrared mode, and the enhanced visible light feature Z is output. rgb and infrared signature Z ir ; S313: Z rgb Z ir Input the second block, and use the DI-MSA module to input Z rgb Mapped to query space Q rgb1 Key space K rgb1 Sum value space V rgb1 , will Z ir Mapped to query space Q ir1 Key space K ir1 Sum value space V ir1 , will Z rgb Q rgb1 K rgb1 With Z ir Q ir1 K ir1 After swapping, the attention score is calculated. Guided by encoding at the same location, attention is focused on the common region of the visible light mode and the infrared mode, and the output is a visible light feature T containing unique information for each mode. rgb and infrared features T ir .

4. The target detection method based on infrared-visible light information fusion according to claim 3, characterized in that, The processing process of the CM-MSA module comprises a mapping process and an interactive self-attention process, and the mapping process is represented as: wherein both represent learnable weight matrices; The interactive self-attention process is represented as: where d k represents the dimension of the key K, K rgb and K ir have the same dimension; δ represents a softmax function; and T represents a matrix transposition operation.

5. The target detection method based on infrared-visible light information fusion according to claim 3, characterized in that, The processing process of the DI-MSA module comprises a mapping process and an interactive self-attention process, and the mapping process is represented as: wherein both represent learnable weight matrices; The interactive self-attention process is represented as: In the formula, represents a learnable relative position encoding.

6. The target detection method based on infrared-visible light information fusion according to claim 3, characterized in that, The loss function is represented as: wherein denotes the total loss, denotes the feature interaction loss; denotes the bounding box regression loss; denotes the object classification loss; denotes the object confidence loss; Wherein: In the formula, D JS (·) indicates JS divergence calculation; j indicates the jth layer feature in the three-layer feature; and a indicates an interval for accelerating loss convergence. In the formula, D KL (·) indicates KL (Kullback-Leibler) divergence calculation.

7. The target detection method based on infrared-visible light information fusion according to claim 1, characterized in that, The processing process of the channel attention branch comprises the following steps: S321: strengthening the common feature and the difference feature of the visible light feature and the infrared feature by using addition and subtraction operation, wherein the process of addition and subtraction operation is represented as: C = T rgb + T ir , D rgb = F rgb - T ir , D ir = F ir - T rgb ; where C represents an enhanced common feature of the optical feature and the infrared feature, D rgb represents an enhanced difference feature of the optical feature and the infrared feature; and D ir represents an enhanced difference feature of the infrared feature and the optical feature. S322: use average pool and max pool operations to aggregate the spatial information of the feature map, generate two different spatial context descriptors, integrate and forward the two descriptors to a multi-layer perceptron network, and generate a channel attention map wherein, represents the channel attention map obtained by enhancing the common features; and respectively, are the channel attention maps obtained by the visible light and infrared filtered disparity features; S323: ensuring the distinction of the two channel attention maps by using concat and softmax operation, represented as: In the formula, sigma represents the sigmoid function, delta represents the softmax function, and [·] represents the concat operation; MLP(·) denotes a multi-layer perceptron network operation including one multi-layer perceptron and one hidden layer with activation size set to R C / r×1×1 where r denotes the reduction rate; S324: the for enhancing strong T rgb and T ir , and will be used to enhance F rgb and F ir , add enhanced common features and difference features, complete the feature enhancement of the channel level, and the enhanced process is represented as: wherein, represents the multiplication of elements; CH rgb represents the channel level that can enhance the light feature; CH ir represents the channel level that can enhance the infrared feature.

8. The target detection method based on infrared-visible light information fusion according to claim 7, characterized in that, The processing process of the spatial attention branch comprises the following steps: S331: Obtain C and D using the same processing manner as the channel attention branch rgb and D ir ; S332: generating a two-dimensional map using a 1x1 convolution kernel, and the spatial information operation process of the aggregated feature map is represented as: wherein denotes a spatial attention map obtained from enhanced common features, and denotes a spatial attention map obtained from visible light and infrared filtered parallax features, respectively. S333: the enhanced features of F for enhancing F rgb and F ir , and the enhanced features of T rgb and T ir , adding the common and different features of the enhanced features, completing the feature enhancement of the space level, and the process of the enhancement is represented as: wherein represents the visible light enhancement feature at the spatial level; represents the infrared enhancement feature at the spatial level; The channel-level enhanced feature and the spatial-level enhanced feature are summed based on elements to obtain the final output feature, represented as: In the formula, W represents the final output feature.

9. An infrared-visible light information fusion-based target detection system, characterized in that, Comprise: Data set: the data set comprises visible light image-infrared image pairs; The feature extraction network comprises two branches with the same structure, and visible light-infrared image pairs are input into the feature extraction network, and visible light features Frgb and infrared features F are extracted by the two branches respectively ir ; The ISF-Net network comprises a feature interaction module and a multi-level feature fusion module, the visible light feature F rgb and the infrared feature F ir Information interaction is performed in the feature interaction module to generate common features and difference features, and then the generated common features and difference features are sent to the multi-level feature fusion module for element-level feature fusion to output final features, and a loss function is used to supervise the training process to promote the approximation of the common features of the two modalities and the separation of the difference features between the modalities; Detection head: receiving the final output feature to obtain a detection result; The feature interaction module comprises a first block and a second block connected in sequence, the first block is provided with a CM-MSA module, and the second block is provided with a DI-MSA module; The multi-level feature fusion module comprises a channel attention branch and a spatial attention branch arranged in parallel, and the common feature and the difference feature are enhanced by using the channel attention branch and the spatial attention branch.

Images