Infrared-visible light fusion method, system and readable storage medium based on feature enhancement

Abstract

This invention discloses a feature-enhanced infrared-visible light fusion method. It uses a YOLOv5 feature extraction network with a dual-stream backbone to extract deep features from visible and infrared images, reducing bias toward a single modality through symmetrical complementary masks. To address the differences between visible and infrared images, a cross-feature enhancement module is added to the fusion module to improve intra-modal feature representation, and a long-range dependency fusion module is added to fuse enhanced features by correlating the positional encoding of multimodal features. This method improves the combined utilization of multimodal images and enhances detection effectiveness in complex scenes. The application also provides a feature-enhanced infrared-visible light fusion system and a readable storage medium.

Description

Technical Field

[0001] The present application belongs to the field of target detection technology, and specifically relates to an infrared-visible light fusion method, system and readable storage medium based on feature enhancement. Background Art

[0002] Object detection algorithms are widely used in fields such as autonomous driving, surveillance, and remote sensing. However, due to the limitations of visible light sensors, most object detection methods for visible light images cannot achieve satisfactory accuracy and are sensitive to adverse environmental factors such as rain, fog, and low light. In contrast, infrared sensors perform well in these harsh environments. However, infrared sensors are significantly affected by temperature. In brightly lit, high-temperature environments, visible light images contain rich texture and color information, while infrared images struggle to effectively distinguish foreground from background. Therefore, by fusing the complementary information from visible and infrared sensors, the accuracy, reliability, and robustness of detection algorithms can be further improved.

[0003] In the existing technology, infrared-visible multimodal target detection is mainly divided into traditional methods and deep learning methods. For traditional methods, features are extracted from visible and infrared images using the Histogram of Oriented Gradients (HOG), and the concatenated fused features are input into a support vector machine (SVM) to obtain detection results. However, the feature extraction capabilities of hand-designed operators are limited, making it difficult to obtain optimal feature extraction results. For deep learning methods, due to its powerful representation learning capabilities, deep learning has shown advantages in visible-infrared fusion target detection. Based on YOLO v4, it has designed four fusion detection strategies: image conversion fusion, early fusion, mid-term fusion, and late fusion. However, methods using convolutional neural networks (CNNs) are based on the non-global receptive field of the convolution operator, resulting in information fusion only in local areas. Although these methods have higher performance than single-modal detection methods, they generally lack long-range dependencies and do not fully utilize the complementarity between modalities, resulting in unsatisfactory detection results.

[0004] Therefore, it is necessary to provide an infrared-visible light fusion method, system and readable storage medium based on feature enhancement to solve the problems raised in the above background technology. Summary of the Invention

[0005] The purpose of this application is to provide an infrared-visible light fusion method, system and readable storage medium based on feature enhancement, which adopts the feature extraction network of YOLOv5 with a dual-stream backbone to extract deep features from visible light and infrared images, and reduce the deviation from a single modality through symmetrical complementary masks; in view of the differences between visible light and infrared images, a cross-feature enhancement module is added to the fusion module to improve the intra-modal feature representation, and a long-distance dependency fusion module is added to fuse the enhanced features by associating the position encoding of multimodal features, which can improve the joint utilization rate of multimodal images and the detection effect of complex scenes.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] An infrared-visible light fusion method based on feature enhancement includes the following steps:

[0008] Data acquisition: acquiring a multi-target data set and preprocessing the multi-target data set, wherein the multi-target data set includes a visible light image and an infrared image;

[0009] Feature extraction: A dual-stream backbone feature extraction network is constructed. The dual-stream backbone feature extraction network includes two branches with the same structure. The visible light image and infrared image are respectively fed into the two branches to extract deep features, thereby obtaining visible light features and infrared features.

[0010] Feature fusion: Construct a feature fusion network, which includes a cross-feature enhancement module and a long-distance dependency fusion module. The cross-feature enhancement module includes a channel attention branch and a spatial attention branch arranged in series. The visible light feature and the infrared feature are sent to the channel attention branch for enhancement, and the visible light feature before enhancement is added to the infrared feature after enhancement by the channel attention branch to obtain a first feature, and the infrared feature before enhancement is added to the visible light feature after enhancement by the channel attention branch to obtain a second feature; the first feature and the second feature are sent to the spatial attention branch for enhancement, and the first feature is added to the second feature after enhancement by the spatial attention branch to obtain the final output of the visible light feature, and the second feature is added to the first feature after enhancement by the spatial attention branch to obtain the final output of the infrared feature; the final outputs of the visible light feature and the infrared feature are sent to the long-distance dependency fusion module, and fused through the correlation of position encoding based on the Swin Transformer model.

[0011] Preferably, the preprocessing method is: using a data enhancement method of generating image masks from random regions to process the visible light image and infrared image in the multi-target dataset. The specific process is: dividing the image into 10×10 checkerboards according to the image size, and selecting two squares set to zero in each row with a probability of 30% as the image mask; then dividing the image mask into two complementary masks by row, one of which is used as the mask for the visible light image and the other as the mask for the infrared image. The mask generation process is expressed as follows:

[0012] Generate mask =RGB mask ∪IR mask ;

[0013] RGB mask |IR mask =1;

[0014] Where, EGB mask is a mask for visible light images, IR mask It is a mask for infrared images, Generate mask It is the total mask.

[0015] Preferably, the dual-stream backbone feature extraction network is a dual-stream backbone YOLOv5 feature extraction network, and the extracted visible light features are expressed as Infrared signature is expressed as Represents a three-dimensional matrix; W is the width; H is the height; C represents the number of channels.

[0016] Preferably, the enhancement process of the channel attention branch is: completely folding the input features in one direction q while maintaining high resolution in the orthogonal direction v of direction q. The operation process is expressed as:

[0017] W RGBq =σ1(F1(X RGB ));W RGBv =σ2(F2(X RGB ));

[0018] E IRq =σ1(F1(X IR ));W IRv =σ2(F2(X IR ));

[0019] Where W RGBq Represents the information of visible light characteristics in the q direction; W RGBv Represents the information of visible light features in the v direction; W IRqRepresents the information of infrared characteristics in the q direction; W IRv represents the information of infrared features in the v direction; σ1 and σ2 both represent tensor shaping operators; F1(·) and F2(·) both represent 1×1 convolution operations;

[0020] W RGBq and W IRq Represent the weights of visible light features and infrared features respectively, input to the Sofmtax function for classification, and output the weight distribution of visible light features and infrared features. The calculation process is expressed as:

[0021]

[0022] Where W RGBk Represents the weight of visible light features; W IRk Represents the weight of infrared features.

[0023] The information W RGBv Multiply by weight W RGBk , the information W IRv Multiply by weight W IRk , then perform a 1×1 convolution operation, use normalization to upgrade the channel dimension from C / 2 to C, and use the Sigmoid function to keep all parameters in the range of 0-1. The calculation process is expressed as:

[0024] W RGBz =Sigmoid(σ3(F3(W RGBv ×W RGBk )));

[0025] W IRz =sigmoid(σ3(F3(W IRv ×W IRk )));

[0026] Where W RGBz Represents the information of visible light features in the z direction; W IRz Indicates the information of infrared features in the z direction; “×” indicates matrix dot product operation; F z (·) represents a 1×1 convolution operation; σ3 represents a tensor reshaping operator;

[0027] Then X RGB and W RGBz Perform channel-level multiplication to obtain the feature W with less noise RGBln ; X IR and W IRz Perform channel-level multiplication to obtain the feature W with less noise IRln , the calculation process is expressed as:

[0028] W RGBln =XRGB ⊙W RGBz ;W IRln =X IR ⊙W IRz ;

[0029] The feature W RGBln With X BGB Add to perform recalibration enhancement and obtain the first feature A RGBch ; The feature W RGBln With X IR Add to perform recalibration enhancement and obtain the second feature A IRch , the calculation process is expressed as:

[0030] A RGBch =W IRln +X RGB ; A IRch =W RGBln +X IR .

[0031] Preferably, the enhancement process of the spatial attention branch is to completely collapse the input features in the direction q while maintaining high resolution in the direction v. The operation process is expressed as:

[0032] A RGBq =σ4(F GP (F4(A RGBch )));A RGBv =σ5(F5(A RGBch ));

[0033] A IRq =σ4(F GP (F4(A IRch )));A IRv =σ5(F5(A IRch ));

[0034] Where σ4 and σ5 both represent tensor reshaping operators; F4(·) and F5(·) both represent 1×1 convolution operations; F GP (·) represents the global aggregation operator,

[0035] A RGBq and A IRq Represent the weights of the first and second features respectively, input to the Softmax function for classification, and output the weight distribution of the first and second features. The calculation process is expressed as follows:

[0036]

[0037] Where A RGBk represents the weight of the first feature, A IRkrepresents the weight of the second feature;

[0038] Information A RGBv Multiply by weight A RGBk , information A IRv Multiply by weight A IRk , then perform information completion, reshaping and Sigmoid function in turn:

[0039] A RGBz =Sigmoid(σ6(A RGBv ×A RFBk ));

[0040] A IRz =Sigmoid(σ6(A IRv ×A IRk ));

[0041] Where A RGBz ∈R 1×HW ; A IRz ∈R 1×HW The space gates representing the first and second characteristics respectively;

[0042] A RGBch and A RGBz Multiply A iRch and A IRz Multiply them to obtain the spatial enhancement features A of the first and second features respectively RGBln and A IRln , the calculation process is expressed as:

[0043] A RGBln =A RGBch ⊙A RGBz ; A IRln =A IRch ⊙A IRz ;

[0044] Feature A IRln With A RGBch Add to perform recalibration enhancement and obtain the final output X of the visible light feature RGBout ; Feature A RGBln With A IRch Add to perform recalibration enhancement and obtain the final output X of infrared features IRout , the calculation process is expressed as:

[0045] X RGBout =A IRln +A RGBch ;X IRout =A RGBln +A IRch .

[0046] Preferably, the feature fusion process is:

[0047] Using the shift window partitioning method, the feature map is alternately divided into M×M dimensions. If the feature map size is smaller than M×M, it will be padded to the size of M×M; then, the window of the next module will be relatively offset by (M / 2, M / 2) pixels; using this calculation method, the calculation formula is as follows:

[0048]

[0049]

[0050]

[0051]

[0052] Where, F i It represents the joint input of visible light and infrared features, F i ={X RGBout ,X IRout};F o is the output feature of the transformer block; and is the intermediate variable; W-MSA and MW-MSA are the window multi-head self-attention operation and the masked window multi-head self-attention operation respectively;

[0053] Considering the partial features after window segmentation, in the process of self-attention calculation, given the input visible light feature map F RGB ∈R 8×8×C and infrared characteristic map F IR ∈R 8×8×C ; Flatten each feature map and arrange the order of the matrix to obtain sentence I RGB ∈R 64×C and I IR ∈R 64×C ; Then provide the input sentence I∈R 128×C Connecting Sentences I RGB and I IR ; Project the input sentence I into three weight matrices to obtain a set of queries Q, keys K, and values V:

[0054] Q=IW Q ,K=IW K ,V=IW V ;

[0055] Where W Q ∈R C×128 、W K ∈R C×128 and W V ∈RC×128 Both represent weight matrices.

[0056] The self-attention calculation process is as follows:

[0057]

[0058] Where d represents the dimension of the query Q or key K; FPRE represents the position encoding of visible light and infrared features. FPRE contains four types of position information: visible light position information RPE RGB , infrared position information RPE IR , visible light and infrared relative position information RPE RGB- , infrared and visible light relative position information RPE iR-RGB ;

[0059]

[0060] Where, T represents the matrix transpose operation;

[0061] After the above calculations, the visible light feature X output after deep interaction is obtained. RGBout and infrared signature X IRout , and add the two together to get the final fusion feature F fusion :

[0062] F fusion =X RGBout +X IRout .

[0063] The present application also provides an infrared-visible light fusion system based on feature enhancement, comprising:

[0064] Data acquisition module: used to acquire multi-target data sets and pre-process the multi-target data sets, wherein the multi-target data sets include visible light images and infrared images.

[0065] Feature extraction module: used to construct a dual-stream backbone feature extraction network. The dual-stream backbone feature extraction network includes two branches with the same structure. The visible light image and infrared image are respectively sent to the two branches to extract deep features to obtain visible light features and infrared features.

[0066] And a feature fusion module: used to construct a feature fusion network, the feature fusion network includes a cross-feature enhancement module and a long-distance dependency fusion module, the cross-feature enhancement module includes a channel attention branch and a spatial attention branch arranged in series, the visible light feature and the infrared feature are sent to the channel attention branch for enhancement, the visible light feature before enhancement is added to the infrared feature after enhancement by the channel attention branch to obtain a first feature, and the infrared feature before enhancement is added to the visible light feature after enhancement by the channel attention branch to obtain a second feature; the first feature and the second feature are sent to the spatial attention branch for enhancement, the first feature is added to the second feature after enhancement by the spatial attention branch to obtain the final output of the visible light feature, and the second feature is added to the first feature after enhancement by the spatial attention branch to obtain the final output of the infrared feature; the final outputs of the visible light feature and the infrared feature are sent to the long-distance dependency fusion module, and fused through the correlation of position encoding based on the Swin Transformer model.

[0067] The present application also provides a readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the above-mentioned feature enhancement-based infrared-visible light fusion method.

[0068] The beneficial effects of this application are:

[0069] (1) Noise data is introduced during data preprocessing to force the network to learn complementary pattern information to reduce the network's bias towards a single pattern;

[0070] (2) Using the cross-feature enhancement module, feature enhancement is performed from both channel and spatial perspectives, including the exchange of complementary information between the two modes, to better integrate multimodal features, effectively overcome the difference between the two modes, the problem of severe lack of visible information, and improve the detection performance of the network;

[0071] (3) A long-distance dependency fusion module is used, focusing on deep interactive information enhancement. The features of the two modes are synchronously segmented by moving the shift window, and the deep interactive information enhancement of the fused features is performed through the multi-head self-retention mechanism, which improves the adaptability of the fused features in complex lighting scenes and reduces the missed detection and false detection of the detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A flowchart showing the infrared-visible light fusion method based on feature enhancement provided by the present application;

[0073] Figure 2 Schematic diagram showing the preprocessing of multi-target datasets;

[0074] Figure 3 Represents the architecture of the cross-feature enhancement module;

[0075] Figure 4 Represents the architecture of the long-range dependency fusion module;

[0076] Figure 5 Flowchart showing long-range attention fusion;

[0077] Figure 6 A schematic diagram showing the test results of the model of the present application in Example 1. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0079] Please refer to Figure 1-6 The present invention provides an infrared-visible light fusion method based on feature enhancement, comprising the following steps:

[0080] Data acquisition: collecting a multi-target data set and preprocessing the multi-target data set, wherein the multi-target data set includes a visible light image and an infrared image.

[0081] The preprocessing method is to use a data augmentation method to generate image masks using random regions to process the visible light images and infrared images in the multi-target dataset. The specific process is as follows: the image is divided into 10×10 checkerboard grids according to image size, and two squares are set to zero in each row with a probability of 30% to generate an image mask. The image mask is then split into two complementary masks by row, one of which serves as the mask for the visible light image and the other as the mask for the infrared image. The mask generation process is expressed as follows:

[0082] Generate mask =RGB mask ∪IR mask ;

[0083] RGB mask |IR mask =1;

[0084] Where, RGB mask is a mask for visible light images, IR mask It is a mask for infrared images, Generate maskIt is the total mask.

[0085] Feature extraction: A dual-stream backbone feature extraction network is constructed. The dual-stream backbone feature extraction network includes two branches with the same structure. The visible light image and infrared image are respectively sent to the two branches to extract deep features to obtain visible light features and infrared features.

[0086] The dual-stream backbone feature extraction network is redesigned based on the YOLOv5 feature extraction network as a dual-stream backbone. The extracted visible light features are expressed as Infrared signature is expressed as Represents a three-dimensional matrix; W is the width; H is the height; C represents the number of channels.

[0087] Feature fusion: Construct a feature fusion network, which includes a cross-feature enhancement module and a long-distance dependency fusion module. The cross-feature enhancement module includes a channel attention branch and a spatial attention branch arranged in series. The visible light feature and the infrared feature are sent to the channel attention branch for enhancement, and the visible light feature before enhancement is added to the infrared feature after enhancement by the channel attention branch to obtain a first feature, and the infrared feature before enhancement is added to the visible light feature after enhancement by the channel attention branch to obtain a second feature; the first feature and the second feature are sent to the spatial attention branch for enhancement, and the first feature is added to the second feature after enhancement by the spatial attention branch to obtain the final output of the visible light feature, and the second feature is added to the first feature after enhancement by the spatial attention branch to obtain the final output of the infrared feature; the final outputs of the visible light feature and the infrared feature are sent to the long-distance dependency fusion module, and fused through the correlation of position encoding based on the Swin Transformer model.

[0088] The enhancement process of the channel attention branch is to completely fold the input features in one direction q while maintaining high resolution in the orthogonal direction v of direction q. The operation process is expressed as:

[0089] W RGBq =σ1(F1(X RGB ));W RGBv =σ2(F2(X RGB ));

[0090] W IRq =σ1(F1(X IR ));W IRv =σ2(F2(X IR ));

[0091] Where W RGBqRepresents the information of visible light characteristics in the q direction; W RGBv Represents the information of visible light features in the v direction; W IRq Represents the information of infrared characteristics in the q direction; W IRv represents the information of infrared features in the v direction; σ1 and σ2 both represent tensor shaping operators; F1(·) and F2(·) both represent 1×1 convolution operations;

[0092] W RGBq and W IRq Represent the weights of visible light features and infrared features respectively, input to the Softmax function for classification, and output the weight distribution of visible light features and infrared features. The calculation process is expressed as:

[0093]

[0094] Where W RGBk Represents the weight of visible light features; W IRk Represents the weight of infrared features.

[0095] In the calculation process of weight distribution, information is severely compressed. In order to maintain the information strength, the information W RGBv Multiply by weight W RGBk , the information W IRv Multiply by weight W IRk , then perform a 1×1 convolution operation, use normalization to upgrade the channel dimension from C / 2 to C, and use the Sigmoid function to keep all parameters in the range of 0-1. The calculation process is expressed as:

[0096] W RGBz =Sigmoid(σ3(F3(W RGBv ×W RGBk )));

[0097] W IRz =Sigmoid(σ3(F3(W IRv ×W IRk )));

[0098] Where W RGBz Represents the information of visible light features in the z direction; W IRz Represents the information of infrared features in the z direction; “×” represents the matrix dot product operation; F3(·) represents the 1×1 convolution operation; σ3 represents the tensor reshaping operator;

[0099] Through the above operations, the visible appearance and geometric features with the most information in the intra-modal representation can be utilized to effectively suppress the feature noise in the inter-modal representation.

[0100] Then X RGB and W RGBz Perform channel-level multiplication to obtain the feature W with less noise RGBln ; X IR and W IRz Perform channel-level multiplication to obtain the feature W with less noise IRln , the calculation process is expressed as: W RGBln =X RGB ⊙W RGBz ;W IRln =X IR ⊙W IRz ;

[0101] Where, ⊙ represents the Hadamard product operation.

[0102] The feature W RGBln With X RGB Add to perform recalibration enhancement and obtain the first feature A RGBch ; The feature W RGBln With X IR Add to perform recalibration enhancement and obtain the second feature A IRch , the calculation process is expressed as:

[0103] A RGBch =W IRln +X RGB ; A IRch =W RGBln +X IR .

[0104] The enhancement process of the spatial attention branch is to completely collapse the input features in direction q while maintaining high resolution in direction v. The operation process is expressed as:

[0105] A RGBq =σ4(F GP (F4(A RGBch )));A RGBv =σ5(F5(A RGBch ));

[0106] A IRq =σ4(F GP (F4(A IRch )));A IRv =σ5(F5(A IRch ));

[0107] Where σ4 and σ5 both represent tensor reshaping operators; F4(·) and F5(·) both represent 1×1 convolution operations; F GP (·) represents the global aggregation operator, A RGBq∈R 1×C / 2 ;

[0108] A RGBv ∈R C / 2×HW ; A IRq ∈R 1×C / 2 ; A IRv ∈R C / 2×HW .

[0109] A RGBq and A IRq Represent the weights of the first and second features respectively, input to the Softmax function for classification, and output the weight distribution of the first and second features. The calculation process is expressed as follows:

[0110]

[0111] Where A RGBk represents the weight of the first feature, A IRk Represents the weight of the second feature.

[0112] In the calculation process of weight distribution, information is severely compressed. In order to maintain the information strength, information A is compressed. RGBv Multiply by weight A RGBk , information A IRv Multiply by weight A IRk , then perform information completion, reshaping and Sigmoid function in turn:

[0113] A RGBz =Sigmoid(σ6(A RGBv ×A RGBk ));

[0114] A IRz =Sigmoid(σ6(A IRv ×A IRk ));

[0115] Where A RGBz ∈R 1×HW ; A IRz ∈R 1×HW Representing the space gates of the first and second characteristics respectively.

[0116] Then, by adding A RGBch and A RGBz Multiply A IRch and A IRz Multiply them to obtain the spatial enhancement features A of the first and second features respectively RGBln and A IRln , the calculation process is expressed as:

[0117] A RGBln =A RGBch ⊙ARGBz ; A IRln =A IRch ⊙A IRz ;

[0118] Where A RGBln ∈R C×H×W ; A IRln ∈R C×H×W .

[0119] Feature A IRln With A RGBch Add to perform recalibration enhancement and obtain the final output X of the visible light feature RGBout ; Feature A RGBln With A IRch Add to perform recalibration enhancement and obtain the final output X of infrared features IRout , the calculation process is expressed as:

[0120] X RGBout =A IRln +A RGBch ;X IRout =A RGBln +A IRch .

[0121] In order to better fuse visible light and infrared features, the long-distance dependency fusion module is based on the SwinTransformer model, which greatly improves the fusion of multimodal complementary information by fusing features through the correlation of position encoding.

[0122] The process of feature fusion is:

[0123] Using the shift window partitioning method, the feature map is alternately divided into M×M dimensions. If the feature map size is smaller than M×M, it is padded to M×M size. Then, the window of the next module will be relatively offset by (M / 2, M / 2) pixels. Using this calculation method, the calculation formula is as follows:

[0124]

[0125]

[0126]

[0127]

[0128] Where, F i It represents the joint input of visible light and infrared features, F i ={X RGBout ,X IRout};F o is the output feature of the transformer block; and is an intermediate variable; W-MSA and MW-MSA are window multi-head self-attention operations and masked window multi-head self-attention operations, respectively.

[0129] Considering the partial features after window segmentation, in the process of self-attention calculation, given the input visible light feature map F RGB ∈R 8×8×C and infrared characteristic map F IR ∈R 8×8×C Then, each feature map is flattened and the order of the matrix is permuted to obtain sentence I RGB ∈R 64×C and I IRt ∈R 64×C . Then provide the input sentence I∈R 128×C Connecting Sentences I RGB and I IR Third, the input sentence I is projected into three weight matrices to obtain a set of queries Q, keys K, and values V:

[0130] Q=IW Q ,K=IW K ,V=IW V ;

[0131] Where W Q ∈R C×128 、W K ∈R C×128 and W V ∈R C×128 Both represent weight matrices.

[0132] The self-attention calculation process is as follows:

[0133]

[0134] Where d represents the dimension of the query Q or key K; FPRE represents the position encoding of visible light and infrared features. FPRE contains four types of position information: visible light position information RPE RGB , infrared position information RPE IR , visible light and infrared relative position information RPE RGB-IR , infrared and visible light relative position information RPE IR-RGB ;

[0135]

[0136] Where, T represents the matrix transpose operation.

[0137] After the above calculations, the visible light feature X output after deep interaction is obtained. RGBoutand infrared signature X IRout , and add the two together to get the final fusion feature F fusion :

[0138] F fusion =X RGBout +X IRout .

[0139] The essence of feature enhancement in the cross-feature enhancement module is to introduce noise data to force the network to learn complementary pattern information, thereby reducing the network's bias towards a single pattern. Feature enhancement is performed from both channel and spatial perspectives, including the exchange of complementary information between the two patterns, to better fuse multimodal features, effectively overcoming the differences between the two patterns and the severe lack of visible information, thereby improving the network's detection performance. The long-distance dependent fusion module focuses on deep interactive information enhancement, synchronously segmenting the features of the two patterns through a shifted window, and performing deep interactive information enhancement on the fused features through a multi-head self-retention mechanism. This improves the adaptability of the fused features in complex lighting scenarios and reduces missed detections and false detections by the detector.

[0140] The present application also provides an infrared-visible light fusion system based on feature enhancement, comprising:

[0141] Data acquisition module: used to acquire multi-target data sets and pre-process the multi-target data sets, wherein the multi-target data sets include visible light images and infrared images.

[0142] Feature extraction module: used to construct a dual-stream backbone feature extraction network. The dual-stream backbone feature extraction network includes two branches with the same structure. The visible light image and infrared image are respectively sent to the two branches to extract deep features to obtain visible light features and infrared features.

[0143] And a feature fusion module: used to construct a feature fusion network, the feature fusion network includes a cross-feature enhancement module and a long-distance dependency fusion module, the cross-feature enhancement module includes a channel attention branch and a spatial attention branch arranged in series, the visible light feature and the infrared feature are sent to the channel attention branch for enhancement, the visible light feature before enhancement is added to the infrared feature after enhancement by the channel attention branch to obtain a first feature, and the infrared feature before enhancement is added to the visible light feature after enhancement by the channel attention branch to obtain a second feature; the first feature and the second feature are sent to the spatial attention branch for enhancement, the first feature is added to the second feature after enhancement by the spatial attention branch to obtain the final output of the visible light feature, and the second feature is added to the first feature after enhancement by the spatial attention branch to obtain the final output of the infrared feature; the final outputs of the visible light feature and the infrared feature are sent to the long-distance dependency fusion module, and fused through the correlation of position encoding based on the Swin Transformer model.

[0144] The present application also provides a readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the above-mentioned feature enhancement-based infrared-visible light fusion method.

[0145] Example 1

[0146] The model was constructed using the feature-enhanced infrared-visible light fusion method proposed in this application. Training was performed on a 1080Ti desktop computer using the SGD optimizer with an initial learning rate of 0.001, a momentum of 0.937, and a weight decay of 0.0005. The overall performance of our proposed method was extensively tested using the VEDAI database. The test results are shown in the following table:

[0147]

[0148]

[0149] As can be seen from Table 1, the method proposed in this application achieves the best performance compared with other methods. Compared with the best single-modal detection algorithm, the mAP indicator (average of average precision) is improved by 12.9%; for multimodal detection methods, the proposed method improves mAP (average of average precision) by 3.1% compared with the best method; compared with the basic detector, the method proposed in this application significantly reduces missed detections and false detections. This method improves the target feature representation through a cross-feature enhancement module, greatly improving the detection performance. In addition, this method conducts deep interaction between the information of the two modes, with higher information fusion and better detection performance.

[0150] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An infrared-visible light fusion method based on feature enhancement, characterized in that: The steps include: Data acquisition: acquiring a multi-target data set and preprocessing the multi-target data set, wherein the multi-target data set includes a visible light image and an infrared image; Feature extraction: A dual-stream backbone feature extraction network is constructed. The dual-stream backbone feature extraction network includes two branches with the same structure. The visible light image and infrared image are respectively fed into the two branches to extract deep features, thereby obtaining visible light features and infrared features. Feature fusion: Construct a feature fusion network, which includes a cross-feature enhancement module and a long-distance dependency fusion module. The cross-feature enhancement module includes a channel attention branch and a spatial attention branch arranged in series. The visible light feature and the infrared feature are sent to the channel attention branch for enhancement, and the visible light feature before enhancement is added to the infrared feature after enhancement by the channel attention branch to obtain a first feature, and the infrared feature before enhancement is added to the visible light feature after enhancement by the channel attention branch to obtain a second feature; the first feature and the second feature are sent to the spatial attention branch for enhancement, and the first feature is added to the second feature after enhancement by the spatial attention branch to obtain the final output of the visible light feature, and the second feature is added to the first feature after enhancement by the spatial attention branch to obtain the final output of the infrared feature; the final outputs of the visible light feature and the infrared feature are sent to the long-distance dependency fusion module, and fused through the correlation of position encoding based on the Swin Transformer model.

2. The infrared-visible light fusion method based on feature enhancement according to claim 1, characterized in that: The preprocessing method is to use a data augmentation method of random region generation of image masks to process the visible light image and infrared image in the multi-target dataset. The specific process is as follows: the image is divided into 10×10 checkerboards according to the image size, and two squares set to zero are selected in each row with a probability of 30% as the image mask; then the image mask is divided into two complementary masks by row, one of which is used as the mask of the visible light image and the other as the mask of the infrared image. The mask generation process is expressed as follows: Generate mask =RGB mask ∪IR mask ; RGB mask |IR mask =1; Where, RGB mask is a mask for visible light images, IR mask It is a mask for infrared images, Generate mask It is the total mask.

3. The infrared-visible light fusion method based on feature enhancement according to claim 2, characterized in that: The dual-stream backbone feature extraction network is a dual-stream backbone YOLOv5 feature extraction network, and the extracted visible light features are expressed as Infrared signature is expressed as Represents a three-dimensional matrix; W is the width; H is the height; C represents the number of channels.

4. The infrared-visible light fusion method based on feature enhancement according to claim 3, characterized in that: The enhancement process of the channel attention branch is to completely fold the input features in one direction q while maintaining high resolution in the orthogonal direction v of direction q. The operation process is expressed as: W RGB q =σ1(F1(X RGB ));W RGB v =σ2(F2(X RGB )); W IR q =σ1(F1(X IR ));W IR v =σ2(F2(X IR )); Where W RGB q Represents the information of visible light characteristics in the q direction; W RGB v Represents the information of visible light features in the v direction; W IR q Represents the information of infrared characteristics in the q direction; W IR v represents the information of infrared features in the v direction; σ1 and σ2 both represent tensor shaping operators; F1(·) and F2(·) both represent 1×1 convolution operations; W RGB q and W IR q Represent the weights of visible light features and infrared features respectively, input to the Softmax function for classification, and output the weight distribution of visible light features and infrared features. The calculation process is expressed as: Where W RGB k Represents the weight of visible light features; W IR k represents the weight of infrared features; The information W RGB v Multiply by weight W RGB k , the information W IR v Multiply by weight W IR k , then perform a 1×1 convolution operation, use normalization to upgrade the channel dimension from C / 2 to C, and use the Sigmoid function to keep all parameters in the range of 0-1. The calculation process is expressed as: W RGB z =Sigmoid(σ3(F3(W RGB v ×W RGB k ))); W IR z =Sigmoid(σ3(F3(W IR v ×W IR k ))); Where W RGB z Represents the information of visible light features in the z direction; W IR z Represents the information of infrared features in the z direction; "×" represents the matrix dot product operation; F3(·) represents the 1×1 convolution operation; σ3 represents the tensor reshaping operator; Then X RGB and W RGB z Perform channel-level multiplication to obtain the feature W with less noise RGB ln ; X IR and W IR z Perform channel-level multiplication to obtain the feature W with less noise IR ln , the calculation process is expressed as: W RGB ln =X RGB ⊙W RGB z ;W IR ln =X IR ⊙W IR z ; Where, ⊙ represents the Hadamard product operation; The feature W RGB ln With X RGB Add to perform recalibration enhancement and obtain the first feature A RGB ch ; The feature W RGB ln With X IR Add to perform recalibration enhancement and obtain the second feature A IR ch , the calculation process is expressed as: A RGB ch =W IR ln +X RGB ;A IR ch =W RGB ln +X IR 。 5. The infrared-visible light fusion method based on feature enhancement according to claim 4, characterized in that: The enhancement process of the spatial attention branch is to completely collapse the input features in direction q while maintaining high resolution in direction v. The operation process is expressed as: A RGB q =σ4(F GP (F4(A RGB ch )));A RGB v =σ5(F5(A RGB ch )); A IR q =σ4(F GP (F4(A IR ch )));A IR v =σ5(F5(A IR ch )); Where σ4 and σ5 both represent tensor reshaping operators; F4(·) and F5(·) both represent 1×1 convolution operations; F GP (·) represents the global aggregation operator, A RGB q and A IR q Represent the weights of the first and second features respectively, input to the Softmax function for classification, and output the weight distribution of the first and second features. The calculation process is expressed as follows: Where A RGB k represents the weight of the first feature, A IR k represents the weight of the second feature; Information A RGB v Multiply by weight A RGB k , information A IR v Multiply by weight A IR k , then perform information completion, reshaping and Sigmoid function in turn: A RGB z =Sigmoid(σ6(A RGB v ×A RGB k )); A IR z =Sigmoid(σ6(A IR v ×A IR k )); Where A RGB z ∈R 1×HW ; A IR z ∈R 1×HW The space gates representing the first and second characteristics respectively; A RGB ch and A RGB z Multiply A IR ch and A IR z Multiply them to obtain the spatial enhancement features A of the first and second features respectively RGB ln and A IR ln , the calculation process is expressed as: A RGB ln =A RGB ch ⊙A RGB z ;A IR ln =A IR ch ⊙A IR z ; Where A RGB ln ∈R C×H×W ; A IR ln ∈R C×H×W ; Feature A IR ln With A RGB ch Add to perform recalibration enhancement and obtain the final output X of the visible light feature RGB out ; Feature A RGB ln With A IR ch Add to perform recalibration enhancement and obtain the final output X of infrared features IR out , the calculation process is expressed as: X RGB out =A IR ln +A RGB ch ;X IR out =A RGB ln +A IR ch 。 6. The infrared-visible light fusion method based on feature enhancement according to claim 5, characterized in that: The process of feature fusion is: Using the shift window partitioning method, the feature map is alternately divided into M×M dimensions. If the feature map size is smaller than M×M, it will be padded to the size of M×M; then, the window of the next module will be relatively offset by (M / 2, M / 2) pixels; using this calculation method, the calculation formula is as follows: Where, F i It represents the joint input of visible light and infrared features, F i ={X RGB out , X IR out };F o is the output feature of the transformer block; and is the intermediate variable; W-MSA and MW-MSA are the window multi-head self-attention operation and the masked window multi-head self-attention operation respectively; Considering the partial features after window segmentation, in the process of self-attention calculation, given the input visible light feature map F RGB ∈R 8×8×C and infrared characteristic map F IR ∈R 8×8×C ; Flatten each feature map and arrange the order of the matrix to obtain sentence I RGB ∈R 64×C and I IR ∈R 64×C ; Then provide the input sentence I∈R 128×C Connecting Sentences I RGB and I IR ; Project the input sentence I into three weight matrices to obtain a set of queries Q, keys K, and values V: Q2IW Q ,KIW K ,VIW V 100. Where W Q ∈R C×128 、W K ∈R C×128 and W V ∈R C×128 Both represent weight matrices; The self-attention calculation process is as follows: Where d represents the dimension of the query Q or key K; FPRE represents the position encoding of visible light and infrared features. FPRE contains four types of position information: visible light position information RPE RGB , infrared position information RPE IR , visible light and infrared relative position information RPE RGB-IR , infrared and visible light relative position information RPE IR-R ; Where, T represents the matrix transpose operation; After the above calculations, the visible light feature X output after deep interaction is obtained. RGB out and infrared signature X IR out , and add the two together to get the final fusion feature F fusion : F fusion =X RGB out +X IR out 。 7. An infrared-visible light fusion system based on feature enhancement, characterized in that: include: Data acquisition module: used for acquiring a multi-target data set and preprocessing the multi-target data set, wherein the multi-target data set includes a visible light image and an infrared image; Feature extraction module: used to build a dual-stream backbone feature extraction network. The dual-stream backbone feature extraction network includes two branches with the same structure. The visible light image and infrared image are respectively fed into the two branches to extract deep features, thereby obtaining visible light features and infrared features. And a feature fusion module: used to construct a feature fusion network, the feature fusion network includes a cross-feature enhancement module and a long-distance dependency fusion module, the cross-feature enhancement module includes a channel attention branch and a spatial attention branch arranged in series, the visible light feature and the infrared feature are sent to the channel attention branch for enhancement, the visible light feature before enhancement is added to the infrared feature after enhancement by the channel attention branch to obtain a first feature, and the infrared feature before enhancement is added to the visible light feature after enhancement by the channel attention branch to obtain a second feature; the first feature and the second feature are sent to the spatial attention branch for enhancement, the first feature is added to the second feature after enhancement by the spatial attention branch to obtain the final output of the visible light feature, and the second feature is added to the first feature after enhancement by the spatial attention branch to obtain the final output of the infrared feature; the final outputs of the visible light feature and the infrared feature are sent to the long-distance dependency fusion module, and fused through the correlation of position encoding based on the Swin Transformer model.

8. A readable storage medium, characterized in that: The readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the infrared-visible light fusion method based on feature enhancement as described in any one of claims 1-6.

Images