An infrared target detection method based on group feature interaction and valued feature fusion

Abstract

The present invention relates to the field of infrared image target detection, and provides an infrared target detection method of group feature interaction and assignment feature fusion, comprising: extracting infrared image information through a backbone network to obtain infrared image features; interacting with infrared image features to obtain fourth, fifth, and sixth features; fusing the fifth and sixth features through a fourth fusion module to obtain and store the fourth fusion feature, fusing the fourth and fourth fusion features through a first fusion module to obtain and store the first fusion feature, fusing the stored first fusion feature and the stored fourth fusion feature through a second fusion module to obtain and store the second fusion feature, fusing the stored second fusion feature and the sixth feature through a third fusion module to obtain the third fusion feature; and classifying and locating the first, second, and third fusion features through a detection head network to obtain a detection result. The present invention achieves higher infrared target detection accuracy with lower parameter quantity and calculation cost.

Description

Technical Field

[0001] The present invention relates to the field of infrared image target detection, and in particular to an infrared target detection method combining group feature interaction with assigned feature fusion. Background Art

[0002] Most of the methods used by early data-driven infrared target detection algorithms make the convolution method and the Transformer method of the self-attention mechanism independent of each other, and the characteristics of the two methods cannot be combined for either method. Taking the YOLO series of algorithms as an example, the algorithm uses a single neural network to complete target detection end-to-end, which has the advantages of real-time and simplicity, but the detection effect on small targets and dense targets is not as good as other methods, and it is easy to miss or misdetect, and the algorithm still has room for improvement. Subsequent algorithms such as the RT-DETR target detection algorithm based on the DETR algorithm combine the convolutional neural network with the Transformer method, and make corresponding improvements to achieve a lightweight model. It has very advanced target detection accuracy data, but its large number of convolution stacks still burdens the network with a huge amount of parameters, and the huge amount of calculations attached to the Transformer structure of its decoder part has higher requirements for hardware computing power.

[0003] Although combining the self-attention mechanism and convolutional neural network can improve the representation ability of features, it also faces some challenges. For example, CNN combined with the self-attention mechanism may increase the number of model parameters, increasing the training and inference costs of the model; the computational complexity of the self-attention mechanism is high, especially when processing large-size images or sequences, which can easily lead to excessive model calculations; too many self-attention mechanisms may cause the model to focus too much on local details or noise, thereby increasing the risk of overfitting of the model.

[0004] Although the conventional attention mechanism can capture the global relationship in the sequence or image and is not limited by the local receptive field, it is limited to one-dimensional information integration, and is prone to ignore the relevant information between feature groups in a certain area and other feature groups in the two-dimensional space, and is easily limited in the interaction of features within the area.

[0005] At the same time, most of the current multi-scale feature fusion methods are simple splicing or addition operations, which cannot clearly distinguish the importance of features of different scales and different channels to the network, thus limiting the performance of the network. Summary of the invention

[0006] The present invention aims to solve the technical problem that the interaction of features in a region is easily restricted and the importance of features of different scales and channels to the network cannot be clearly identified during feature fusion. To this end, the present invention provides an infrared target detection method for group feature interaction and valued feature fusion.

[0007] The present invention provides an infrared target detection method combining group feature interaction and valued feature fusion, comprising:

[0008] S1: Extract feature information from infrared images by using layer-by-layer convolution through the backbone network to obtain infrared image features;

[0009] S2: performing feature interaction on the infrared image features through the first module to obtain interactive features, wherein the interactive features include a fourth feature, a fifth feature, and a sixth feature;

[0010] S3: fusing the fifth feature and the sixth feature through the fourth fusion module to obtain and store the fourth fusion feature, fusing the fourth feature and the fourth fusion feature through the first fusion module to obtain and store the first fusion feature, fusing the stored first fusion feature and the stored fourth fusion feature through the second fusion module to obtain and store the second fusion feature, and fusing the stored second fusion feature and the sixth feature through the third fusion module to obtain the third fusion feature;

[0011] S4: Classify and locate the first fusion feature, the second fusion feature, and the third fusion feature through the YOLOv8s detection head network to obtain the target detection result.

[0012] According to an infrared target detection method for group feature interaction and valued feature fusion provided by the present invention, step S1 comprises:

[0013] S11: extracting features from the infrared image through a first extraction module to obtain first shallow features;

[0014] S12: extracting the first shallow feature through the second extraction module to obtain a first feature, and saving the first feature to the first feature storage module;

[0015] S13: extracting the first feature in the first feature storage module through the third extraction module to obtain a second feature, and saving the second feature to the second feature storage module;

[0016] S14: extracting the second feature in the second feature storage module through the fourth extraction module to obtain a third feature.

[0017] According to an infrared target detection method for group feature interaction and assignment feature fusion provided by the present invention, step S11 is: subjecting the infrared image to first CBR processing, second CBR processing, third CBR processing and maximum pooling calculation in sequence to obtain a first shallow sub-feature, and extracting the first shallow sub-feature through a first C2f module to obtain a first shallow feature;

[0018] In step S12: the feature extraction process of the second extraction module is: performing CBS processing on the first shallow feature, and performing feature extraction through the second C2f module to obtain the first feature;

[0019] In step S13, the feature extraction process of the third extraction module is: performing CBS processing on the first feature in the first feature storage module, and extracting the feature through the third C2f module to obtain the second feature;

[0020] In step S14, the feature extraction process of the fourth extraction module is: CBS processing is performed on the second feature in the second feature storage module, and feature extraction is performed through the fourth C2f module to obtain the third feature.

[0021] According to an infrared target detection method for group feature interaction and valued feature fusion provided by the present invention, step S2 comprises:

[0022] S21: performing convolution and CPE processing on the infrared image features in sequence to obtain and store the first sub-feature;

[0023] S22: performing LN layer and Linear linearization processing on the first sub-feature in sequence to obtain a first linear feature, a second linear feature, and a third linear feature;

[0024] S23: performing multi-scale mapping on the first linear feature, the second linear feature, and the third linear feature through the first grouping module to obtain a first splicing feature, a second splicing feature, a third splicing feature, and a fourth splicing feature;

[0025] S24: performing EA processing on the first splicing feature, the second splicing feature, and the third splicing feature to obtain a first attention feature;

[0026] S25: performing CRPE processing on the first attention feature and concatenating it with the fourth concatenated feature to obtain a second attention feature;

[0027] S26: Linearize the second attention feature and add it to the stored first sub-feature, and convolve the addition result to obtain the interaction feature.

[0028] According to an infrared target detection method for group feature interaction and valued feature fusion provided by the present invention, step S23 includes:

[0029] S231: split the first linear feature to obtain five first linear sub-features L1, L2, L3, L4 and L5, split the second linear feature to obtain five second linear sub-features M1, M2, M3, M4 and M5, split the third linear feature to obtain five third linear sub-features N1, N2, N3, N4 and N5;

[0030] S232: perform 3×3 convolution, BN layer and Hardswish function processing on L2, M2, and N2 in sequence to obtain L21, M21, and N21 in sequence; perform 5×5 convolution, BN layer and Hardswish function processing on L3, M3, and N3 in sequence to obtain L31, M31, and N31 in sequence; perform 7×7 convolution, BN layer and Hardswish function processing on L4, M4, and N4 in sequence to obtain L41, M41, and N41 in sequence;

[0031] S233: splicing L1, L21, L31 and L41 to obtain a first splicing feature, splicing M1, M21, M31 and M41 to obtain a second splicing feature, and splicing N1, N21, N31 and N41 to obtain a third splicing feature;

[0032] S234: concatenate L5, M5 and N5 to obtain a fourth concatenated sub-feature, and sequentially perform DWConv, LN layer and Hardswish processing on the fourth concatenated sub-feature to obtain a fourth concatenated feature.

[0033] According to an infrared target detection method for group feature interaction and valued feature fusion provided by the present invention, step S3 comprises:

[0034] S31: upsampling the sixth feature to obtain a first upsampling result, performing feature fusion on the first upsampling result and the fifth feature through the fourth fusion submodule to obtain a fourth fused sub-feature, performing feature extraction on the fourth fused sub-feature through the fifth C2f module to obtain a fourth fused feature and store it in the first storage module;

[0035] S32: upsampling the fourth fused feature to obtain a second upsampling result, performing feature fusion on the second upsampling result and the fourth feature through the first fusion submodule to obtain a first fused sub-feature, performing feature extraction on the first fused sub-feature through the sixth C2f module to obtain a first fused feature and store it in the second storage module;

[0036] S33: down-sample the first fusion feature in the second storage module to obtain a first down-sampling result, perform feature fusion on the first down-sampling result and the fourth fusion feature of the first storage module through the second fusion sub-module to obtain a second fusion sub-feature, perform feature extraction on the second fusion sub-feature through the seventh C2f module, obtain a second fusion feature and store it in the third storage module;

[0037] S34: down-sample the second fusion feature of the third storage module to obtain a second down-sampling result, perform feature fusion on the second down-sampling result and the sixth feature through the third fusion sub-module to obtain a third fusion sub-feature, and perform feature extraction on the third fusion sub-feature through the eighth C2f module to obtain a third fusion feature.

[0038] According to an infrared target detection method for group feature interaction and valued feature fusion provided by the present invention, in step S31, the feature fusion process of the fourth fusion submodule is: assigning values ​​to the first upsampling result and the fifth feature respectively through the first weight module to obtain the first weight feature and the second weight feature, adding the first weight feature and the second weight feature to obtain the fourth fusion sub-feature;

[0039] In step S32, the feature fusion process of the first fusion submodule is: assigning values ​​to the second upsampling result and the fourth feature respectively through the second weight module to obtain a third weight feature and a fourth weight feature; adding the third weight feature and the fourth weight feature to obtain the first fusion sub-feature;

[0040] In step S33, the feature fusion process of the second fusion submodule is: assigning values ​​to the first downsampling result and the fourth fusion feature of the first storage module respectively through the third weight module to obtain the fifth weight feature and the sixth weight feature; adding the fifth weight feature and the sixth weight feature to obtain the second fusion sub-feature;

[0041] In step S34, the feature fusion process of the third fusion submodule is: assigning values ​​to the second downsampling result and the sixth feature respectively through the fourth weight module to obtain the seventh weight feature and the eighth weight feature; adding the seventh weight feature and the eighth weight feature to obtain the third fusion sub-feature.

[0042] According to an infrared target detection method for group feature interaction and assignment feature fusion provided by the present invention, the first weight module assignment process is: normalizing the first weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed first weight matrix to the first upsampling result and the fifth feature respectively, to obtain the first weight feature and the second weight feature;

[0043] The second weight module assignment process is: normalizing the second weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed second weight matrix to the second upsampling result and the fourth feature respectively to obtain the third weight feature and the fourth weight feature;

[0044] The third weight module assignment process is: normalizing the third weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed third weight matrix to the first downsampling result and the fourth fusion feature of the first storage module respectively to obtain the fifth weight feature and the sixth weight feature;

[0045] The fourth weight module assignment process is: normalize the fourth weight matrix, perform Sigmoid function and Split splitting processing in turn, assign the processed fourth weight matrix to the second downsampling result and the sixth feature respectively, and obtain the seventh weight feature and the eighth weight feature.

[0046] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0047] 1. The present invention can fully and efficiently extract feature information in infrared images through the backbone network, thereby achieving efficient and accurate target detection.

[0048] 2. The present invention performs feature interaction on infrared image features, taking full account of surrounding environment and background information related to the current position or feature.

[0049] 3. The present invention performs multi-scale feature fusion on interactive features, fully considers the importance of features of different scales and different channels, improves the quality and efficiency of feature fusion, and improves the accuracy of infrared image target detection.

[0050] 4. The present invention classifies and locates the first fusion feature, the second fusion feature and the third fusion feature through the YOLOv8s detection head network to achieve high-precision infrared target detection.

[0051] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a schematic diagram of the process provided by the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0055] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0056] Combine the following Figure 1 The present invention is further described in detail, and a method for infrared target detection by group feature interaction and valued feature fusion of the present invention is described.

[0057] In this embodiment, Figure 1 As shown, a method for infrared target detection by group feature interaction and assignment feature fusion is provided, comprising:

[0058] S1: Extract feature information from infrared images by using layer-by-layer convolution through the backbone network to obtain infrared image features;

[0059] The backbone network is based on CSPDarknet (cross-stage local network) of YOLOv8s, wherein CSPDarknet includes four stages (first stage, second stage, third stage and fourth stage), utilizes the initial module part (Stem) of Resnet18 network to replace the first stage of CSPDarknet, uses C2f (Convolution to Feature) modules with different numbers of Bottleneck (bottleneck structure), and removes the original SPPF (fast spatial pyramid pooling) module in the fourth stage of CSPDarknet. The backbone network improved based on CSPDarknet can, on the one hand, maintain the similarity of features of each scale for subsequent intra-scale feature interaction and multi-scale feature fusion, and on the other hand, can fully extract infrared images and improve the accuracy of feature extraction of the backbone network; infrared image features include first features, second features and third features.

[0060] Using Z (representing any number) Bottleneck C2f modules, the process of processing the input features is:

[0061] (1) The input feature map is first subjected to a standard convolution with a convolution kernel size of 1×1 for feature mapping, and then the feature map is split into two halves in the channel dimension through the Split operation;

[0062] (2) Half of the feature maps are processed by N Bottlenecks of the same structure in sequence. At the same time, the feature maps output by each Bottleneck are concatenated with the other half of the feature maps after the Split operation by the concat (feature connection) method. The concatenated result is then subjected to a standard convolution with a convolution kernel size of 1×1 for feature mapping to obtain the final output result of the C2f module. The Bottleneck processing process is to extract feature information by sequentially passing through two standard convolutions with a convolution kernel size of 3×3.

[0063] Wherein, step S1 comprises:

[0064] S11: extracting features from the infrared image through a first extraction module to obtain first shallow features;

[0065] Step S11 is: subjecting the infrared image to first CBR (Convolution Batch Normalization ReLU, convolution normalization ReLU activation) processing, second CBR processing, third CBR processing and maximum pooling calculation in sequence to obtain a first shallow sub-feature, and extracting the first shallow sub-feature through a first C2f module to obtain a first shallow feature;

[0066] The first CBR processing is to pass through a standard convolution with a convolution kernel size of 3×3 (where the convolution kernel sliding step is 2), a BN (batch normalization) layer, and a ReLU (Rectified Linear Unit) activation function in sequence. At this time, the feature map size is reduced to half the size of the input image, and the number of output feature map channels is 32; the second CBR processing is to pass through a standard convolution with a convolution kernel size of 3×3 (where the convolution kernel sliding step is 1), a BN layer, and a ReLU activation function in sequence. At this time, the feature map size remains unchanged, and the number of output feature map channels is still 32; the third CBR processing is to pass through a standard convolution with a convolution kernel size of 3×3 (where the convolution kernel sliding step is 1), a BN layer, and a ReLU activation function in sequence. The feature map size remains unchanged, and the number of output feature map channels increases to 64. The three CBR processes all achieved the retention of feature maps and feature information; the maximum pooling calculation can retain the main features in the infrared image, while reducing the size of the infrared image and reducing the amount of calculation. The pooling window size is 3×3, the window sliding step is 2, the feature map size is reduced to half of the output feature map size of the third CBR process, and the number of output feature map channels is maintained at 64; the first C2f module is a Bottleneck C2f module that extracts features of different levels and abstraction levels in the infrared image, and the feature map size and number of channels remain unchanged.

[0067] S12: extracting the first shallow feature through the second extraction module to obtain a first feature, and saving the first feature to the first feature storage module;

[0068] In step S12: the feature extraction process of the second extraction module is: CBS (Convolution Batch Normalization SiLU, convolution normalization SiLU activation) processing is performed on the first shallow feature, and feature extraction is performed through the second C2f module to obtain the first feature;

[0069] The CBS processing is to sequentially process a set of convolutions with a convolution kernel size of 1×1, a BN layer and a SiLU (Sigmoid Linear Unit) activation function to retain the feature map and feature information; the second C2f module is a C2f module using two Bottlenecks to extract features of different levels and abstract degrees in the infrared image; the number of channels of the first feature is 128.

[0070] S13: extracting the first feature in the first feature storage module through the third extraction module to obtain a second feature, and saving the second feature to the second feature storage module;

[0071] In step S13, the feature extraction process of the third extraction module is: performing CBS processing on the first feature in the first feature storage module, and extracting the feature through the third C2f module to obtain the second feature;

[0072] The third C2f module is a C2f module using two Bottlenecks; the number of channels of the second feature is 256.

[0073] S14: extracting the second feature in the second feature storage module through the fourth extraction module to obtain a third feature;

[0074] In step S14, the feature extraction process of the fourth extraction module is: CBS processing is performed on the second feature in the second feature storage module, and feature extraction is performed through the fourth C2f module to obtain the third feature.

[0075] The fourth C2f module is a C2f module using one Bottleneck; the number of channels of the third feature is 512.

[0076] S2: performing feature interaction on the infrared image features through the first module to obtain interactive features, wherein the interactive features include a fourth feature, a fifth feature, and a sixth feature;

[0077] Infrared image features include a first feature, a second feature and a third feature; the first module includes three sub-modules, namely the first sub-module, the second sub-module and the third sub-module, the first sub-module performs feature interaction on the first feature to obtain a fourth feature, the second sub-module performs feature interaction on the second feature to obtain a fifth feature, and the third sub-module performs feature interaction on the third feature to obtain a sixth feature; the feature interaction processes of the three sub-modules are the same, realizing local and global interactions of group features of different scales within the scale, and fully considering the surrounding environment and background information related to the current position or feature on the basis of the local relationship of the group features, and the higher position information ratio characteristic compared to the visible light image is more suitable for infrared images, and a significant improvement in accuracy is achieved at a lower model scale cost.

[0078] Wherein, step S2 comprises:

[0079] S21: performing convolution and CPE (Conditional Positional Encoding) processing on the infrared image features in sequence to obtain and store the first sub-feature;

[0080] The convolution uses a standard convolution with a convolution kernel size of 1×1 to adjust the number of feature map channels. The first submodule is used to perform feature interaction on the first feature, and the number of feature channels of the fourth feature is adjusted to 120. The second submodule is used to perform feature interaction on the second feature, and the number of feature channels of the fifth feature is adjusted to 240. The third submodule is used to perform feature interaction on the third feature, and the number of feature channels of the sixth feature is adjusted to 480. Through CPE processing, absolute position encoding is achieved, with better encoding performance.

[0081] S22: performing LN layer (layer normalization) and Linear (fully connected layer) linearization processing on the first sub-feature in sequence to obtain a first linear feature, a second linear feature, and a third linear feature;

[0082] The LN layer normalizes the input of each layer to make the network training more stable and efficient, thereby improving the generalization ability of the model. The Linear linearization process linearly transforms the input features and projects the input space into a higher-dimensional space, providing rich dimensions for the learning and understanding of the model, greatly improving its expressiveness in complex tasks. The first linear feature, the second linear feature, and the third linear feature are the core components of the self-attention mechanism, representing query, key, and value, respectively.

[0083] S23: performing multi-scale mapping on the first linear feature, the second linear feature, and the third linear feature through the first grouping module to obtain a first splicing feature, a second splicing feature, a third splicing feature, and a fourth splicing feature;

[0084] Wherein, step S23 comprises:

[0085] S231: split the first linear feature to obtain five first linear sub-features L1, L2, L3, L4 and L5, split the second linear feature to obtain five second linear sub-features M1, M2, M3, M4 and M5, split the third linear feature to obtain five third linear sub-features N1, N2, N3, N4 and N5;

[0086] The first linear feature, the second linear feature and the third linear feature are respectively split into corresponding 5 first linear sub-features, 5 second linear sub-features and 5 third linear sub-features along the channel dimension.

[0087] S232: Perform 3×3 convolution, BN layer and Hardswish (hard-coded scaled exponential linear unit with compression) function processing on L2, M2, and N2 in turn, and obtain L21, M21, and N21 in turn; perform 5×5 convolution, BN layer and Hardswish function processing on L3, M3, and N3 in turn, and obtain L31, M31, and N31 in turn; perform 7×7 convolution, BN layer and Hardswish function processing on L4, M4, and N4 in turn, and obtain L41, M41, and N41 in turn;

[0088] Standard convolution with kernel size of 3×3 is performed on L2, M2, and N2, standard convolution with kernel size of 5×5 is performed on L3, M3, and N3, and standard convolution with kernel size of 7×7 is performed on L4, M4, and N4. The convolution operation of convolution kernels of different sizes maps the group features within the size through convolution, and the mapped group features represent various information between the group features in subsequent attention operations. Hardswish is a pruned linear activation function that is easy to integrate and has high computational efficiency.

[0089] S233: splicing L1, L21, L31 and L41 to obtain a first splicing feature, splicing M1, M21, M31 and M41 to obtain a second splicing feature, and splicing N1, N21, N31 and N41 to obtain a third splicing feature;

[0090] The splicing is performed in a concat manner, which splices different features together according to the channel dimension to make full use of feature information from different levels and realize multi-scale feature fusion.

[0091] S234: concatenate L5, M5 and N5 to obtain a fourth concatenated sub-feature, and sequentially perform DWConv (separable convolution), LN layer and Hardswish processing on the fourth concatenated sub-feature to obtain a fourth concatenated feature.

[0092] DWConv separates spatial and channel operations, making the model lighter and more efficient. The specific expression is:

[0093]

[0094] in, For the first of the five first, five second and five third linear sub-features The result of processing the linear sub-features separately is: To perform Hardswish activation, To perform BN layer operation, To perform LN layer operations, For convolution kernel size The convolution operation, To perform a depth-wise separable convolution operation with a kernel size of 1×1, Indicates the feature number, represents the first of the five first, five second, and five third linear sub-features Linear sub-features.

[0095] S24: performing EA (EfficientAttention) processing on the first splicing feature, the second splicing feature, and the third splicing feature to obtain a first attention feature;

[0096] The first splicing feature, the second splicing feature and the third splicing feature are respectively , , To indicate that , , , represents the set of real numbers, represents the feature size, , and Respectively , , The dimension of , , The EA calculation is expressed as :

[0097]

[0098] in, Represents matrix transpose, with Softmax as the normalization function, and Apply the Softmax function to each row or column of the matrix separately.

[0099] S25: Perform CRPE (Convolutional Relative Position Encoding) processing on the first attention feature and concatenate it with the fourth concatenated feature to obtain the second attention feature;

[0100] CRPE processing can better adapt to the transformation of feature maps.

[0101] S26: Linearize the second attention feature and add it to the stored first sub-feature, and convolve the addition result to obtain the interaction feature.

[0102] The second attention feature is linearized and added to the stored first sub-feature. The result of the addition is subjected to a standard convolution calculation with a convolution kernel size of 1×1 to adjust the feature map channels. At this time, the number of the fourth feature channel is restored to 128, the number of the fifth feature channel is restored to 256, and the number of the sixth feature channel is restored to 512.

[0103] S3: fusing the fifth feature and the sixth feature through the fourth fusion module to obtain and store the fourth fusion feature, fusing the fourth feature and the fourth fusion feature through the first fusion module to obtain and store the first fusion feature, fusing the stored first fusion feature and the stored fourth fusion feature through the second fusion module to obtain and store the second fusion feature, and fusing the stored second fusion feature and the sixth feature through the third fusion module to obtain the third fusion feature;

[0104] The structures and functions of the four fusion modules (the first fusion module, the second fusion module, the third fusion module and the fourth fusion module) are the same, which can identify the feature contributions of different scales and different channels at each scale, and make full use of important information to improve the target detection performance of infrared image detection.

[0105] Wherein, step S3 comprises:

[0106] S31: upsampling the sixth feature to obtain a first upsampling result, performing feature fusion on the first upsampling result and the fifth feature through the fourth fusion submodule to obtain a fourth fused sub-feature, performing feature extraction on the fourth fused sub-feature through the fifth C2f module to obtain a fourth fused feature and store it in the first storage module;

[0107] S32: upsampling the fourth fused feature to obtain a second upsampling result, performing feature fusion on the second upsampling result and the fourth feature through the first fusion submodule to obtain a first fused sub-feature, performing feature extraction on the first fused sub-feature through the sixth C2f module to obtain a first fused feature and store it in the second storage module;

[0108] S33: down-sample the first fusion feature in the second storage module to obtain a first down-sampling result, perform feature fusion on the first down-sampling result and the fourth fusion feature of the first storage module through the second fusion sub-module to obtain a second fusion sub-feature, perform feature extraction on the second fusion sub-feature through the seventh C2f module, obtain a second fusion feature and store it in the third storage module;

[0109] S34: down-sample the second fusion feature of the third storage module to obtain a second down-sampling result, perform feature fusion on the second down-sampling result and the sixth feature through the third fusion sub-module to obtain a third fusion sub-feature, and perform feature extraction on the third fusion sub-feature through the eighth C2f module to obtain a third fusion feature.

[0110] After upsampling the high semantic features, the length and width of the high semantic features are made the same as the corresponding low semantic features; after downsampling the low semantic features, the length and width of the low semantic features are made the same as the corresponding high semantic features; the fifth C2f module, the sixth C2f module, the seventh C2f module and the eighth C2f module have the same structure and function, and all use a Bottleneck C2f module to extract features of different levels and abstraction levels in infrared images.

[0111] Furthermore, each fusion module includes one fusion sub-module (i.e., the first fusion module includes the first fusion sub-module, the second fusion module includes the second fusion sub-module, the third fusion module includes the third fusion sub-module, and the fourth fusion module includes the fourth fusion sub-module), wherein the structure and function of each fusion sub-module are the same, fully considering the importance of different scale features and different channel features, performing fully and efficiently multi-scale feature fusion, and improving the accuracy of infrared target image detection.

[0112] In step S31, the feature fusion process of the fourth fusion submodule is: assigning values ​​to the first upsampling result and the fifth feature respectively through the first weight module to obtain a first weight feature and a second weight feature, adding the first weight feature and the second weight feature to obtain a fourth fusion sub-feature;

[0113] The first weight module assignment process is: normalizing the first weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed first weight matrix to the first upsampling result and the fifth feature respectively to obtain the first weight feature and the second weight feature;

[0114] In step S32, the feature fusion process of the first fusion submodule is: assigning values ​​to the second upsampling result and the fourth feature respectively through the second weight module to obtain a third weight feature and a fourth weight feature; adding the third weight feature and the fourth weight feature to obtain the first fusion sub-feature;

[0115] The second weight module assignment process is: normalizing the second weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed second weight matrix to the second upsampling result and the fourth feature respectively to obtain the third weight feature and the fourth weight feature;

[0116] In step S33, the feature fusion process of the second fusion submodule is: assigning values ​​to the first downsampling result and the fourth fusion feature of the first storage module respectively through the third weight module to obtain the fifth weight feature and the sixth weight feature; adding the fifth weight feature and the sixth weight feature to obtain the second fusion sub-feature;

[0117] The third weight module assignment process is: normalizing the third weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed third weight matrix to the first downsampling result and the fourth fusion feature of the first storage module respectively to obtain the fifth weight feature and the sixth weight feature;

[0118] In step S34, the feature fusion process of the third fusion submodule is: assigning values ​​to the second downsampling result and the sixth feature respectively through the fourth weight module to obtain the seventh weight feature and the eighth weight feature; adding the seventh weight feature and the eighth weight feature to obtain the third fusion sub-feature;

[0119] The fourth weight module assignment process is: normalize the fourth weight matrix, perform Sigmoid function and Split splitting processing in turn, assign the processed fourth weight matrix to the second downsampling result and the sixth feature respectively, and obtain the seventh weight feature and the eighth weight feature.

[0120] Specifically, a weight matrix of a trainable channel is generated, which can update infrared image data through network training. The weight matrix is ​​quickly normalized and then processed by Sigmoid function to achieve softened channel weight distribution. Then, the weight matrix is ​​split to achieve average splitting to obtain two weight sub-matrices. The two weight sub-matrices are assigned to the features of the two inputs to obtain corresponding weight features.

[0121] The calculation formula is expressed as:

[0122]

[0123] in, is the initial weight matrix after training, It is the weight matrix after the positive value screening of the ReLU function No. elements, is the weight matrix After fast normalization elements, Denominator bias. In order to avoid unstable calculation values, for example, when the denominator is not zero, and based on experience, a smaller value is determined. In this embodiment, =0.0001; Represents the sum of all elements of the weight matrix. ReLU and Sigmoid are activation functions. ReLU ensures that the weight value is positive, and Sigmoid can smooth the distribution of weight values ​​to prevent the loss of features caused by excessive dilution of some features after weight training.

[0124]

[0125] in, and The weight matrices are Weight features corresponding to the two inputs after splitting and The weight sub-matrix of is the multiplication assignment operation, is the feature addition fusion operation, Indicates a Split operation. is the output fusion sub-feature.

[0126] S4: Classify and locate the first fusion feature, the second fusion feature, and the third fusion feature through the YOLOv8s detection head network to obtain the target detection result.

[0127] The first fusion feature, the second fusion feature, and the third fusion feature are input into the YOLOv8s detection head network. The prediction tasks of targets of different sizes are assigned according to the accuracy of targets of different scales detected during network training. The output results of three scales, small targets, medium targets, and large targets, are output. Finally, the comprehensive results (bounding box coordinates, types, and type probabilities of all targets) are output to obtain the final target detection results. The YOLOv8s detection head uses multi-scale prediction technology to better detect objects of different sizes. This enables the model to perform better when processing images with targets of various sizes.

[0128] In a specific embodiment:

[0129] A1: Build infrared image target detection training set and validation set;

[0130] In the first stage of model training, the method of the present invention uses the FLIR thermal infrared image target detection dataset to train the target detection network architecture (including the backbone network, the neck network and the detection head network) designed by the present invention.

[0131] The FLIR thermal infrared dataset is obtained by a thermal imager installed on a vehicle. The forward-looking infrared thermal imager FLIRTau2 is used, and the parameters are set to 13mm f / 1.0, 45-degree horizontal field of view (HFOV) and 37-degree vertical field of view (VFOV) to obtain thermal images. Both cameras are running in default mode. The infrared thermal imager FLIR Tau2 dataset contains a total of 14,452 annotated thermal images, of which 10,228 are from short videos and 4,224 are from 144 seconds of continuous video.

[0132] We conducted training experiments on objects with a reasonable number of labels for the three categories in the dataset, and finally selected "people", "bicycles" and "cars" as experimental objects. The number of target annotations in the training set was 22,372 for "people", 3,986 for "bicycles" and 41,260 for "cars", and the number of "people", "bicycles" and "cars" in the validation set was 5,779 for "people", 471 for "bicycles" and 5,432 for "cars". The training used cocodataset-detection-eval (COCO dataset) as the accuracy evaluation standard.

[0133] A2: Train the network model;

[0134] The overall network model is trained based on the Pytorch deep learning framework, and the performance of the network model is verified using the validation set. During the experiment, the NVIDIA GeForce RTX 3080Ti graphics card is used for training and testing, and the experimental parameter settings are shown in Table 1.

[0135] Table 1 Experimental parameter settings

[0136]

[0137] Among them, epochs represents the number of model training rounds, batch size represents the size of each batch of data selected during training, workers represents the number of threads used by the CPU when loading data, image size represents the image size, amp represents automatic mixed precision training, mosaic (probability) represents data enhancement, optimizer represents optimizer, finallr represents feedforward neural network learning rate, momentum represents momentum, weight_decay represents weight decay, which aims to reduce the complexity of the model and prevent overfitting, warmup_epochs represents the number of training rounds for model warmup, and warmup_momentum represents the momentum setting during warmup training.

[0138] A3: Build a new infrared image target detection dataset;

[0139] Fine-tune the network model training parameters such as batch size, final lr, momentum, etc., and train the network model constructed by the algorithm multiple times to obtain better training results, so that the network model has better target detection performance.

[0140] A4: Input the trained target detection model to obtain the infrared target detection result.

[0141] The existing public data set and the infrared imaging device are used to obtain infrared images and manually annotate the image targets to form a new infrared image target detection data set. A PC (personal computer) device is selected to input the newly formed infrared image target detection data set images into the trained network model designed by the present invention to obtain the final infrared image target detection results.

[0142] The method provided by the present invention is trained and verified with a number of mainstream YOLO series algorithms and the advanced Transformer application algorithm RT-DETR under the same data set, and the experimental results of the algorithm model detection accuracy data and model scale data comparison are shown in Table 2.

[0143] Table 2 Experimental results compared with YOLO series and Transform architecture algorithms

[0144]

[0145] Among them, mAP(0.5) represents the average accuracy of target detection when the intersection-over-union ratio is 0.5, mAP(0.5:0.95) represents the average accuracy of target detection when the intersection-over-union ratio is 0.5, 0.55, 0.6...0.95, Parameters(M) represents the number of network model parameters, and FLOPs(G) represents the amount of network model calculation.

[0146] The higher the mAP(0.5) and mAP(0.5:0.95) values ​​are, the higher the accuracy of target detection is; the lower the Parameters(M) and FLOPs(G) values ​​are, the lighter the network model of target detection is.

[0147] The experimental data in Table 2 show that compared with YOLOv8s, the infrared image target detection method proposed in the present invention has a slightly improved model scale, but greatly improved detection accuracy; with a parameter amount of 1.9M and a computational cost of 8.9G, a 2.4% mAP(0.5) improvement and a 1.5% mAP(0.5:0.95) improvement are obtained, which has a considerable performance improvement.

[0148] Compared with a series of detection network models with huge model scale parameters such as YOLOv5m, YOLOv6s, RT-DETR and YOLOv8m, the infrared image target detection method proposed in this invention can achieve detection accuracy performance that exceeds that of a number of high-precision models with a smaller network model scale. With only 50.5% of the parameters and 47.5% of the calculation amount of the YOLOv8m model, the detection accuracy mAP (0.5) and mAP (0.5:0.95) indicators are increased by 1% and 0.4% respectively, which verifies the effectiveness and superiority of the method of this invention.

[0149] The beneficial effects of the present invention are as follows: the infrared target detection method proposed by the present invention combines group feature interaction and assigned feature, and performs sufficient feature extraction through the proposed efficient convolution backbone network to improve the accuracy of infrared target detection; the first module proposed by the present invention realizes local interaction and global interaction of group features of different scales within the scale, and fully considers the surrounding environment and background information related to the current position or feature on the basis of the local relationship of group features; the fusion module proposed by the present invention fully considers the importance of features of different scales and features of different channels, and performs sufficient and efficient multi-scale feature fusion, achieving a substantial increase in detection accuracy at a low cost of model parameters and computational complexity. Compared with today's high-precision detection algorithms, it has a lighter model scale, that is, it achieves a substantial improvement in infrared image target detection accuracy at a low cost of model parameters and computational complexity, and has a huge advantage over existing algorithms.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infrared target detection method based on group feature interaction and valued feature fusion. It is characterized in that include: S1: Extract feature information from infrared images by using layer-by-layer convolution through the backbone network to obtain infrared image features; Specifically include: S11: extracting features from the infrared image through a first extraction module to obtain first shallow features; S12: extracting the first shallow feature through the second extraction module to obtain a first feature, and saving the first feature to the first feature storage module; S13: extracting the first feature in the first feature storage module through the third extraction module to obtain a second feature, and saving the second feature to the second feature storage module; S14: extracting the second feature in the second feature storage module through the fourth extraction module to obtain a third feature; S2: performing feature interaction on the infrared image features through the first module to obtain interactive features, wherein the interactive features include a fourth feature, a fifth feature and a sixth feature; wherein the first module includes three submodules, namely a first submodule, a second submodule and a third submodule, performing feature interaction on the first feature through the first submodule to obtain a fourth feature, performing feature interaction on the second feature through the second submodule to obtain a fifth feature, and performing feature interaction on the third feature through the third submodule to obtain a sixth feature; S3: fusing the fifth feature and the sixth feature through the fourth fusion module to obtain and store the fourth fusion feature, fusing the fourth feature and the fourth fusion feature through the first fusion module to obtain and store the first fusion feature, fusing the stored first fusion feature and the stored fourth fusion feature through the second fusion module to obtain and store the second fusion feature, and fusing the stored second fusion feature and the sixth feature through the third fusion module to obtain the third fusion feature; specifically including; S31: upsampling the sixth feature to obtain a first upsampling result, performing feature fusion on the first upsampling result and the fifth feature through the fourth fusion submodule to obtain a fourth fused sub-feature, performing feature extraction on the fourth fused sub-feature through the fifth C2f module to obtain a fourth fused feature and store it in the first storage module; S32: upsampling the fourth fused feature to obtain a second upsampling result, performing feature fusion on the second upsampling result and the fourth feature through the first fusion submodule to obtain a first fused sub-feature, performing feature extraction on the first fused sub-feature through the sixth C2f module to obtain a first fused feature and store it in the second storage module; S33: down-sample the first fusion feature in the second storage module to obtain a first down-sampling result, perform feature fusion on the first down-sampling result and the fourth fusion feature of the first storage module through the second fusion sub-module to obtain a second fusion sub-feature, perform feature extraction on the second fusion sub-feature through the seventh C2f module, obtain a second fusion feature and store it in the third storage module; S34: downsampling the second fusion feature of the third storage module to obtain a second downsampling result, performing feature fusion on the second downsampling result and the sixth feature through the third fusion submodule to obtain a third fusion sub-feature, and performing feature extraction on the third fusion sub-feature through the eighth C2f module to obtain a third fusion feature; In step S31, the feature fusion process of the fourth fusion submodule is: assigning values ​​to the first upsampling result and the fifth feature respectively through the first weight module to obtain a first weight feature and a second weight feature, adding the first weight feature and the second weight feature to obtain a fourth fusion sub-feature; In step S32, the feature fusion process of the first fusion submodule is: assigning values ​​to the second upsampling result and the fourth feature respectively through the second weight module to obtain a third weight feature and a fourth weight feature; adding the third weight feature and the fourth weight feature to obtain the first fusion sub-feature; In step S33, the feature fusion process of the second fusion submodule is: assigning values ​​to the first downsampling result and the fourth fusion feature of the first storage module respectively through the third weight module to obtain the fifth weight feature and the sixth weight feature; adding the fifth weight feature and the sixth weight feature to obtain the second fusion sub-feature; In step S34, the feature fusion process of the third fusion submodule is: assigning values ​​to the second downsampling result and the sixth feature respectively through the fourth weight module to obtain the seventh weight feature and the eighth weight feature; adding the seventh weight feature and the eighth weight feature to obtain the third fusion sub-feature; The first weight module assignment process is: normalizing the first weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed first weight matrix to the first upsampling result and the fifth feature respectively to obtain the first weight feature and the second weight feature; The second weight module assignment process is: normalizing the second weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed second weight matrix to the second upsampling result and the fourth feature respectively to obtain the third weight feature and the fourth weight feature; The third weight module assignment process is: normalizing the third weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed third weight matrix to the first downsampling result and the fourth fusion feature of the first storage module respectively to obtain the fifth weight feature and the sixth weight feature; The fourth weight module assignment process is: normalizing the fourth weight matrix, performing Sigmoid function and Split splitting processing in sequence, and assigning the processed fourth weight matrix to the second downsampling result and the sixth feature respectively to obtain the seventh weight feature and the eighth weight feature; S4: Classify and locate the first fusion feature, the second fusion feature, and the third fusion feature through the YOLOv8s detection head network to obtain the target detection result.

2. According to claim 1, a method for infrared target detection by combining group feature interaction and valued feature fusion, It is characterized in that Step S11 is: subjecting the infrared image to first CBR processing, second CBR processing, third CBR processing and maximum pooling calculation in sequence to obtain a first shallow sub-feature, and extracting the first shallow sub-feature through a first C2f module to obtain a first shallow feature; In step S12: the feature extraction process of the second extraction module is: performing CBS processing on the first shallow feature, and performing feature extraction through the second C2f module to obtain the first feature; In step S13, the feature extraction process of the third extraction module is: performing CBS processing on the first feature in the first feature storage module, and extracting the feature through the third C2f module to obtain the second feature; In step S14, the feature extraction process of the fourth extraction module is: CBS processing is performed on the second feature in the second feature storage module, and feature extraction is performed through the fourth C2f module to obtain the third feature; CBS is convolution normalization SiLU activation.

3. According to claim 1, a method for infrared target detection by combining group feature interaction and valued feature fusion, It is characterized in that Step S2 includes: S21: performing convolution and CPE processing on the infrared image features in sequence to obtain and store the first sub-feature; CPE is conditional position encoding; S22: performing LN layer and Linear linearization processing on the first sub-feature in sequence to obtain a first linear feature, a second linear feature, and a third linear feature; S23: performing multi-scale mapping on the first linear feature, the second linear feature, and the third linear feature through the first grouping module to obtain a first splicing feature, a second splicing feature, a third splicing feature, and a fourth splicing feature; S24: performing EA processing on the first splicing feature, the second splicing feature, and the third splicing feature to obtain a first attention feature; EA is efficient attention; S25: Perform CRPE processing on the first attention feature and concatenate it with the fourth concatenation feature to obtain a second attention feature; CRPE is a convolution relative position encoding; S26: Linearize the second attention feature and add it to the stored first sub-feature, and convolve the addition result to obtain the interaction feature.

4. According to claim 3, a method for infrared target detection by combining group feature interaction and valued feature fusion, It is characterized in that Step S23 includes: S231: split the first linear feature to obtain five first linear sub-features L1, L2, L3, L4 and L5, split the second linear feature to obtain five second linear sub-features M1, M2, M3, M4 and M5, split the third linear feature to obtain five third linear sub-features N1, N2, N3, N4 and N5; S232: perform 3×3 convolution, BN layer and Hardswish function processing on L2, M2, and N2 in sequence to obtain L21, M21, and N21 in sequence; perform 5×5 convolution, BN layer and Hardswish function processing on L3, M3, and N3 in sequence to obtain L31, M31, and N31 in sequence; perform 7×7 convolution, BN layer and Hardswish function processing on L4, M4, and N4 in sequence to obtain L41, M41, and N41 in sequence; S233: splicing L1, L21, L31 and L41 to obtain a first splicing feature, splicing M1, M21, M31 and M41 to obtain a second splicing feature, and splicing N1, N21, N31 and N41 to obtain a third splicing feature; S234: concatenate L5, M5 and N5 to obtain a fourth concatenated sub-feature, and sequentially perform DWConv, LN layer and Hardswish processing on the fourth concatenated sub-feature to obtain a fourth concatenated feature.

Images