A YOLO-based infrared image target detection method

Abstract

The present invention relates to the field of infrared image target detection technology, and provides an infrared image target detection method based on YOLO, comprising: using CSPDarknet as a backbone network, including a CBS module and a C2f module; performing CBS processing on an infrared image through the CBS module to obtain a first CBS result, performing feature extraction on the first CBS result through the C2f module to obtain a first feature, a second feature, and a third feature; processing the first feature, the second feature, and the third feature to obtain a first fusion feature, a second fusion feature, and a third fusion feature; and classifying and locating the first fusion feature, the second fusion feature, and the third fusion feature through a YOLOv8s detection head network, respectively, to obtain a detection result set, and to obtain a target detection result. The present invention realizes an infrared image target detection method with lower computational complexity and higher detection accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared image target detection, and in particular to an infrared image target detection method based on YOLO. Background Art

[0002] The characteristics of infrared images, such as distance, noise, poor spatial resolution, and sensitivity to ambient temperature changes, limit the construction of infrared image models. Traditional target detection algorithms rely heavily on manually designed features and region proposal methods, including techniques based on sliding windows and image segmentation. Some studies have explored the possibility of detecting targets in infrared images using methods such as spatial filtering, frequency domain filtering, and sparse representation. However, these traditional infrared image target detection methods are limited to specific application scenarios, have slow recognition speeds, weak generalization capabilities, and are difficult to extract important features in multi-scenario and real-time detection applications.

[0003] The YOLO (You Only Look Once) series of algorithms use a single neural network to complete target detection end-to-end, which has the advantages of real-time and simplicity, but also has some disadvantages. First, the detection effect of the YOLO series of algorithms on small and dense targets is not as good as other methods, and it is easy to miss or misdetect. Secondly, because the YOLO series of algorithms use a single-scale feature map for target detection, the performance is not ideal when processing multi-scale targets. In addition, the YOLO series of algorithms has relatively low positioning accuracy for targets, which is easy to cause deviation or misalignment of the bounding box.

[0004] YOLOv8s is a target detection algorithm in the YOLO series of algorithms, which is famous for its simple and efficient design and detection speed. However, the generalization ability of YOLOv8s in some complex scenarios needs to be improved, and it still has certain deficiencies in small target detection. In addition, the C2f module in the feature fusion part of YOLOv8s processes half of the feature tensor, which will lead to insufficient feature fusion and inadequate feature summary after fusion. In addition, the slicing and splicing operations of the feature tensor are not friendly to calculation on low-computing hardware. Summary of the invention

[0005] The present invention aims to solve the technical problems of insufficient feature fusion of YOLOv8s, inadequate feature summary after fusion, unfriendly calculation on low-computing hardware and poor small target detection. To this end, the present invention provides an infrared image target detection method based on YOLO, which achieves lower parameter amount, lower calculation amount and higher infrared image target detection accuracy compared with YOLOv8s.

[0006] The present invention provides an infrared image target detection method based on YOLO, comprising the following steps:

[0007] S1, using CSPDarknet as the backbone network, including CBS module and C2f module;

[0008] S2, performing CBS processing on the infrared image through the CBS module to obtain a first CBS result, and performing feature extraction on the first CBS result through the C2f module to obtain a first feature, a second feature, and a third feature;

[0009] S3, performing CBS processing, GSConv processing, and DWCBS processing on the first feature, the second feature, and the third feature to obtain a first fusion feature, a second fusion feature, and a third fusion feature;

[0010] S4. Classify and locate the first fusion feature, the second fusion feature and the third fusion feature respectively through the YOLOv8s detection head network, and obtain the first target detection result, the second target detection result and the third target detection result accordingly. The first target detection result, the second target detection result and the third target detection result form a detection result set to obtain the target detection result.

[0011] According to a YOLO-based infrared image target detection method provided by the present invention, step S2 includes:

[0012] S21: performing CBS processing on the infrared image through the CBS module to obtain a first CBS result, and performing feature extraction on the first CBS result through the first C2f module to obtain a first shallow feature;

[0013] S22: extracting the first shallow feature through the second C2f module to obtain a first feature, and saving the first feature to a first feature storage module;

[0014] S23: Perform CBS processing on the first feature in the first feature storage module, perform feature extraction through the third C2f module to obtain a second feature, and save the second feature to the second feature storage module;

[0015] S24: Perform CBS processing on the second feature in the second feature storage module, perform feature extraction through the fourth C2f module to obtain the second deep feature, and perform feature extraction on the second deep feature through the SPPF module to obtain the third feature.

[0016] According to a YOLO-based infrared image target detection method provided by the present invention, step S3 includes:

[0017] S31: upsampling the third feature to achieve the same size as the second feature, obtaining a fourth feature, concatenating the fourth feature with the second feature to obtain a first concatenated result, performing feature fusion on the first concatenated result through a first fusion module to obtain a fourth fused feature, and saving the fourth fused feature to the first storage module;

[0018] S32: upsampling the fourth fused feature to achieve the same size as the first feature to obtain a fifth feature, splicing the fifth feature with the first feature to obtain a second splicing result, performing feature fusion on the second splicing result through a second fusion module to obtain a first fused feature, and saving the first fused feature to a second storage module;

[0019] S33: performing CBS processing on the first fused feature in the second storage module, and splicing it with the fourth fused feature in the first storage module to obtain a third splicing result, performing feature fusion on the third splicing result through the third fusion module to obtain a second fused feature, and saving the second fused feature to the third storage module;

[0020] S34: Perform CBS processing on the second fused feature in the third storage module, and concatenate it with the third feature to obtain a fourth concatenation result, and perform feature fusion on the fourth concatenation result through a fourth fusion module to obtain a third fused feature.

[0021] According to a YOLO-based infrared image target detection method provided by the present invention, in step S31, the feature fusion process of the first fusion module includes:

[0022] S311: performing CBS processing on the first splicing result to obtain a first splicing layer, and saving the first splicing layer to a first splicing storage module;

[0023] S312: Perform GSConv processing on the first splicing layer to obtain a first splicing feature, and perform DWCBS processing on the first splicing layer to obtain a first processing result; perform group convolution, rearrangement, CA mechanism and convolution processing on the first splicing feature in sequence through the first submodule to obtain a fourth sub-feature; the first fusion submodule sums the fourth sub-feature and the first processing result to obtain a fourth fusion sub-feature;

[0024] S313: Splicing the fourth fusion sub-feature and the first splicing layer in the first splicing storage module to obtain a fifth splicing result, and performing CBS processing on the fifth splicing result to obtain a fourth fusion feature.

[0025] According to a YOLO-based infrared image target detection method provided by the present invention, in step S32, the feature fusion process of the second fusion module includes:

[0026] S321: Perform CBS processing on the second splicing result to obtain a second splicing layer, and save the second splicing layer to a second splicing storage module;

[0027] S322: Perform GSConv processing on the second splicing layer to obtain a second splicing feature, and perform DWCBS processing on the second splicing layer to obtain a second processing result; perform group convolution, rearrangement, CA mechanism and convolution processing on the second splicing feature in sequence through the second submodule to obtain a first sub-feature; sum the first sub-feature and the second processing result through the second fusion submodule to obtain a first fusion sub-feature;

[0028] S323: Splice the first fusion sub-feature and the second splicing layer in the second splicing storage module to obtain a sixth splicing result, and perform CBS processing on the sixth splicing result to obtain a first fusion feature.

[0029] According to a YOLO-based infrared image target detection method provided by the present invention, in step S33, the feature fusion process of the third fusion module includes:

[0030] S331: performing CBS processing on the third splicing result to obtain a third splicing layer, and saving the third splicing layer to a third splicing storage module;

[0031] S332: Perform GSConv processing on the third splicing layer to obtain a third splicing feature, and perform DWCBS processing on the third splicing layer to obtain a third processing result; perform group convolution, rearrangement, CA mechanism and convolution processing on the third splicing feature in sequence through the third submodule to obtain a second sub-feature; sum the second sub-feature and the third processing result through the third fusion submodule to obtain a second fusion sub-feature;

[0032] S333: Splice the second fusion sub-feature and the third splicing layer in the third splicing storage module to obtain a seventh splicing result, and perform CBS processing on the seventh splicing result to obtain a second fusion feature.

[0033] According to a YOLO-based infrared image target detection method provided by the present invention, in step S34, the feature fusion process of the fourth fusion module includes:

[0034] S341: Perform CBS processing on the fourth splicing result to obtain a fourth splicing layer, and save the fourth splicing layer to a fourth splicing storage module;

[0035] S342: performing GSConv processing on the fourth splicing layer to obtain a fourth splicing feature, and performing DWCBS processing on the fourth splicing layer to obtain a fourth processing result; performing group convolution, rearrangement, CA mechanism and convolution processing on the fourth splicing feature in sequence through the fourth submodule to obtain a third sub-feature; summing the third sub-feature and the fourth processing result through the fourth fusion submodule to obtain a third fusion sub-feature;

[0036] S343: Splice the third fusion sub-feature and the fourth splicing layer in the fourth splicing storage module to obtain an eighth splicing result, and perform CBS processing on the eighth splicing result to obtain a third fusion feature.

[0037] According to a YOLO-based infrared image target detection method provided by the present invention, step S4 includes:

[0038] S41: Classifying and locating the first fusion feature through the first detection head to obtain a first target detection result;

[0039] S42: Classifying and locating the second fusion feature through the second detection head to obtain a second target detection result;

[0040] S43: classifying and locating the third fusion feature by a third detection head to obtain a third target detection result;

[0041] S44: The first target detection result, the second target detection result, and the third target detection result are formed into a set to obtain a target detection result.

[0042] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0043] 1. The present invention can effectively extract deep feature information in infrared images through the backbone network CSPDarknet, thereby achieving efficient and accurate target detection.

[0044] 2. The present invention performs multi-scale feature fusion on the first feature, the second feature and the third feature extracted by the backbone network, fully performs feature fusion and feature summary on the basis of reducing the number of parameters and computational costs, thereby improving the quality and efficiency of feature fusion and improving the accuracy of infrared image target detection.

[0045] 3. The present invention classifies and locates the first fusion feature, the second fusion feature and the third fusion feature through the YOLOv8s detection head network to achieve multi-scale detection of the target.

[0046] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a schematic diagram of the process provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0050] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0051] Combine the following Figure 1 The present invention is further described in detail, and a YOLO-based infrared image target detection method of the present invention is described.

[0052] In this embodiment, Figure 1 As shown, a YOLO-based infrared image target detection method is provided, comprising the following steps:

[0053] S1: The feature information in the infrared image is extracted by using CSPDarknet (cross-stage local network) in a layer-by-layer convolution manner to obtain the first feature, the second feature, and the third feature;

[0054] The CSPDarknet of YOLOv8s is used as the backbone network to extract the feature information of the input infrared image. The CSPDarknet includes 5 CBS modules (used to perform CBS processing operations), 4 C2f (Convolution to Feature) modules and 1 SPPF (Fast Spatial Pyramid Pooling) module. The CBS processing is to sequentially pass through a set of convolutions with a convolution kernel size of 1×1, BN (batch normalization) layers and SiLU (Sigmoid Linear Unit) activation functions to achieve the retention of feature maps and feature information.

[0055] The four C2f modules are the same and are all used for deep feature extraction. Specifically, the C2f module performs feature transformation on the input infrared image through two convolutional layers to extract features of different levels and abstraction levels in the infrared image; the SPPF module can rescale input images of any size to a fixed size and generate a feature vector of fixed length, reducing the amount of calculation, realizing feature extraction and encoding of infrared images at different scales, helping to extract the most significant features in the image and better capture useful feature information.

[0056] Wherein step S2 comprises:

[0057] S21: Perform CBS processing on the infrared image to obtain a first CBS result, and perform feature extraction on the first CBS result through a first C2f module to obtain a first shallow feature;

[0058] In step S21, the first C2f module feature extraction process is: CBS processing is performed on the first CBS result, and feature extraction is performed through the first C2f module to obtain a first shallow feature;

[0059] CBS processing is performed on the infrared image to achieve preliminary feature mapping and channel amplification. The number of channels of the first CBS result is 64; the number of channels of the first shallow feature is 128.

[0060] S22: extracting the first shallow feature through the second C2f module to obtain a first feature, and saving the first feature to a first feature storage module;

[0061] In step S22, the feature extraction process of the second C2f module is: performing CBS processing on the first shallow feature, and extracting the feature through the second C2f module to obtain the first feature;

[0062] The number of channels of the first feature is 128.

[0063] S23: extracting the first feature in the first feature storage module through the third C2f module to obtain a second feature, and saving the second feature to the second feature storage module;

[0064] In step S23, the feature extraction process of the third C2f module is: performing CBS processing on the first feature in the first feature storage module, and extracting the feature through the third C2f module to obtain the second feature;

[0065] The number of channels of the second feature is 256.

[0066] S24: extracting the second feature in the second feature storage module through the fourth C2f module to obtain a third feature;

[0067] In step S24, the feature extraction process of the fourth C2f module is: CBS processing is performed on the second feature in the second feature storage module, and feature extraction is performed through the fourth C2f module to obtain the second deep feature, and feature extraction is performed on the second deep feature through the SPPF module to obtain the third feature.

[0068] The number of channels of the second deep feature is 512; the number of channels of the third feature remains unchanged from the second deep feature, and the number of channels of the third feature is 512.

[0069] S3: upsampling the third feature, and performing a first fusion with the second feature to obtain a fourth fusion feature and storing it at the same time, upsampling the fourth fusion feature, and performing a second fusion with the first feature to obtain a first fusion feature and storing it at the same time, performing CBS processing on the stored first fusion feature, and performing a third fusion with the stored fourth fusion feature to obtain a second fusion feature and storing it at the same time, performing CBS processing on the stored second fusion feature, and performing a fourth fusion with the third feature to obtain a third fusion feature;

[0070] The neck network of YOLOv8s consists of upsampling, downsampling and four C2f modules. The present invention improves the neck network of YOLOv8s by replacing the four C2f modules in the neck network of YOLOv8s with four fusion modules (a first fusion module, a second fusion module, a third fusion module and a fourth fusion module) to form a new neck network. The first feature, the second feature and the third feature output by the backbone network CSPDarknet are fused at multiple scales through the new neck network, and shallow textures, edge details and deep semantic information (such as shape and size) interact with each other to construct more diverse and representative feature representations, thereby further enhancing the expression and generalization capabilities of the model.

[0071] The structures and functions of the four fusion modules are the same. They adopt the CSP (cross-stage partial connection) structure. On the one hand, in tensor processing, they avoid the frequent slicing and splicing of feature tensors by the C2f module, which is more friendly to low-computing power hardware. In addition, they integrate the features of all channels, retain more feature information, and fuse features more accurately and efficiently. The summarized features are of higher quality. On the other hand, they continue to be lightweight and reduce the number of parameters and the amount of calculation.

[0072] Step S3 specifically includes:

[0073] S31: upsampling the third feature to achieve the same size as the second feature, obtaining a fourth feature, concatenating the fourth feature with the second feature to obtain a first concatenated result, performing feature fusion on the first concatenated result through a first fusion module to obtain a fourth fused feature, and saving the fourth fused feature to the first storage module;

[0074] S32: upsampling the fourth fused feature to achieve the same size as the first feature to obtain a fifth feature, splicing the fifth feature with the first feature to obtain a second splicing result, performing feature fusion on the second splicing result through a second fusion module to obtain a first fused feature, and saving the first fused feature to a second storage module;

[0075] S33: performing CBS processing on the first fused feature in the second storage module, and splicing it with the fourth fused feature in the first storage module to obtain a third splicing result, performing feature fusion on the third splicing result through the third fusion module to obtain a second fused feature, and saving the second fused feature to the third storage module;

[0076] S34: Perform CBS processing on the second fused feature in the third storage module, and concatenate it with the third feature to obtain a fourth concatenation result, and perform feature fusion on the fourth concatenation result through a fourth fusion module to obtain a third fused feature.

[0077] After upsampling the high semantic features, the length and width of the high semantic features are made the same as the corresponding low semantic features; the splicing is performed in the concat (feature connection) way, and different features are spliced ​​together according to the channel dimension to make full use of feature information from different levels and realize multi-scale feature fusion; the number of channels of the fourth fusion feature is 256, the number of channels of the first fusion feature is 128, the number of channels of the second fusion feature is 256, and the number of channels of the third fusion feature is 512.

[0078] Furthermore, each fusion module includes one fusion sub-module (i.e., the first fusion module includes the first fusion sub-module, the second fusion module includes the second fusion sub-module, the third fusion module includes the third fusion sub-module, and the fourth fusion module includes the fourth fusion sub-module), wherein the structure and function of each fusion sub-module are the same, and feature fusion is fully performed on the basis of maintaining lightweight to improve feature quality.

[0079] In step S31, the feature fusion process of the first fusion module includes:

[0080] S311: performing CBS processing on the first splicing result to obtain a first splicing layer, and saving the first splicing layer to a first splicing storage module;

[0081] S312: summing up the features of the first concatenated layer through the first fusion submodule to obtain a fourth fusion sub-feature;

[0082] S313: splicing the fourth fusion sub-feature and the first splicing layer in the first splicing storage module to obtain a fifth splicing result, and performing CBS processing on the fifth splicing result to obtain a fourth fusion feature;

[0083] In step S32, the feature fusion process of the second fusion module includes:

[0084] S321: Perform CBS processing on the second splicing result to obtain a second splicing layer, and save the second splicing layer to a second splicing storage module;

[0085] S322: Sum the features of the second concatenated layer through the second fusion submodule to obtain the first fusion sub-feature;

[0086] S323: Splicing the first fusion sub-feature and the second splicing layer in the second splicing storage module to obtain a sixth splicing result, and performing CBS processing on the sixth splicing result to obtain a first fusion feature;

[0087] In step S33, the feature fusion process of the third fusion module includes:

[0088] S331: performing CBS processing on the third splicing result to obtain a third splicing layer, and saving the third splicing layer to a third splicing storage module;

[0089] S332: summing up the features of the third concatenated layer through the third fusion submodule to obtain a second fusion sub-feature;

[0090] S333: Splicing the second fusion sub-feature and the third splicing layer in the third splicing storage module to obtain a seventh splicing result, and performing CBS processing on the seventh splicing result to obtain a second fusion feature;

[0091] In step S34, the feature fusion process of the fourth fusion module includes:

[0092] S341: Perform CBS processing on the fourth splicing result to obtain a fourth splicing layer, and save the fourth splicing layer to a fourth splicing storage module;

[0093] S342: summing up the features of the fourth concatenated layer through the fourth fusion submodule to obtain a third fusion sub-feature;

[0094] S343: Splice the third fusion sub-feature and the fourth splicing layer in the fourth splicing storage module to obtain an eighth splicing result, and perform CBS processing on the eighth splicing result to obtain a third fusion feature.

[0095] The fusion submodule (including the first fusion submodule, the second fusion submodule, the third fusion submodule and the fourth fusion submodule) is a bottleneck structure. On the one hand, GSConv (group shuffle convolution) is used instead of ordinary convolution, which can maintain the lightweight advantage of depthwise separable convolution as much as possible on the basis of the closest calculation accuracy of ordinary convolution. On the other hand, it makes the feature fusion more thorough and the summarized feature quality higher.

[0096] Furthermore, each fusion submodule includes one submodule (i.e., the first fusion submodule includes the first submodule, the second fusion submodule includes the second submodule, the third fusion submodule includes the third submodule, and the fourth fusion submodule includes the fourth submodule). The structure and function of each submodule are the same. By adding the attention mechanism, the feature fusion is more complete and accurate.

[0097] In step S312, the process of summing up the features of the first fusion submodule includes:

[0098] S3121: Perform GSConv processing on the first splicing layer to obtain a first splicing feature, and perform DWCBS processing on the first splicing layer to obtain a first processing result;

[0099] S3122: extracting features from the first splicing feature through the first submodule to obtain a fourth sub-feature;

[0100] S3123: summing the fourth sub-feature and the first processing result to obtain a fourth fused sub-feature;

[0101] In step S322, the feature summing process of the second fusion submodule includes:

[0102] S3221: performing GSConv processing on the second splicing layer to obtain a second splicing feature, and performing DWCBS processing on the second splicing layer to obtain a second processing result;

[0103] S3222: extracting the second splicing feature through the second sub-module to obtain the first sub-feature;

[0104] S3223: summing the first sub-feature and the second processing result to obtain a first fused sub-feature;

[0105] In step S332, the feature summing process of the third fusion submodule includes:

[0106] S3321: performing GSConv processing on the third splicing layer to obtain a third splicing feature, and performing DWCBS processing on the third splicing layer to obtain a third processing result;

[0107] S3322: extracting features from the third splicing feature through the third sub-module to obtain a second sub-feature;

[0108] S3323: summing the second sub-feature and the third processing result to obtain a second fused sub-feature;

[0109] In step S342, the feature summing process of the fourth fusion submodule includes:

[0110] S3421: performing GSConv processing on the fourth splicing layer to obtain a fourth splicing feature, and performing DWCBS processing on the fourth splicing layer to obtain a fourth processing result;

[0111] S3422: extracting features from the fourth splicing feature through the fourth sub-module to obtain a third sub-feature;

[0112] S3423: Sum the third sub-feature and the fourth processing result to obtain a third fused sub-feature.

[0113] DWCBS processing is a process of sequentially passing through a set of depth-separable convolution, BN layer and SiLU activation function to achieve the retention of feature maps and feature information. Compared with CBS, it reduces the required parameters and achieves the separation of channels and regions.

[0114] The GSConv processing process is as follows: perform CBS processing on the input features to obtain and save feature map GS1, perform DWCBS processing on feature map GS1 to obtain feature map GS2, concat feature map GS1 and feature map GS2 along the channel dimension to obtain feature map GS3, and perform shuffle (rearrange) operation on feature map GS3, that is, rearrange the feature maps in corresponding positions of feature map GS1 and feature map GS2 to obtain the final output features.

[0115] In step S3122, the first submodule feature extraction process is: performing group convolution, rearrangement, CA mechanism and convolution processing on the first splicing feature in sequence to obtain the fourth sub-feature;

[0116] In step S3222, the second submodule feature extraction process is: performing group convolution, rearrangement, CA mechanism and convolution processing on the second splicing feature in sequence to obtain the first sub-feature;

[0117] In step S3322, the feature extraction process of the third submodule is: performing group convolution, rearrangement, CA mechanism and convolution processing on the third concatenated feature in sequence to obtain the second sub-feature;

[0118] In step S3422, the fourth sub-module feature extraction process is: performing group convolution, rearrangement, CA mechanism and convolution processing on the fourth splicing feature in sequence to obtain the third sub-feature.

[0119] The present invention proposes a PD (Parameter Diversify) method, which is specifically implemented as follows: the submodules (including the first submodule, the second submodule, the third submodule and the fourth submodule) first perform group convolution on the input features to achieve feature mapping for each channel of the input features, and then re-arrange the pixels at each pixel position. Group The number of pixel features are reorganized into the same feature map and arranged as The square feature block is then assigned attention by the CA (coordinate attention) mechanism, and then processed by a convolution with a step size of 3 and a convolution kernel size of 3×3, and the feature tensor after convolution processing with diversified parameters is output.

[0120] Through the PD method, the parameters are expanded so that the neural network can better fit the objective function and improve the diversity of feature extraction. At the same time, the CA mechanism is increased to achieve effective attention to multi-scale feature fusion, thereby improving the quality and efficiency of feature fusion and improving the accuracy of infrared image target detection.

[0121] Furthermore, the input feature Divided along the channel dimension Group, where is the height of the feature map, is the width of the feature map, Represents the shape of the feature tensor.

[0122] The specific feature mapping process is:

[0123]

[0124] in, represents the feature tensor after channel amplification, is the convolution operation, Indicates the number of output channels, Indicates the number of channels after channel amplification, Indicates Feature map of the channel.

[0125] Perform group convolution on the input features to achieve feature mapping for each layer of the input features, increasing the number of channels compared to the original times, that is Then, through the rearrangement operation, the Group The pixel features are reorganized into the same feature map, forming a shape of The square feature blocks are rearranged. Channel feature map for:

[0126]

[0127] in, It means that the feature tensor after channel amplification is rearranged. Represents the square feature block obtained by transforming the feature tensor after channel amplification, which is calculated as:

[0128]

[0129] in, Represents the feature points taken from the feature tensor after channel amplification, Indicates the vertical coordinate of the feature point, the value range is , Indicates the horizontal coordinate of the feature point, the value range is .

[0130] The local feature interaction and extraction are realized in the initial convolution, and then the parameter diversification mapping is performed. Subsequently, the global feature information interaction of the feature map is completed through the CA mechanism. Mapping Processing for:

[0131]

[0132] in, represents the step size, that is, the distance of the convolution translation, Indicates that the step size is equal to the convolution kernel size Convolution operation, Indicates that CA mechanism processing is performed. Indicates the use of convolution to amplify feature channels.

[0133] The attention weight is weighted to each pixel feature. In the subsequent convolution, the convolution kernel slides without overlap, which is equivalent to assigning a new convolution kernel to each receptive field through the attention weight, avoiding the weight sharing of the standard convolution, thereby improving the performance of the algorithm, thereby improving the efficiency of feature fusion and improving the accuracy of target detection.

[0134] Furthermore, the CA mechanism is as follows: average pooling operations are performed on the input features in the A and B dimensions respectively, the two pooling results are concatenated at the same latitude, and a 1×1 convolution is used to perform two-dimensional feature interaction. The interaction results are successively processed by the BN layer and nonlinear transformation, and then tensor splitting is performed at the concatenation position of the two-dimensional feature tensors to divide them into two-dimensional feature tensors. The two-dimensional feature tensors are then activated by the sigmoid function to obtain the two-dimensional attention weights. Finally, the two-dimensional attention weights are jointly weighted to the input feature tensor to complete the attention assignment. The CA mechanism can complement the input features and enhance the representation of objects of interest.

[0135] S4: Classify and locate the first fusion feature, the second fusion feature, and the third fusion feature through the YOLOv8s detection head network to obtain the target detection result.

[0136] Step S4 includes:

[0137] S41: Classifying and locating the first fusion feature through the first detection head to obtain a first target detection result;

[0138] S42: Classifying and locating the second fusion feature through the second detection head to obtain a second target detection result;

[0139] S43: classifying and locating the third fusion feature by a third detection head to obtain a third target detection result;

[0140] S44: The first target detection result, the second target detection result, and the third target detection result are formed into a set to obtain a target detection result.

[0141] The YOLOv8s detection head network includes three detection heads (including the first detection head, the second detection head and the third detection head). The first detection head is a decoupled two-branch. The first fusion feature is input into one branch, and is processed by CBS, ordinary convolution, and Bbox Loss (positioning loss) in sequence to complete the positioning of small targets. At the same time, the first fusion feature is input into another branch, and is processed by CBS, ordinary convolution, and Cls Loss (classification loss) in sequence to complete the classification of small targets.

[0142] The second detection head is composed of two decoupled branches. The second fused features are input into one branch, and processed by CBS, ordinary convolution, and Bbox Loss (positioning loss) in sequence to complete the positioning of the target. At the same time, the second fused features are input into another branch, and processed by CBS, ordinary convolution, and Cls Loss (classification loss) in sequence to complete the classification of the target.

[0143] The third detection head consists of two decoupled branches. The third fusion feature is input into one branch, and processed by CBS, ordinary convolution, and Bbox Loss (positioning loss) in sequence to complete the positioning of large targets. At the same time, the third fusion feature is input into another branch, and processed by CBS, ordinary convolution, and Cls Loss (classification loss) in sequence to complete the classification of large targets.

[0144] A picture has multiple objects of different sizes (including small objects, medium objects and large objects). The YOLOv8s detection head network will assign the object detection tasks of different sizes to the three detection heads for classification and positioning. Finally, the detection results of the three detection heads are formed into a set to obtain the final object detection result.

[0145] In a specific embodiment:

[0146] A1: Build infrared image target detection training set and validation set;

[0147] In the first stage of model training, the method of the present invention uses the FLIR thermal infrared image target detection dataset to train the target detection network architecture (including the backbone network, the neck network and the detection head network) designed by the present invention.

[0148] The FLIR thermal infrared dataset is obtained by a thermal imager installed on a vehicle. The forward-looking infrared thermal imager FLIRTau2 is used, and the parameters are set to 13mm f / 1.0, 45-degree horizontal field of view (HFOV) and 37-degree vertical field of view (VFOV) to obtain thermal images. Both cameras are running in default mode. The infrared thermal imager FLIR Tau2 dataset contains a total of 14,452 annotated thermal images, of which 10,228 are from short videos and 4,224 are from 144 seconds of continuous video.

[0149] We conducted training experiments on objects with a reasonable number of labels for the three categories in the dataset, and finally selected "people", "bicycles" and "cars" as experimental objects. The number of target annotations in the training set was 22,372 for "people", 3,986 for "bicycles" and 41,260 for "cars", and the number of "people", "bicycles" and "cars" in the validation set was 5,779 for "people", 471 for "bicycles" and 5,432 for "cars". The training used cocodataset-detection-eval (COCO dataset, which has become the standard test platform for image captions) as the accuracy evaluation standard.

[0150] A2: Train the network model;

[0151] The overall network model is trained based on the Pytorch deep learning framework, and the performance of the network model is verified using the validation set. During the experiment, the NVIDIA GeForce RTX 3080Ti graphics card is used for training and testing, and the experimental parameter settings are shown in Table 1.

[0152] Table 1 Experimental parameter settings

[0153]

[0154] Among them, epochs represents the number of model training rounds, batch size represents the size of each batch of data selected during training, workers represents the number of subprocesses used when loading data, image size represents the image size, amp represents automatic mixed precision training, mosaic (probability) represents data enhancement, optimizer represents optimizer, finallr represents feedforward neural network learning rate, momentum represents momentum, weight_decay represents weight decay, which aims to reduce the complexity of the model and prevent overfitting, warmup_epochs represents the number of training rounds for model warmup, and warmup_momentum represents the momentum setting during warmup training.

[0155] A3: Build a new infrared image target detection dataset;

[0156] Fine-tune the network model training parameters such as batch size, final lr, momentum, etc., and train the network model constructed by the algorithm multiple times to obtain better training results, so that the network model has better target detection performance.

[0157] A4: Input the trained target detection model to obtain the infrared target detection result.

[0158] A new infrared image target detection dataset is constructed by mixing existing public datasets and infrared imaging equipment to obtain infrared images and manually annotate image targets. A PC (personal computer) device is selected to input the images of the newly constructed infrared image target detection dataset into the trained network model designed by the present invention to obtain the final infrared image target detection results. The experimental results compared with the YOLO series are shown in Table 2.

[0159] Table 2 Experimental results compared with the YOLO series

[0160]

[0161] Among them, mAP(0.5) represents the average accuracy of target detection when the intersection-over-union ratio is 0.5, mAP(0.5:0.95) represents the average accuracy of target detection when the intersection-over-union ratio is 0.5, 0.55, 0.6...0.95, mAR(0.5) represents the average recall rate, Parameters(M) represents the number of network model parameters, and FLOPs(G) represents the amount of network model calculation.

[0162] The higher the mAP(0.5), mAP(0.5:0.95) and mAR(0.5) values ​​are, the higher the accuracy of target detection is; the lower the Parameters(M) and FLOPs(G) values ​​are, the lighter the network model of target detection is.

[0163] The method of the present invention is trained and verified with a number of mainstream YOLO series algorithms under the same data set to obtain comparative experimental results of algorithm model detection accuracy data and model scale data.

[0164] The experimental data in Table 2 show that the network model of the infrared image target detection method proposed in the present invention is more lightweight and has a more obvious accuracy advantage.

[0165] Furthermore, ablation experiments were performed on various improvements of the fusion module in the infrared image target detection method proposed in the present invention to verify the effectiveness and advancement of the method.

[0166] Taking the original C2f module of YOLOv8s using ordinary convolution as the benchmark experimental control, we replaced the original convolution with CBAM (convolutional block attention module), EMA (efficient multi-scale attention) and CA mechanism using the PD method for control experiments, and conducted control experiments on whether the CA mechanism uses the PD method; at the same time, the present invention replaces the original C2f module of YOLOv8s with a fusion module including a submodule (CAPD, Coordinate Attention Parameter Diversify) as an experimental control, and then replaces the submodules with GSConv, CBAMPD (CBAM using the PD method), EMAPD (EMA using the PD method) and CA for control experiments, and the control experimental results are shown in Table 3.

[0167] Table 3 Control experiment results

[0168]

[0169] The experimental data in Table 3 show that replacing ordinary convolution with the attention mechanism can improve detection accuracy, and the CAPD module using the PD method has the most significant improvement on the results. It also verifies that the CA mechanism is more compatible with the PD method than other attention mechanisms, and can more effectively achieve higher growth effects for infrared target detection. However, this also results in a certain sacrifice in the amount of parameters and calculations.

[0170] The above results are compared with the CSP structure using GSConv. Although the target detection accuracy of all attention control groups has slightly decreased, the most obvious effect is the reduction of the number of parameters and calculations in all attention mechanism experimental groups. However, the fusion module proposed in this invention still achieves a higher detection effect than the original C2f module (using ordinary convolution). This verifies the effectiveness and advancement of the method proposed in this invention.

[0171] The beneficial effects of the present invention are as follows: the infrared image target detection method based on YOLO proposed by the present invention inherits the backbone network and detection head network of YOLOv8s, improves the neck network of YOLOv8s, realizes effective attention to multi-scale feature fusion by using the CA mechanism, and solves the problem of parameter sharing of ordinary convolution by using the PD method, realizes parameter diversification, and finally improves the quality and efficiency of feature fusion, and improves the detection accuracy of infrared targets. Replacing the C2f module with the fusion module of GSConv not only suppresses the increase in the amount of parameters and calculations brought by the submodule (CAPD), realizes the incremental suppression of parameters and calculations, but also does not affect the detection accuracy of the infrared image target of the entire method too much. An infrared image target detection method with lower parameter amount and calculation amount than YOLOv8s but higher detection accuracy is realized.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A YOLO-based infrared image target detection method, characterized in that: The steps include: S1, using CSPDarknet as the backbone network, including CBS module and C2f module; S2, performing CBS processing on the infrared image through the CBS module to obtain a first CBS result, and performing feature extraction on the first CBS result through the C2f module to obtain a first feature, a second feature, and a third feature; S3, performing CBS processing, GSConv processing and DWCBS processing on the first feature, the second feature and the third feature to obtain a first fusion feature, a second fusion feature and a third fusion feature; specifically including: S31: upsampling the third feature to achieve the same size as the second feature, obtaining a fourth feature, concatenating the fourth feature with the second feature to obtain a first concatenated result, performing feature fusion on the first concatenated result through a first fusion module to obtain a fourth fused feature, and saving the fourth fused feature to the first storage module; S32: upsampling the fourth fused feature to achieve the same size as the first feature to obtain a fifth feature, splicing the fifth feature with the first feature to obtain a second splicing result, performing feature fusion on the second splicing result through a second fusion module to obtain a first fused feature, and saving the first fused feature to a second storage module; S33: performing CBS processing on the first fused feature in the second storage module, and splicing it with the fourth fused feature in the first storage module to obtain a third splicing result, performing feature fusion on the third splicing result through the third fusion module to obtain a second fused feature, and saving the second fused feature to the third storage module; S34: performing CBS processing on the second fused feature in the third storage module, and concatenating the second fused feature with the third feature to obtain a fourth concatenated result, and performing feature fusion on the fourth concatenated result through a fourth fusion module to obtain a third fused feature; Among them, the first fusion module, the second fusion module, the third fusion module, and the fourth fusion module are fusion modules including submodules, wherein: The submodule first performs group convolution on the input features to achieve feature mapping for each channel of the input features, and then re-arranges the pixels at each pixel position. Group The number of pixel features are reorganized into the same feature map and arranged as The square feature block is then processed by convolution after attention assignment by the CA mechanism, and the feature tensor after convolution with diversified parameters is output; The CA mechanism is as follows: average pooling operations are performed on the input features in the A and B dimensions respectively, the two pooling results are spliced ​​in the same dimension, and the two-dimensional feature interaction is performed by convolution. The interaction results are processed by the BN layer and nonlinear transformation in sequence, and the tensor is split at the splicing position of the two-dimensional feature tensors to be divided into two-dimensional feature tensors. The sigmoid function is then activated on the two-dimensional feature tensors to obtain the two-dimensional attention weights. Finally, the two-dimensional attention weights are weighted together to the input feature tensor to complete the attention assignment. S4. Classify and locate the first fusion feature, the second fusion feature and the third fusion feature respectively through the YOLOv8s detection head network, and obtain the first target detection result, the second target detection result and the third target detection result accordingly. The first target detection result, the second target detection result and the third target detection result form a detection result set to obtain the target detection result.

2. The infrared image target detection method based on YOLO according to claim 1, characterized in that: Step S2 includes: S21: performing CBS processing on the infrared image through the CBS module to obtain a first CBS result, and performing feature extraction on the first CBS result through the first C2f module to obtain a first shallow feature; S22: extracting the first shallow feature through the second C2f module to obtain a first feature, and saving the first feature to a first feature storage module; S23: Perform CBS processing on the first feature in the first feature storage module, perform feature extraction through the third C2f module to obtain a second feature, and save the second feature to the second feature storage module; S24: Perform CBS processing on the second feature in the second feature storage module, perform feature extraction through the fourth C2f module to obtain the second deep feature, and perform feature extraction on the second deep feature through the SPPF module to obtain the third feature.

3. The infrared image target detection method based on YOLO according to claim 1, characterized in that: In step S31, the feature fusion process of the first fusion module includes: S311: performing CBS processing on the first splicing result to obtain a first splicing layer, and saving the first splicing layer to a first splicing storage module; S312: Perform GSConv processing on the first splicing layer to obtain a first splicing feature, and perform DWCBS processing on the first splicing layer to obtain a first processing result; perform group convolution, rearrangement, CA mechanism and convolution processing on the first splicing feature in sequence through the first submodule to obtain a fourth sub-feature; the first fusion submodule performs Sum them to obtain the fourth fusion sub-feature; S313: Splicing the fourth fusion sub-feature and the first splicing layer in the first splicing storage module to obtain a fifth splicing result, and performing CBS processing on the fifth splicing result to obtain a fourth fusion feature.

4. The infrared image target detection method based on YOLO according to claim 1, characterized in that: In step S32, the feature fusion process of the second fusion module includes: S321: Perform CBS processing on the second splicing result to obtain a second splicing layer, and save the second splicing layer to a second splicing storage module; S322: Perform GSConv processing on the second splicing layer to obtain a second splicing feature, and perform DWCBS processing on the second splicing layer to obtain a second processing result; perform group convolution, rearrangement, CA mechanism and convolution processing on the second splicing feature in sequence through the second submodule to obtain the first subfeature; and perform fusion of the first subfeature and the second processing result through the second fusion submodule Perform summation to obtain the first fusion sub-feature; S323: Splice the first fusion sub-feature and the second splicing layer in the second splicing storage module to obtain a sixth splicing result, and perform CBS processing on the sixth splicing result to obtain a first fusion feature.

5. The infrared image target detection method based on YOLO according to claim 1, characterized in that: In step S33, the feature fusion process of the third fusion module includes: S331: performing CBS processing on the third splicing result to obtain a third splicing layer, and saving the third splicing layer to a third splicing storage module; S332: Perform GSConv processing on the third splicing layer to obtain a third splicing feature, and perform DWCBS processing on the third splicing layer to obtain a third processing result; perform group convolution, rearrangement, CA mechanism and convolution processing on the third splicing feature in sequence through the third submodule to obtain a second sub-feature; and perform fusion of the second sub-feature and the third processing result through the third fusion submodule. Perform summation to obtain the second fusion sub-feature; S333: Splice the second fusion sub-feature and the third splicing layer in the third splicing storage module to obtain a seventh splicing result, and perform CBS processing on the seventh splicing result to obtain a second fusion feature.

6. The infrared image target detection method based on YOLO according to claim 1, characterized in that: In step S34, the feature fusion process of the fourth fusion module includes: S341: Perform CBS processing on the fourth splicing result to obtain a fourth splicing layer, and save the fourth splicing layer to a fourth splicing storage module; S342: Perform GSConv processing on the fourth splicing layer to obtain a fourth splicing feature, and perform DWCBS processing on the fourth splicing layer to obtain a fourth processing result; perform group convolution, rearrangement, CA mechanism and convolution processing on the fourth splicing feature in sequence through the fourth submodule to obtain a third sub-feature; and perform fusion of the third sub-feature and the fourth processing result through the fourth fusion submodule. Perform summation to obtain the third fusion sub-feature; S343: Splice the third fusion sub-feature and the fourth splicing layer in the fourth splicing storage module to obtain an eighth splicing result, and perform CBS processing on the eighth splicing result to obtain a third fusion feature.

7. The infrared image target detection method based on YOLO according to claim 1, characterized in that: Step S4 includes: S41: Classifying and locating the first fusion feature through the first detection head to obtain a first target detection result; S42: Classifying and locating the second fusion feature through the second detection head to obtain a second target detection result; S43: classifying and locating the third fusion feature by a third detection head to obtain a third target detection result; S44: The first target detection result, the second target detection result, and the third target detection result are formed into a set to obtain a target detection result.

Images