Infrared small target detection method and system based on multi-task mutual promotion learning

Abstract

The invention discloses an infrared small target detection method and system based on multi-task mutual promotion learning, and the method achieves the better feature extraction of an infrared small target through the fusion of features of different stages and different receptive fields through a multi-branch feature aggregation module. By mixing different pooling operations in the down-sampling module, more complementary information is reserved, and information loss of the infrared small target in the down-sampling process is relieved. In the hierarchical feature fusion module, according to the relative importance of different semantic layer information to a prediction result, adaptive selection and fusion are performed on the different semantic layer information so as to realize a more flexible and excellent feature fusion effect. And respectively outputting a detection head and a segmentation head according to prediction results, and jointly performing network optimization by using bounding box supervision information and segmentation mask supervision information.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared small target detection, and in particular to an infrared small target detection method and system based on multi-task mutual promotion learning. Background Art

[0002] Infrared small target detection is a very important detection method because it is not affected by weather and lighting conditions and is a passive detection method. It is widely used in national defense security, maritime surveillance, traffic monitoring, precision guidance and other fields. However, during the detection process, the distance between the target and the detector is often far away, and the background of the target is complex and changeable, resulting in the characteristics of small targets, complex backgrounds, and low signal-to-noise ratio, which brings challenges to the accurate detection of infrared small targets.

[0003] Compared with traditional methods such as filtering or human visual system, data-driven deep learning methods have stronger environmental adaptability and feature extraction capabilities, and are currently a hot research topic in the field of infrared small target detection. Deep learning-based methods can be roughly divided into two paradigms, namely: segmentation-based paradigm and detection-based paradigm. The segmentation-based method distinguishes between the target and the background at the pixel level and focuses more on pixel-level results. The detection-based method uses a bounding box to display the target, emphasizing the target-level results.

[0004] Although both methods have achieved good results, they only utilize single supervisory information. The insufficient utilization of supervisory information limits the detector from achieving more outstanding detection performance. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an infrared small target detection method based on multi-task mutual promotion learning, which improves the detection performance of infrared small targets by adopting a multi-task mutual promotion learning network for infrared small target detection and segmentation.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The infrared small target detection method based on multi-task mutual promotion learning provided by the present invention comprises the following steps:

[0008] S1: Acquire infrared small target image;

[0009] S2: Inputting the infrared small target image into the backbone network for processing, the backbone network is used to alternately calculate from different semantic layers to obtain different scale features, the different scale features including shallow features and deep features;

[0010] S3: The shallow features and deep features obtained by the backbone network processing are simultaneously input into the hierarchical feature fusion module for processing to obtain the prediction results;

[0011] S4: Output prediction results.

[0012] Further, the backbone network includes a multi-branch feature aggregation module and a hybrid downsampling module;

[0013] S21: The multi-branch feature aggregation module is used to extract different receptive field features of the infrared small target image and then fuse them with the original features to obtain multi-stage receptive field fusion features;

[0014] S22: The hybrid downsampling module is used to process the multi-stage receptive field fusion features and then connect them in series to obtain the comprehensive downsampling features.

[0015] Further, the multi-branch feature aggregation module is performed according to the following steps:

[0016] S211: Segment the input features along the channel to obtain a first feature map and a second feature map;

[0017] S212: performing convolution processing of different sizes on the second feature map F2 to obtain different receptive field features, and connecting the different receptive field features by channel to obtain a comprehensive receptive field feature FC;

[0018] S213: Calculate the average value and the maximum value along the channel dimension according to the comprehensive receptive field feature FC to obtain the weight of each channel attention, that is, obtain the channel average attention and channel maximum attention of the comprehensive receptive field feature FC;

[0019] S214: fusing the channel average attention and the channel maximum attention with different receptive field features obtained by convolution processing of the second feature map to obtain two attention features;

[0020] S215: fusing the two attention features processed by the convolution kernel with the features of the second feature map processed by the convolution to obtain a third feature map;

[0021] S216: performing a convolution operation on the third feature map to obtain a fourth feature map;

[0022] S217: Perform a convolution operation on the first feature map, the third feature map, and the fourth feature map to obtain an output feature.

[0023] Further, the hybrid downsampling module is performed according to the following steps:

[0024] S221: performing a channel segmentation operation on the input sampling features to obtain a first sampling feature map and a second sampling feature map;

[0025] S222: performing a maximum pooling operation on the first sampling feature map and then performing a channel segmentation operation;

[0026] S223: Perform channel connection on the features after channel segmentation of the first sampling feature map and then perform channel connection on the features after average pooling of the second sampling feature map to finally obtain output features.

[0027] Further, the hierarchical feature fusion module is performed according to the following steps:

[0028] S31: Fusing the input shallow features with the deep features to obtain fused features;

[0029] S32: performing spatial attention and channel attention operations on the fused features respectively and fusing the features obtained by performing pixel attention operations on the fused features to obtain feature attention features Att;

[0030] S33: The features obtained by fusing the attention features with the shallow features, the deep features with the (1-Att) attention features, and the fusion features are fused to obtain the output features.

[0031] Furthermore, the prediction results are output to the detection head and the segmentation head respectively in the following manner:

[0032] The detection head uses a rectangular frame to frame the target object, and the rectangular frame frames the target object using the target position and size error provided by the bounding box supervision information;

[0033] The segmentation head uses segmentation mask annotation, and the segmentation mask annotation performs pixel-level segmentation on the target object in the image, thereby separating the target pixels.

[0034] The infrared small target detection system based on multi-task mutual promotion learning provided by the present invention includes an input module, a backbone network, a hierarchical feature fusion module, and an output module;

[0035] The input module is used to obtain an infrared small target image;

[0036] The backbone network is used to input the infrared small target image into the backbone network for processing, and the backbone network is used to alternately calculate different scale features from different semantic layers, and the different scale features include shallow features and deep features;

[0037] The hierarchical feature fusion module is used to simultaneously input the shallow features and deep features obtained through the backbone network processing into the hierarchical feature fusion module for processing to obtain prediction results;

[0038] The output module is used to output the prediction results.

[0039] Further, the backbone network includes a multi-branch feature aggregation module and a hybrid downsampling module;

[0040] The multi-branch feature aggregation module is used to extract different receptive field features of the infrared small target image and then fuse them with the original features to obtain multi-stage receptive field fusion features;

[0041] The hybrid downsampling module is used to process the multi-stage receptive field fusion features and then perform concatenation to obtain the comprehensive downsampling features.

[0042] Further, the multi-branch feature aggregation module is performed according to the following steps:

[0043] S211: Segment the input features along the channel to obtain a first feature map and a second feature map;

[0044] S212: performing convolution processing of different sizes on the second feature map F2 to obtain different receptive field features, and connecting the different receptive field features by channel to obtain a comprehensive receptive field feature FC;

[0045] S213: Calculate the average value and the maximum value along the channel dimension according to the comprehensive receptive field feature FC to obtain the weight of each channel attention, that is, obtain the channel average attention and channel maximum attention of the comprehensive receptive field feature FC;

[0046] S214: fusing the channel average attention and the channel maximum attention with different receptive field features obtained by convolution processing of the second feature map to obtain two attention features;

[0047] S215: fusing the two attention features processed by the convolution kernel with the features of the second feature map processed by the convolution to obtain a third feature map;

[0048] S216: performing a convolution operation on the third feature map to obtain a fourth feature map;

[0049] S217: Perform a convolution operation on the first feature map, the third feature map, and the fourth feature map to obtain an output feature;

[0050] The hybrid downsampling module is performed according to the following steps:

[0051] S221: performing a channel segmentation operation on the input sampling features to obtain a first sampling feature map and a second sampling feature map;

[0052] S222: performing a maximum pooling operation on the first sampling feature map and then performing a channel segmentation operation;

[0053] S223: Perform channel connection on the features after channel segmentation of the first sampling feature map and then perform channel connection on the features after average pooling of the second sampling feature map to finally obtain output features.

[0054] Further, the hierarchical feature fusion module is performed according to the following steps:

[0055] S31: Fusing the input shallow features with the deep features to obtain fused features;

[0056] S32: performing spatial attention and channel attention operations on the fused features respectively and fusing the features obtained by performing pixel attention operations on the fused features to obtain feature attention features Att;

[0057] S33: The features obtained by fusing the attention features with the shallow features, the deep features with the (1-Att) attention features, and the fusion features are fused to obtain the output features.

[0058] The beneficial effects of the present invention are:

[0059] The present invention provides an infrared small target detection method based on multi-task mutual promotion learning. The method adopts a multi-task mutual promotion learning network for infrared small target detection and segmentation to fully utilize the supervision information of the bounding box and the segmentation mask to make up for the deficiency of single supervision information.

[0060] This method uses a multi-branch feature aggregation module to fuse features from different stages and different receptive fields to achieve better feature extraction of small infrared targets. By mixing different pooling operations in the downsampling module, more complementary information is retained to alleviate the information loss of small infrared targets during the downsampling process. In the hierarchical feature fusion module, information from different semantic layers is adaptively selected and fused according to the relative importance of information from different semantic layers to the prediction results, so as to achieve a more flexible and excellent feature fusion effect. The prediction results are output to the detection head and the segmentation head respectively, and the network is optimized together using the bounding box supervision information and the segmentation mask supervision information.

[0061] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0063] Figure 1 This is a flow chart of the infrared small target recognition method based on a multi-task mutually-promoting learning network.

[0064] Figure 2 A network framework for multi-task mutual promotion learning.

[0065] Figure 3 It is the MFA module structure.

[0066] Figure 4 It is the HPDown module structure.

[0067] Figure 5 It is a HFF module structure.

[0068] Figure 6 Visualization of prediction results for different methods.

[0069] Figure 7 Comparison of ROC curves of different methods.

[0070] Figure 8 Comparison of PR curves of different methods.

[0071] Fig. 9 Feature extraction visualization for MFA.

[0072] Fig.10 Feature extraction visualization for HPDown.

[0073] Fig.11 Visualization of HFF feature fusion. DETAILED DESCRIPTION

[0074] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0075] Example 1

[0076] like Figure 1 and Figure 2 As shown, the infrared small target detection method based on multi-task mutual promotion learning provided in this embodiment includes the following steps:

[0077] S1: Acquire infrared small target image;

[0078] S2: Input the infrared small target image into the backbone network for processing. The backbone network mainly includes a multi-branch feature aggregation module (MFA) and a hybrid downsampling module (HPDown). The backbone network extracts different gradient features of the infrared small target image and the features obtained by multiple pooling downsampling to perform traffic calculation processing, thereby obtaining different scale features; the different scale features include shallow features and deep features; the features output by the backbone network in this embodiment are obtained by alternately calculating between the two modules to obtain features of different scales;

[0079] S21: The multi-branch feature aggregation module (MFA) is used to extract different gradient flows and different receptive field features of the infrared small target image and then fuse them with the original features to obtain multi-stage receptive field fusion features;

[0080] S22: the hybrid downsampling module (HPDown) is used to obtain comprehensive complementary downsampling features through multiple pooling downsampling operations after processing the multi-stage receptive field fusion features;

[0081] S3: The shallow features and deep features in the comprehensive complementary down-sampling features obtained by the backbone network are simultaneously input into the hierarchical feature fusion module (HFF) for processing to obtain the prediction results;

[0082] That is, the comprehensive complementary down-sampling features processed by the backbone network are directly input into the hierarchical feature fusion module (HFF); at the same time, the multi-stage receptive field features obtained by the feature fusion processing of the features processed by the backbone network are input into the hierarchical feature fusion module (HFF) for processing;

[0083] The hierarchical feature fusion module (HFF) is used to adaptively select shallow features from the comprehensive complementary down-sampling features output from the backbone network and deep features after feature fusion, focusing on feature information that is more beneficial to the prediction results;

[0084] S4: Output the prediction results to the detection head and the segmentation head respectively, and use the target position and size error provided by the bounding box supervision information and the category error of each pixel provided by the segmentation mask supervision information to jointly perform network optimization;

[0085] The output infrared small target image in this embodiment is respectively annotated with a bounding box and annotated with a segmentation mask to form an output image;

[0086] The boundary box annotation is to frame the target object with a rectangular frame, and the segmentation mask annotation is to perform pixel-level segmentation on the target object in the image to separate the target pixels.

[0087] The multi-branch feature aggregation module (MFA) in this embodiment achieves better feature extraction of infrared small targets by fusing features of different stages and different receptive fields; learns more gradient and receptive field features through multi-branch design, and adopts large convolution kernels and multi-stage design to fuse more gradient information in information transmission; the specific structure is to split the input features along the channel, perform a convolution operation with double the number of channels, process the split features separately, fuse features of different receptive fields, and calculate channel attention, and finally obtain the output by fusing the original features and multi-receptive field features.

[0088] The hybrid downsampling module (HPDown) in this embodiment retains more complementary information through different pooling operations, thereby alleviating the information loss of small infrared targets during the downsampling process; through the hierarchical feature fusion module, the information of different semantic layers is adaptively selected and fused according to the relative importance of the information of different semantic layers to the prediction results, so as to achieve a more flexible and excellent feature fusion effect;

[0089] The hybrid downsampling module in this embodiment adopts different pooling operations and receptive field operations to reduce the information loss of small infrared targets during the downsampling process. The input features are segmented into channels, and corresponding pooling and convolution operations are performed on different parts. Finally, the comprehensive downsampling feature output is obtained by connecting the feature maps in series.

[0090] The hierarchical feature fusion module (HFF) in this embodiment considers the relative importance of feature information at different semantic layers and adaptively selects and fuses features at different semantic layers. For cross-level semantic features, a preliminary fusion is first performed, and then spatial attention, channel attention, and pixel attention operations are performed, and finally fused with the original features to obtain the final output.

[0091] The multi-task mutual promotion learning network (MTL) established in this embodiment can make more full use of the bounding box and segmentation mask supervision information, achieve better feature extraction by fusing features of different stages and different receptive fields, alleviate information loss by using multiple pooling operations, and adaptively fuse hierarchical features considering the relative importance of features at different levels.

[0092] Example 2

[0093] like Figure 3 As shown, Figure 3 It is a multi-branch feature aggregation module MFA structure, where: Figure 3 (a) in the figure represents the MFA module principle diagram; (b) represents the MFA module principle framework diagram; the multi-branch feature aggregation module (MFA) is performed according to the following steps:

[0094] S211: Segment the input features along the channel to obtain a first feature map and a second feature map;

[0095] S212: performing convolution processing of different sizes on the second feature map F2 to obtain different receptive field features, and connecting the different receptive field features by channel to obtain a comprehensive receptive field feature FC;

[0096] S213: Calculate the average value and the maximum value along the channel dimension according to the comprehensive receptive field feature FC to obtain the weight of each channel attention, that is, obtain the channel average attention and channel maximum attention of the comprehensive receptive field feature FC;

[0097] S214: fusing the channel average attention and the channel maximum attention with different receptive field features obtained by convolution processing of the second feature map to obtain two attention features;

[0098] S215: fusing the two attention features processed by the convolution kernel with the features of the second feature map processed by the convolution to obtain a third feature map;

[0099] S216: performing a convolution operation on the third feature map to obtain a fourth feature map;

[0100] S217: Perform a convolution operation with a convolution kernel of 1 on the first feature map, the third feature map, and the fourth feature map to obtain an output feature.

[0101] The specific working process of the multi-branch feature aggregation module (MFA) in this embodiment is as follows: first, the input features need to be segmented along the channel to obtain more gradient information, and the input features need to be segmented along the channel to obtain the first feature map and the second feature map. At the same time, in order to ensure that the number of channels remains unchanged, a convolution operation with double the number of channels is required, which can be expressed as:

[0102] F1,F2=split(Conv 1×1 (F)

[0103] Among them, Conv 1×1 It is represented as a convolution operation with a kernel size of 1 and double the number of channels. F represents the original input feature, and split means that the feature is evenly divided into two parts along the channel direction, namely F1 and F2.

[0104] The shortest gradient path is performed on F1 without any additional computation blocks to preserve more local information.

[0105] In this embodiment, different receptive field features are selected and fused for the second feature map F2 to capture more complex global information;

[0106] F 21 =Conv 5×5 (F2)

[0107] F 22 =Conv 7×7 (F 21 )

[0108] Among them, F21 and F22 represent the features after being processed by convolution kernels of different sizes. Conv5×5 and Conv7×7 represent the convolution operations with convolution kernel sizes of 5 and 7, respectively. Then, the features with different receptive fields are connected by channel so that the mixed features have multiple receptive fields.

[0109] FC=Cat(F 21 ,F22 )

[0110] Among them, Cat(.) represents the concatenation of feature maps along the channel dimension;

[0111] FC represents the integration of features of different receptive fields;

[0112] FC calculates the average and maximum values ​​along the channel dimension to obtain the attention weight of each channel to capture the overall information and significant information in each channel, which is beneficial for the model to learn the importance of each channel, thereby enhancing the expressiveness of the features. The specific expression is as follows:

[0113] FA Avg =C Avg (FC),FA Max =C Max (FC)

[0114] Among them, F Aavg and F Amax Represents the channel average attention and channel maximum attention of the comprehensive feature FC;

[0115] C Avg and C Max It is based on the average value and maximum value calculation of the channel;

[0116] Applying different channel attention to the features of different receptive fields can adaptively adjust the weights of features of different receptive fields, and dynamically adjust the contribution of features according to the characteristics of different inputs:

[0117]

[0118]

[0119] in, It is the feature after channel attention modulation;

[0120] FCatt represents the multi-receptive field features after weighted fusion;

[0121] The final output is the fusion of the original features and the multi-receptive field features to obtain the third feature Figure 3 :

[0122]

[0123] For the third feature Figure 3 Perform a convolution operation with a convolution kernel of 1 to obtain the fourth feature Figure 4 ;

[0124] The first feature map, the third feature map, and the fourth feature map are convolved with a convolution kernel of 1 to obtain the output feature; that is, after the feature F passes through the MFA module, the fused multi-receptive field multi-gradient feature can be expressed as:

[0125]

[0126] Among them, Conv 1×1 Represents a convolution operation with a convolution kernel of 1.

[0127] like Figure 4 As shown, Figure 4 It is the structure of the hybrid downsampling module HPDown. Figure 4 (a) in the figure shows the principle diagram of the HPDown module; (b) shows the principle framework diagram of the HPDown module; the hybrid downsampling module is performed according to the following steps:

[0128] S221: performing a channel segmentation operation on the input sampling features to obtain a first sampling feature map and a second sampling feature map;

[0129] S222: performing a maximum pooling operation on the first sampling feature map and then performing a channel segmentation operation;

[0130] S223: performing channel connection on the features after channel segmentation of the first sampling feature map and then performing channel connection on the features after average pooling of the second sampling feature map, and finally obtaining output features;

[0131] The working process of the HPDown module in this embodiment is as follows: different pooling operations and different receptive field operations are used to preserve as much complementary information as possible, thereby alleviating the information loss of small infrared targets during the downsampling process.

[0132] First, the input feature X is segmented along the channel direction to retain features of different forms, which can be expressed as the following formula:

[0133] X1, X2 = split(X)

[0134] Among them, X1, X2 means that the input feature X is evenly divided into two parts along the channel direction.

[0135] A maximum pooling operation is performed on X2 to retain the significant features in the image, followed by a conventional CBS operation including Conv, BN, and Silu to introduce learnable parameters, and the most important features are adaptively learned. This process can be expressed by the following formula:

[0136]

[0137] Among them, MaxPool represents the maximum pooling operation. Represents the features after processing X2.

[0138] For X1, an average pooling operation is first performed to obtain a smooth feature map. In order to obtain richer feature information, it is segmented along the channel, which can be expressed as:

[0139] X 11 , X 12 =chunk(AvgPool(X1))

[0140] Among them, AvgPool represents the average pooling operation, and X11 and X12 represent the features of channel segmentation after average pooling of feature X1.

[0141] Similarly, learnable parameters are introduced through the Conv operation, and convolution kernels of different sizes are used to retain more information.

[0142] Finally, the comprehensive down-sampled feature output of the input feature X after passing through the HPDown module can be expressed as:

[0143]

[0144] Among them, Cat(.) represents the concatenation of feature maps along the channel dimension.

[0145] like Figure 5 As shown, Figure 5 It is a HFF module structure. Figure 5 (a) in the figure shows the principle diagram of the HFF module; (b) shows the principle framework diagram of the HFF module; the hierarchical feature fusion module (HFF) is performed according to the following steps:

[0146] S31: Fusing the input shallow features with the deep features to obtain fused features;

[0147] S32: performing spatial attention and channel attention operations on the fused features respectively and fusing the features obtained by performing pixel attention operations on the fused features to obtain feature attention features Att;

[0148] S33: The feature obtained by fusing the attention feature with the shallow feature, the deep feature with the (1-Att) attention feature, and the fusion feature are fused to obtain the output feature;

[0149] The working process of the hierarchical feature fusion module (HFF) in this embodiment is as follows: for cross-semantic layer feature information, the relative importance of feature information of different semantic layers is learned through HFF. Specifically, for cross-layer semantic features, the shallow features and deep features are initially fused together using element addition to form a preliminary comprehensive feature X raw :

[0150] X raw =X low +X hig h ;

[0151] Among them, X low and X high They respectively represent the shallow semantic features extracted by the backbone network and the deep semantic features after feature fusion.

[0152] Then perform spatial attention and channel attention operations respectively:

[0153] X SA =SA(X raw ),X CA =CA(X raw )

[0154] Among them, X SA ,X CA Respectively represent the features after the spatial attention operation SA(.) and the spatial attention operation CA(.):

[0155] Att=PA(X raw ).(X SA +X CA );

[0156] Among them, PA represents the pixel attention mechanism with a convolution kernel of 1; Att represents the attention feature after the space-channel-pixel attention operation, which is harmonized with the original feature to obtain the final output feature.

[0157] X out =X raw +Att*X low +(1―Att)*X hig h ;

[0158] Among them, X out Represents the output after the HFF operation, where the obtained attention Att and (1-Att) represent the relative importance of features at different semantic levels. The model can adaptively learn the optimal feature fusion method between features at different semantic levels, taking into account the contribution of features at different semantic levels to the infrared small target detection task.

[0159] Example 3

[0160] like Figure 6-11 As shown, this embodiment demonstrates the effect of the infrared small target detection method through experiments, and the specific experimental process is as follows:

[0161] Dataset introduction: According to previous experimental experience, although it is difficult for the human eye to distinguish simulated datasets, the network is able to effectively learn their underlying patterns. Therefore, the numerical indicators obtained by the model on the simulated dataset are often higher than those on the actual dataset. However, the detector trained on the simulated dataset may learn the generated patterns in the simulated data, and the detection performance in the real scene may be limited.

[0162] This embodiment manually annotates the infrared small target image, uses a rectangular frame to frame the target object and a segmentation mask to annotate the target object. Both annotation forms play the role of providing supervisory information, and different supervisory information is used to optimize the network constructed by this method.

[0163] Therefore, this embodiment selects two actual data sets, IRSTD-1k and SIRST-V2, for all experiments of this method. In addition, since the scale of each individual data set is relatively small and the performance fluctuates greatly during training, this embodiment merges the two data sets to obtain more stable performance indicators. The training set and the test set are divided in a ratio of 4:1.

[0164] Evaluation metrics: Common segmentation and detection evaluation metrics such as precision, recall, F1 score, AP, and IoU are used to evaluate the segmentation performance of each model. The calculation method of each metric is as follows:

[0165] IoU: IoU is a pixel-level evaluation metric that calculates the ratio of the intersection area and the union area of ​​the predicted area with the ground truth area:

[0166] IoU=A i / A u

[0167] In the formula, Ai and Au represent the sizes of the intersection domain and the union domain respectively;

[0168] Precision: Precision is calculated as the ratio of true positives (TP) to true positives and false positives (FP):

[0169] Precision=TP / (TP+FP)

[0170] Recall: Recall is calculated as the ratio of true positives to the sum of true positives and false negatives (FN):

[0171] Recall=TP / (TP+FN)

[0172] F1 and AP: These metrics are combined with P and R to provide a more comprehensive performance evaluation. They can be described as:

[0173] F1=2*(P*R) / (P+R)

[0174]

[0175] Compared with model-based methods, deep learning-based methods have been shown to have better performance. Based on this, this embodiment is mainly compared with the most advanced deep learning-based methods. These methods include segmentation paradigm methods MDvsMA, ACM, ALC, ISNet, DNANet, UIU, and detection paradigm methods OSCAR, EFLNet, SSTNet, MSML. In order to make a fair comparison, all methods were retrained on the selected data sets, and the training and testing of each method were based on the default hyperparameters of the method.

[0176] Table 1 shows a quantitative comparison with 10 state-of-the-art segmentation and detection paradigm methods. These include 6 segmentation paradigm methods and 4 detection paradigm methods. As can be seen from the table, the multi-task mutual promotion learning network we proposed not only achieves the best performance in the segmentation task, but also performs best in the detection task, proving the effectiveness of the multi-task mutual promotion learning network designed by this method. In the detection and segmentation tasks, this method exceeds the suboptimal method by 4.7% and 3.6% in the comprehensive evaluation indicators AP and IoU, respectively. Since the current infrared small target detection methods based on deep learning mostly regard them as detection tasks or segmentation tasks separately, the form of supervision information is often single, and the inability to fully utilize different forms of supervision information limits these methods from achieving better performance.

[0177] Therefore, these methods have not achieved the best results in either detection or segmentation tasks. The multi-task mutual promotion learning network provided in this embodiment can introduce additional supervision information to train the model, thereby achieving better performance.

[0178] Table 1. Evaluation metrics obtained by SOTA-based detection and segmentation methods

[0179] method R P F1 AP Rec(pixel) Pre(pixel) F1(pixel) IoU MDvsMA / / / / 0.383 0.435 0.407 / ACM / / / / 0.735 0.806 0.769 0.627 ALC / / / / 0.540 0.860 0.663 0.639 ISNet / / / / 0.802 0.695 0.744 0.625 DNANet / / / / 0.784 0.801 0.793 0.656 UIU / / / / 0.832 0.214 0.340 0.612 Ours / / / / 0.819 0.826 0.822 0.692 OSCAR / / / 0.620 / / / / EFL 0.747 0.775 0.761 0.767 / / / / MSML 0.773 0.423 0.547 0.752 / / / / SSTNet 0.355 0.581 0.441 0.659 / / / / Ours 0.818 0.747 0.781 0.814 / / / /

[0180] In order to verify the effectiveness of each module, an ablation experiment was conducted on the combined effect of multiple modules. As can be seen from Table 2, the MFA module and HLoss have brought an overall improvement to the model, and all indicators have been greatly improved, indicating that these two modules have promoted both the segmentation task and the detection task. HPDown and HFF are more conducive to the segmentation task, and the segmentation indicators have been improved, which has a slight impact on the detection task, but does not hinder the overall promotion of the model.

[0181] Table 2. Modules MFA, HPDown, HFF, and L multi Ablation studies

[0182] MFA HPDown HFF <![CDATA[L multi ]]> F1(target) AP F1(pixel) IoU × × × × 0.7524 0.789 0.7348 0.581 √ × × × 0.7892 0.811 0.7711 0.646 √ √ × × 0.7710 0.794 0.7915 0.655 √ √ √ × 0.7685 0.805 0.8046 0.673 √ √ √ √ 0.7809 0.814 0.8225 0.692

[0183] Tables 3 and 4 are ablation experiments on different convolution kernel sizes for the MFA and HPD modules, respectively. It can be seen that a larger convolution kernel is beneficial to the infrared small target detection task, and both the detection index and the segmentation index are improved to a certain extent. However, the performance of the model does not improve blindly with the increase of the convolution kernel. When the convolution kernel size is 5 and 7, the comprehensive performance of the model reaches the best. When the convolution kernel continues to increase, the model performance shows a downward trend, indicating that the size of the convolution kernel should be appropriately increased. Excessive increase in the size of the convolution kernel will have the opposite effect.

[0184] Table 3. Effect of different convolution kernel sizes on the MFA module

[0185]

[0186] Table 4. Effect of different convolution kernel sizes on the HPDown module

[0187]

[0188]

[0189] Table 5 shows the ablation experiment results for detection and segmentation tasks. From the table, we can see that the performance of the multi-task network model is better than that of the single-task network model. In addition, compared with the detection task, the improvement of the segmentation task is more significant. This is because the multi-task model introduces more additional supervision information for the segmentation task, while the detection task is relatively small, so the segmentation performance improvement is more significant.

[0190] Table 5. Ablation experiment results of detection and segmentation tasks

[0191]

[0192] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. An infrared small target detection method based on multi-task mutual promotion learning, characterized by: The following steps are involved: S1: Acquire infrared small target image; S2: Inputting the infrared small target image into the backbone network for processing, the backbone network is used to alternately calculate from different semantic layers to obtain different scale features, the different scale features including shallow features and deep features; S3: The shallow features and deep features obtained by the backbone network processing are simultaneously input into the hierarchical feature fusion module for processing to obtain the prediction results; S4: Output prediction results.

2. The infrared small target detection method based on multi-task mutual promotion learning as claimed in claim 1 is characterized by: The backbone network includes a multi-branch feature aggregation module and a hybrid downsampling module; S21: The multi-branch feature aggregation module is used to extract different receptive field features of the infrared small target image and then fuse them with the original features to obtain multi-stage receptive field fusion features; S22: The hybrid downsampling module is used to process the multi-stage receptive field fusion features and then connect them in series to obtain the comprehensive downsampling features.

3. The infrared small target detection method based on multi-task mutual promotion learning as claimed in claim 1 is characterized in that: The multi-branch feature aggregation module is performed according to the following steps: S211: Segment the input features along the channel to obtain a first feature map and a second feature map; S212: performing convolution processing of different sizes on the second feature map F2 to obtain different receptive field features, and connecting the different receptive field features by channel to obtain a comprehensive receptive field feature FC; S213: Calculate the average value and the maximum value along the channel dimension according to the comprehensive receptive field feature FC to obtain the weight of each channel attention, that is, obtain the channel average attention and channel maximum attention of the comprehensive receptive field feature FC; S214: fusing the channel average attention and the channel maximum attention with different receptive field features obtained by convolution processing of the second feature map to obtain two attention features; S215: fusing the two attention features processed by the convolution kernel with the features of the second feature map processed by the convolution to obtain a third feature map; S216: performing a convolution operation on the third feature map to obtain a fourth feature map; S217: Perform a convolution operation on the first feature map, the third feature map, and the fourth feature map to obtain an output feature.

4. The infrared small target detection method based on multi-task mutual promotion learning as claimed in claim 1, characterized in that: The hybrid downsampling module is performed according to the following steps: S221: performing a channel segmentation operation on the input sampling features to obtain a first sampling feature map and a second sampling feature map; S222: performing a maximum pooling operation on the first sampling feature map and then performing a channel segmentation operation; S223: Perform channel connection on the features after channel segmentation of the first sampling feature map and then perform channel connection on the features after average pooling of the second sampling feature map to finally obtain output features.

5. The infrared small target detection method based on multi-task mutual promotion learning as claimed in claim 1, characterized in that: The hierarchical feature fusion module is performed according to the following steps: S31: Fusing the input shallow features with the deep features to obtain fused features; S32: performing spatial attention and channel attention operations on the fused features respectively and fusing the features obtained by performing pixel attention operations on the fused features to obtain feature attention features Att; S33: The features obtained by fusing the attention features with the shallow features, the deep features with the (1-Att) attention features, and the fusion features are fused to obtain the output features.

6. The infrared small target detection method based on multi-task mutual promotion learning as claimed in claim 1, characterized in that: The prediction results are output to the detection head and the segmentation head respectively in the following manner: The detection head uses a rectangular frame to frame the target object, and the rectangular frame frames the target object using the target position and size error provided by the bounding box supervision information; The segmentation head uses segmentation mask annotation, and the segmentation mask annotation performs pixel-level segmentation on the target object in the image, thereby separating the target pixels.

7. Infrared small target detection system based on multi-task mutual promotion learning, characterized by: It includes input module, backbone network, hierarchical feature fusion module and output module; The input module is used to obtain an infrared small target image; The backbone network is used to input the infrared small target image into the backbone network for processing, and the backbone network is used to alternately calculate different scale features from different semantic layers, and the different scale features include shallow features and deep features; The hierarchical feature fusion module is used to simultaneously input the shallow features and deep features obtained through the backbone network processing into the hierarchical feature fusion module for processing to obtain prediction results; The output module is used to output the prediction results.

8. The infrared small target detection system based on multi-task mutual promotion learning as claimed in claim 7, characterized in that: The backbone network includes a multi-branch feature aggregation module and a hybrid downsampling module; The multi-branch feature aggregation module is used to extract different receptive field features of the infrared small target image and then fuse them with the original features to obtain multi-stage receptive field fusion features; The hybrid downsampling module is used to process the multi-stage receptive field fusion features and then perform concatenation to obtain the comprehensive downsampling features.

9. The infrared small target detection system based on multi-task mutual promotion learning as claimed in claim 8, characterized in that: The multi-branch feature aggregation module is performed according to the following steps: S211: Segment the input features along the channel to obtain a first feature map and a second feature map; S212: performing convolution processing of different sizes on the second feature map F2 to obtain different receptive field features, and connecting the different receptive field features by channel to obtain a comprehensive receptive field feature FC; S213: Calculate the average value and the maximum value along the channel dimension according to the comprehensive receptive field feature FC to obtain the weight of each channel attention, that is, obtain the channel average attention and channel maximum attention of the comprehensive receptive field feature FC; S214: fusing the channel average attention and the channel maximum attention with different receptive field features obtained by convolution processing of the second feature map to obtain two attention features; S215: fusing the two attention features processed by the convolution kernel with the features of the second feature map processed by the convolution to obtain a third feature map; S216: performing a convolution operation on the third feature map to obtain a fourth feature map; S217: Perform a convolution operation on the first feature map, the third feature map, and the fourth feature map to obtain an output feature; The hybrid downsampling module is performed according to the following steps: S221: performing a channel segmentation operation on the input sampling features to obtain a first sampling feature map and a second sampling feature map; S222: performing a maximum pooling operation on the first sampling feature map and then performing a channel segmentation operation; S223: Perform channel connection on the features after channel segmentation of the first sampling feature map and then perform channel connection on the features after average pooling of the second sampling feature map to finally obtain output features.

10. The infrared small target detection system based on multi-task mutual promotion learning as claimed in claim 7, characterized in that: The hierarchical feature fusion module is performed according to the following steps: S31: Fusing the input shallow features with the deep features to obtain fused features; S32: performing spatial attention and channel attention operations on the fused features respectively and fusing the features obtained by performing pixel attention operations on the fused features to obtain feature attention features Att; S33: The features obtained by fusing the attention features with the shallow features, the deep features with the (1-Att) attention features, and the fusion features are fused to obtain the output features.

Images