Infrared dim target detection method and device, computing device, and storage medium

Abstract

Embodiments of the present application relate to the technical field of image processing, and particularly relate to an infrared dim small target detection method and device, a computing device and a storage medium. The method comprises: acquiring an infrared dim small target image to be detected; inputting the infrared dim small target image into a detection model generated by pre-training; wherein the detection model is obtained based on a preset neural network, the preset neural network comprising a backbone network formed by connecting four residual networks in series, a context feature extraction network composed of multiple hollow convolution layers, a feature fusion network and a detection head module; and obtaining a detection result of the infrared dim small target image according to an output result of the detection head module. The scheme can improve the detection capability of the detection model for the infrared dim small target.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and particularly relate to an infrared dim small target detection method and device, a computing device and a storage medium. BACKGROUND

[0002] In the field of computer vision, the detection of infrared dim small targets has always been a popular and challenging topic. In recent years, there has been a demand for higher detection rates and lower false alarm rates in the detection of infrared dim small targets in various fields.

[0003] Existing infrared dim small target detection algorithms are almost all modified from single-stage target detection algorithms such as Single shot multibox detector (SSD) and You Only Look Once (YOLO), and the general target detection network is extended to the detection task of infrared dim small targets by modifying the network structure and the size of the preset anchor box. However, in the extension process, these detection methods with preset anchor box size are sensitive to the size of the small target, which makes the network optimization prone to large deviations, thereby affecting the detection accuracy of the infrared dim small target. SUMMARY

[0004] In order to improve the detection accuracy of existing infrared dim small target detection algorithms, embodiments of the present application provide an infrared dim small target detection method, device, computing device and storage medium.

[0005] In a first aspect, embodiments of the present application provide an infrared dim small target detection method, comprising:

[0006] obtaining an infrared dim small target image to be detected;

[0007] inputting the infrared dim small target image into a detection model generated by pre-training; wherein the detection model is obtained based on a preset neural network, and the preset neural network comprises a backbone network composed of four residual networks connected in series, a context feature extraction network composed of multiple atrous convolution layers, a feature fusion network, and a detection head module;

[0008] obtaining a detection result of the infrared dim small target image according to an output result of the detection head module.

[0009] Preferably, the detection head module comprises a first detection head and a second detection head.

[0010] The training method of the detection model comprises:

[0011] obtaining a plurality of training samples labeled with labels;

[0012] Multiple training samples are input into the backbone network, and the four residual networks in the backbone network are used to perform first-stage feature extraction, second-stage feature extraction, third-stage feature extraction and fourth-stage feature extraction on each training sample respectively, so as to obtain the feature map after each stage of feature extraction;

[0013] The feature map extracted in the fourth stage is input into each dilated convolutional layer in the context feature extraction network to obtain the expanded feature map corresponding to each training sample.

[0014] Using the feature fusion network, based on the feature map extracted in the third stage and the expanded feature map, a first fusion feature map corresponding to each training sample is obtained. After obtaining the first fusion feature map, a second fusion feature map corresponding to each training sample is obtained based on the first fusion feature map and the feature map extracted in the second stage.

[0015] The first fused feature map and the second fused feature map are respectively input into the first detection head and the second detection head to obtain the output result of each training sample;

[0016] Based on the output of each training sample and the label corresponding to each training sample, the network parameters of the neural network are adjusted until a detection model that meets the expectations is obtained.

[0017] Preferably, the first detection head and the second detection head share the same set of network parameters; the first detection head and the second detection head train the network parameters using a grouping normalization method.

[0018] Preferably, both the first detection head and the second detection head include a classification branch, a regression branch, and a centrality branch;

[0019] The step of inputting the first fused feature map and the second fused feature map into the first detection head and the second detection head respectively to obtain the output result of each training sample includes:

[0020] The first fused feature map and the second fused feature map are respectively input into the classification branch of the first detection head and the classification branch of the second detection head to obtain the classification result of each pixel in the first fused feature map and the second fused feature map;

[0021] The first fused feature map is input into the regression branch and the centrality branch of the first detection head, respectively, and the second fused feature map is input into the regression branch and the centrality branch of the second detection head, respectively, to obtain the regression results of the first fused feature map and the regression results of the second fused feature map.

[0022] Preferably, the regression branch and the centrality branch share the same set of convolutional layers.

[0023] Preferably, the output of the detection head module includes the classification and regression results of the first target feature map, and the classification and regression results of the second target feature map; the classification and regression results of the first target feature map are the output results of the first target feature map obtained by using the first detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model; the classification and regression results of the second target feature map are the output results of the second target feature map obtained by using the second detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model.

[0024] The step of obtaining the detection result of the infrared weak target image based on the output result of the detection head module includes:

[0025] Each pixel in the first target feature map and the second target feature map is mapped back into the infrared weak target image, so as to determine the initial target based on the classification results corresponding to the first target feature map and the second target feature map, respectively.

[0026] Based on the regression results corresponding to the first target feature map and the second target feature map, a candidate box is generated for each of the initial targets;

[0027] Calculate the distance from the center pixel of each initial target in the infrared weak target image to the corresponding candidate box, so as to filter each initial target according to the distance and obtain infrared weak targets;

[0028] Calculate the normalized distance from the center pixel of each infrared weak target in the infrared weak target image to the corresponding candidate box, so as to filter the candidate boxes corresponding to the infrared weak target according to the normalized distance and obtain the detection box of the infrared weak target.

[0029] Preferably, the distance is calculated using the following formula:

[0030]

[0031]

[0032]

[0033]

[0034] In the formula, the distance is (l* ,t * ,r * ,b * ), l * t * r * and b * These are the distances from the center pixel of the initial target in the infrared weak target image to the left, top, right, and bottom of the corresponding candidate box, respectively. Let be the coordinates of the upper left corner of the candidate box. The coordinates of the lower right corner of the candidate box;

[0035] The normalized distance is calculated using the following formula:

[0036]

[0037] In the formula, Centerness is the normalized distance, l * t * r * and b * These are the left, top, right, and bottom distances of the center pixel of the infrared weak target in the infrared weak target image to the corresponding candidate box.

[0038] Secondly, embodiments of the present invention also provide a detection device for infrared weak targets, comprising:

[0039] The acquisition unit is used to acquire images of the infrared weak targets to be detected;

[0040] The detection unit is used to input the infrared weak target image into a pre-trained detection model; wherein the detection model is trained based on a preset neural network, and the preset neural network includes a backbone network composed of four residual networks connected in series, a context feature extraction network composed of multiple dilated convolutional layers, a feature fusion network, and a detection head module;

[0041] The regression unit is used to obtain the detection result of the infrared weak target image based on the output result of the detection head module.

[0042] Thirdly, embodiments of the present invention also provide a computing device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of this specification.

[0043] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.

[0044] This invention provides a method, apparatus, computing device, and storage medium for detecting infrared weak targets. First, the image of the infrared weak target to be detected is input into a pre-trained detection model. The detection model is trained based on a preset neural network, which includes a backbone network consisting of four concatenated residual networks, a context feature extraction network composed of multiple dilated convolutional layers, a feature fusion network, and a detection head module. Therefore, the backbone network in the detection model can be used to extract features from the infrared weak target image, and the context feature extraction network can be used to introduce contextual information surrounding the infrared weak target. Then, the feature fusion network is used to fuse the features extracted by the backbone network and the context feature extraction network. Finally, a detection head module without preset anchor boxes is used to detect the infrared weak target image, thereby improving the detection model's ability to detect infrared weak targets. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of a method for detecting weak infrared targets according to an embodiment of the present invention;

[0047] Figure 2 This is a hardware architecture diagram of an electronic device provided in an embodiment of the present invention;

[0048] Figure 3 This is a structural diagram of an infrared weak target detection device provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] As mentioned earlier, almost all existing infrared weak target detection algorithms are modified from single-stage target detection algorithms such as Single Shot Multibox Detection (SSD) and You Only Look Once (YOLO). These general target detection networks are extended to the task of detecting infrared weak targets by modifying the network structure and the size of the preset anchor boxes. However, during the extension process, these detection methods with preset anchor box sizes are quite sensitive to the size of weak targets, which makes network optimization prone to significant deviations, thus affecting the detection accuracy of infrared weak targets.

[0051] To solve the above-mentioned technical problems, the inventors may consider using an algorithm without a preset anchor frame to detect infrared weak target images. Therefore, the detection head module in the detection model of this application does not have a preset anchor frame, thereby improving the detection accuracy of infrared weak targets.

[0052] The following describes the specific implementation of the above concept.

[0053] Please refer to Figure 1 This invention provides a method for detecting weak infrared targets, the method comprising:

[0054] Step 100: Acquire an image of the infrared weak target to be detected;

[0055] Step 102: Input the infrared weak target image into the pre-trained detection model; wherein, the detection model is trained based on a preset neural network, which includes a backbone network consisting of four residual networks connected in series, a context feature extraction network consisting of multiple dilated convolutional layers, a feature fusion network, and a detection head module;

[0056] Step 104: Based on the output of the detection head module, obtain the detection results of the infrared weak target image.

[0057] In this embodiment of the invention, the image of the infrared weak target to be detected is first input into a pre-trained detection model. The detection model is trained based on a preset neural network, which includes a backbone network consisting of four residual networks connected in series, a context feature extraction network consisting of multiple dilated convolutional layers, a feature fusion network, and a detection head module. Therefore, the backbone network in the detection model can be used to extract features from the image of the infrared weak target to be detected, and the context feature extraction network can be used to introduce the context information around the infrared weak target. Then, the feature fusion network is used to fuse the features extracted by the backbone network and the context feature extraction network. Finally, the detection head module without preset anchor boxes is used to detect the infrared weak target image, thereby improving the detection model's ability to detect infrared weak targets.

[0058] The following description Figure 1 The execution method for each step is shown.

[0059] For step 100:

[0060] It should be noted that the infrared weak target image to be detected in this step may or may not contain any category of infrared weak targets; no specific limitation is made here. Furthermore, the infrared weak target image to be detected is not labeled.

[0061] Regarding step 102:

[0062] In some implementations, the detection head module includes a first detection head and a second detection head.

[0063] Next, we will explain the process of building the detection model.

[0064] In some implementations, the training of the detection model may include the following steps H1-H4:

[0065] Step H1: Obtain multiple labeled training samples;

[0066] In step H1, several infrared weak target images captured in the actual environment are obtained, and the categories and bounding boxes of infrared weak targets in these images are manually labeled to serve as training samples.

[0067] Step H2 involves inputting multiple training samples into the backbone network, and then using the four residual networks in the backbone network to perform first-stage feature extraction, second-stage feature extraction, third-stage feature extraction, and fourth-stage feature extraction on each training sample to obtain the feature map after each stage of feature extraction.

[0068] In step H2, the backbone network comprises four stages of residual networks, and each stage of the residual network contains four residual blocks. First, each training sample is input into the first-stage residual network for first-stage feature extraction; then, the feature map extracted in the first stage is input into the second-stage residual network for second-stage feature extraction; next, the feature map extracted in the second stage is input into the third-stage residual network for third-stage feature extraction; finally, the feature map extracted in the third stage is input into the fourth-stage residual network for fourth-stage feature extraction, resulting in the fourth-stage feature map.

[0069] In this model, from the first to the fourth stage, the scale of the input feature maps gradually decreases while the number of feature layers gradually increases. Since the features contained in feature maps at different scales are not the same, the feature maps extracted in the earlier stages contain rich detailed features, while those extracted in the later stages contain rich semantic features. By using four residual networks connected in series with the backbone network to extract features from the training samples at multiple scales, multi-scale features of infrared weak targets can be obtained, which is beneficial for improving the detection rate of the detection model.

[0070] Step H3: Input the feature map extracted in the fourth stage into each dilated convolutional layer in the context feature extraction network to obtain the expanded feature map corresponding to each training sample.

[0071] In step H3, multiple dilated convolutional layers are used to extract contextual features from the feature maps after the fourth stage feature extraction, which expands the receptive field without reducing the feature resolution and without significantly increasing the model size.

[0072] In some implementations, the context feature extraction network includes a first dilated convolutional layer, a second dilated convolutional layer, a third dilated convolutional layer, a fourth dilated convolutional layer, and a fifth dilated convolutional layer.

[0073] Therefore, step H3 may include:

[0074] The feature map extracted from the fourth stage of each training sample is input into the first dilated convolutional layer to perform the first expansion of the feature map extracted from the fourth stage, thus obtaining the first feature map.

[0075] The first feature map and the feature map extracted in the fourth stage are concatenated and then input into the second dilated convolutional layer to obtain the second feature map;

[0076] The second feature map, the feature map extracted in the fourth stage, and the first feature map are concatenated and then input into the third dilated convolutional layer to obtain the third feature map.

[0077] The third feature map, the feature map extracted from the fourth stage, the first feature map, and the second feature map are concatenated and then input into the fourth dilated convolutional layer to obtain the fourth feature map.

[0078] The fourth feature map, the feature map extracted in the fourth stage, the first feature map, the second feature map, and the third feature map are concatenated and then input into the fifth dilated convolutional layer to obtain the fifth feature map.

[0079] By concatenating the fifth feature map, the feature map extracted in the fourth stage, the first feature map, the second feature map, the third feature map, and the fourth feature map, an expanded feature map corresponding to each training sample is obtained.

[0080] In this embodiment, the dilation rates of the first, second, third, fourth, and fifth dilated convolutional layers in the context feature extraction network are 1, 3, 6, 12, and 18, respectively. By densely connecting these dilated convolutional layers with different dilation rates, the feature resolution of each layer can be increased. The dense connections between the dilated convolutional layers and the selection of their dilation rates can cover a sufficiently large receptive field, forming sufficiently dense features, effectively enhancing the ability to perceive multi-scale infrared weak targets and context.

[0081] Step H4: Using a feature fusion network, based on the feature map extracted in the third stage and the expanded feature map, a first fusion feature map corresponding to each training sample is obtained. After obtaining the first fusion feature map, a second fusion feature map corresponding to each training sample is obtained based on the first fusion feature map and the feature map extracted in the second stage.

[0082] In step H4, the feature fusion network includes a first feature fusion module and a second feature fusion module. First, the first feature fusion module fuses the expanded feature map obtained from the context feature extraction network and the feature map extracted in the third stage of the backbone network to obtain a first fused feature map corresponding to each training sample. This aims to fuse features at different scales. The feature map extracted in the third stage has a relatively large scale, preserving local details such as image edges and contours, which is beneficial for target localization. The expanded feature map obtained from the context feature extraction network has the smallest scale, containing more abstract semantic information, but with poor detail perception. The first fused feature map obtained by fusing the two not only contains rich semantic information but also target details, which is beneficial for detecting weak infrared targets. Then, the second feature fusion module fuses the first fused feature map and the feature map extracted in the second stage of the backbone network to construct finer-grained features and features with richer semantic information, resulting in a more descriptive second fused feature map. This improves the detection rate of weak infrared targets and reduces the false alarm rate.

[0083] In some implementations, the first feature fusion module includes a spatial attention branch and a channel attention branch.

[0084] Therefore, the step "using a feature fusion network, based on the feature map extracted in the third stage and the expanded feature map, to obtain the first fused feature map corresponding to each training sample" can include:

[0085] The expanded feature map is interpolated, and the number of channels of the interpolated expanded feature map is reduced to the same number of channels as the feature map after the third stage feature extraction. Then, it is added to the feature map after the third stage feature extraction to obtain the first fusion map.

[0086] The first fused image is input into the spatial attention branch and the channel attention branch respectively, and the feature maps output by the spatial attention branch and the channel attention branch are added together to obtain the second fused image;

[0087] The result of multiplying the second fused map with the expanded feature map after interpolation and reduction of channel number is added to the result of multiplying the second fused map with the feature map after the third stage feature extraction to obtain the first fused feature map.

[0088] In this embodiment, in the first feature fusion module, since the expanded feature map obtained in step H3 has a small size and a large number of channels, in order to better fuse the expanded feature map and the feature map after the third stage feature extraction, the expanded feature map corresponding to each training sample obtained in step H3 can be interpolated to the same size as the feature map after the third stage feature extraction. Then, a 1×1 convolution is used to reduce the number of channels of the interpolated expanded feature map to the same number of channels as the feature map after the third stage feature extraction. Finally, it is added to the feature map after the third stage feature extraction to obtain the first fusion map. In order to highlight the intrinsic information of the infrared weak target, retain valuable features, and remove worthless features, this embodiment inputs the first fusion map into the spatial attention branch and the channel attention branch respectively, and adds the feature maps output by the spatial attention branch and the channel attention branch to obtain the second fusion map. Finally, the result of multiplying the second fusion map with the expanded feature map obtained after interpolation and channel reduction is added to the result of multiplying the second fusion map with the feature map after the third stage feature extraction to obtain the first fusion feature map containing rich semantic information and target details.

[0089] In some implementations, the spatial attention branch includes two convolutional layers with 1×1 kernels; the channel attention branch includes an adaptive pooling layer and a one-dimensional convolutional layer.

[0090] In this embodiment, the first fused image is input into the spatial attention branch and the channel attention branch, respectively. The spatial attention branch uses two 1×1 convolutions to output weights on the first fused image. The channel attention branch first uses adaptive pooling to reduce the input to a size of C×1×1, and then uses 1D convolution to capture cross-channel interactions in the five nearest channels to generate weights on each channel. In this way, the spatial attention branch and the channel attention branch can be used to highlight the intrinsic information of infrared weak targets in the first fused image, retain valuable features, and remove worthless features to obtain a more descriptive first fused feature map, thereby improving the detection rate of infrared weak targets and reducing the false alarm rate.

[0091] In some implementations, the second feature fusion module has the same network structure as the first feature fusion module.

[0092] In this embodiment, to construct more fine-grained features and features with richer semantic information, the network structure of the second feature fusion module can be the same as that of the first feature fusion module. Therefore, the step "based on the first fused feature map and the feature map extracted in the second stage, obtain the second fused feature map corresponding to each training sample" can include:

[0093] The first fused feature map is interpolated, and the number of channels of the interpolated first fused feature map is reduced to the same number of channels as the feature map after the second stage feature extraction. Then, it is added to the feature map after the second stage feature extraction to obtain the third fused map.

[0094] The third fusion map is input into the spatial attention branch and the channel attention branch respectively, and the feature maps output by the spatial attention branch and the channel attention branch are added together to obtain the fourth fusion map;

[0095] The result of multiplying the fourth fusion map with the first fusion feature map after interpolation and channel reduction is added to the result of multiplying the fourth fusion map with the feature map after the second stage feature extraction to obtain the second fusion feature map.

[0096] In this embodiment, the third fused image is input into the spatial attention branch and the channel attention branch, respectively. The spatial attention branch uses two 1×1 convolutions to output weights on the first fused image. The channel attention branch first uses adaptive pooling to reduce the input to a size of C×1×1, and then uses 1D convolution to capture cross-channel interactions in the five nearest channels to generate weights on each channel. In this way, the spatial attention branch and the channel attention branch can be used to highlight the intrinsic information of infrared weak targets in the third fused image, retain valuable features, and remove worthless features to obtain a second fused feature map with stronger descriptiveness, thereby improving the detection rate of infrared weak targets and reducing the false alarm rate.

[0097] Step H5: Input the first fused feature map and the second fused feature map into the first detection head and the second detection head respectively to obtain the output result of each training sample.

[0098] In step H5, since the first fused feature map and the second fused feature map have different scales and contain different semantic and detail features, the first detection head and the second detection head are used to detect the first fused feature map and the second fused feature map respectively, so as to improve the multi-scale detection capability of the detection model.

[0099] Step H6: Adjust the network parameters of the neural network based on the output of each training sample and the corresponding label of each training sample until a detection model that meets the expectations is obtained.

[0100] In step H6, after each batch of training samples is input into the preset neural network, the parameters of each layer in the neural network need to be calculated and stored. The deviation between the existing parameters and the label is calculated according to the loss function. The error gradient is calculated based on the deviation value and used as the basis for updating the neural network parameters for the next batch of training samples until a detection model that meets the expectations is obtained.

[0101] In some implementations, the first and second detection heads share the same set of network parameters; the first and second detection heads train the network parameters using a grouping normalization method.

[0102] In this embodiment, the first and second detection heads share the same set of network parameters, which significantly reduces the number of parameters and solves the problem of scale imbalance of the input target during training. Furthermore, both the first and second detection heads use group normalization. This is because since the two detection heads share the same set of network parameters, it is undesirable for each detection head to independently store a set of parameters during the normalization operation in the training phase. If batch normalization were used, each detection head would update the parameters of the batch normalization layer, causing parameter confusion and having a very detrimental effect on the inference results. The parameters of the group normalization layer are independent of the batch size and can be directly calculated during the inference phase, avoiding the problem of parameter confusion.

[0103] In some implementations, both the first and second detection heads include a classification branch, a regression branch, and a centrality branch.

[0104] Therefore, step H5 can include:

[0105] The first fused feature map and the second fused feature map are respectively input into the classification branch of the first detection head and the classification branch of the second detection head to obtain the classification result of each pixel in the first fused feature map and the second fused feature map;

[0106] The first fused feature map is input into the regression branch and centrality branch of the first detection head, and the second fused feature map is input into the regression branch and centrality branch of the second detection head, respectively, to obtain the regression results of the first fused feature map and the regression results of the second fused feature map.

[0107] In this embodiment, the classification branch transforms the input feature map into (B×H×W×C) layers using 4 convolutional layers and 1 output layer, where B represents the batch size, H and W represent the dimensions of the input feature map, and C represents the number of classes. The classification result of the classification branch is the class classification of each point in the input feature map. The regression branch regresses the offset of each point from the corresponding boundary of the input feature map using 4 convolutional layers and 1 output layer, and its output shape is (B×H×W×4). The centrality branch regresses the centrality of each point using 4 convolutional layers and 1 output layer.

[0108] In some implementations, the regression branch and the centrality branch share the same set of convolutional layers.

[0109] In this embodiment, since both the regression branch and the centrality branch need to obtain the position of each point, in order to reduce network parameters, the regression branch and the centrality branch can share the same set of convolutional layers. In this embodiment, the centrality branch and the regression branch share the first 4 convolutional layers, and the output shape is (B×H×W×1).

[0110] In summary, this outlines the process of constructing the detection model.

[0111] Regarding step 104:

[0112] In some implementations, after the infrared weak target image to be detected is input into the detection model trained according to the above construction process, the output results of the detection head module in the detection model include the classification and regression results of the first target feature map, and the classification and regression results of the second target feature map; wherein, the classification and regression results of the first target feature map are the output results of the first target feature map obtained by using the first detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model; the classification and regression results of the second target feature map are the output results of the second target feature map obtained by using the second detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model.

[0113] Therefore, step 104 may include:

[0114] Each pixel in the first target feature map and the second target feature map is mapped back to the infrared weak target image, so as to determine the initial target based on the classification results corresponding to the first target feature map and the second target feature map respectively;

[0115] Based on the regression results corresponding to the first target feature map and the second target feature map, generate candidate boxes for each initial target;

[0116] Calculate the distance from the center pixel of each initial target in the infrared weak target image to the corresponding candidate box, and filter each initial target according to the distance to obtain the infrared weak targets;

[0117] Calculate the normalized distance from the center pixel of each infrared weak target in the infrared weak target image to the corresponding candidate box, and then filter the candidate boxes corresponding to the infrared weak targets based on the normalized distance to obtain the detection boxes of the infrared weak targets.

[0118] In this embodiment, the output of the detection model on the infrared weak target image to be detected is a pixel-level classification and localization result similar to semantic segmentation. In order to achieve target detection, the output result needs to be converted back to the target.

[0119] Taking the first target feature map as an example, each pixel in the first target feature map is mapped back to its position in the original image, that is, its position in the infrared weak target image to be detected. If the step size is s, the positions of these pixels in the original image may fall within a size of s. 2 In the region, that is, each point on the first target feature map, can be mapped to a region of size s. 2 The grid region is defined. Initial targets are determined based on the classification results corresponding to the first target feature map. These initial targets include various categories of weak infrared targets. Then, based on the regression results corresponding to the first target feature map, candidate boxes are generated centered on each initial target. The determination method for the initial targets and their corresponding candidate boxes in the second target feature map is the same as that for the first target feature map.

[0120] Next, the distance from the center pixel of each initial target in the infrared weak target image to the corresponding candidate box is calculated.

[0121] In this embodiment, the distance is calculated using the following formula:

[0122]

[0123]

[0124]

[0125]

[0126] In the formula, the distance is (l * ,t * ,r * ,b * ), l * t * r * and b *These represent the distances from the center pixel of the initial target in the infrared image of small targets to the left, top, right, and bottom edges of the corresponding candidate bounding box, respectively. The coordinates of the top-left corner of the candidate box. These are the coordinates of the bottom right corner of the candidate selection box.

[0127] It should be noted that because the first target feature map and the second target feature map have different scales, i.e., different step sizes, the areas of the initial target mapped back to the original image on the first target feature map and the second target feature map are different, and therefore the sizes of targets suitable for detection are different. In this embodiment, the size of the first target feature map is smaller than the size of the second target feature map, which are 1 / 16 and 1 / 8 of the size of the infrared weak target image, respectively (i.e., the step size is set to 16 and 8). Therefore, the larger the step size, the larger the area of ​​the initial target mapped back to the original image, and the more suitable it is for detecting large targets. Thus, the first target feature map is suitable for detecting larger targets, and the second target feature map is suitable for detecting smaller targets.

[0128] Therefore, by predefining the upper and lower limits (floor, celling) of the target size that the first and second detection heads can detect, the size of the target detected by each detection head can be limited.

[0129] For example, since the first detection head detects the first target feature map, and the first target feature map is suitable for detecting larger targets, the upper and lower limits of the target size that the first detection head can detect are limited to (16, 32). Similarly, the upper and lower limits of the target size that the second detection head can detect are limited to (0, 16). Therefore, when the distance from the center pixel of the initial target in the infrared weak target image obtained from the output of the first detection head to the corresponding candidate box exceeds the upper and lower limits of the target size, i.e. This initial target is then recorded as a negative sample. This method limits the size of targets that each detector head can learn, ensuring that overlapping targets are detected without mistaking them for a single target, and also reducing the probability of two detector heads detecting the same target repeatedly. Therefore, each initial target can be filtered based on distance to obtain weak infrared targets.

[0130] Finally, the candidate bounding box for each infrared weak target may be significantly deviated from the center of the true bounding box. In methods without preset anchor boxes, the centrality value is used to suppress boxes that deviate from the target. Specifically, the normalized distance from the center pixel of each infrared weak target in the infrared weak target image to the corresponding candidate bounding box is calculated.

[0131] In this embodiment of the invention, the normalized distance is calculated using the following formula:

[0132]

[0133] In the formula, Centerness is the normalized distance, l * t * r * and b * These represent the left, top, right, and bottom distances of the center pixel of the infrared weak target in the infrared weak target image to the corresponding candidate box.

[0134] As shown in the above formula, the Centerness value ranges from (0,1). The larger this value, the closer the point is to the center of the true bounding box, and the greater the Centerness weight of the bounding box centered at this point. By sorting the bounding boxes using confidence and Centerness and using non-maximum suppression, low-quality bounding boxes can be filtered out, resulting in the detection bounding boxes for infrared weak targets.

[0135] Therefore, the detection results of infrared weak target images are the category and detection box of the infrared weak target.

[0136] To verify the detection performance of the proposed algorithm for weak infrared targets in this embodiment, the detection performance was actually tested on the test set, and the accuracy and speed indicators are shown in Table 1 below.

[0137] Table 1

[0138] Average precision (AP50) Processing rate The present invention 92.40％ 105 FPS

[0139] like Figure 2 , Figure 3 As shown, this embodiment of the invention provides a device for detecting small infrared targets. The device embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware architecture diagram of a computing device housing an infrared weak target detection device provided in an embodiment of the present invention, except for... Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 3 As shown, a device in a logical sense is formed by the CPU of the computing device in which it is located reading the corresponding computer program from the non-volatile memory into the memory for execution.

[0140] like Figure 3 As shown, this embodiment provides a detection device for infrared weak targets, comprising:

[0141] Acquisition unit 301 is used to acquire an image of a small infrared target to be detected;

[0142] The detection unit 302 is used to input the infrared weak target image into the pre-trained detection model; wherein, the detection model is trained based on a preset neural network, which includes a backbone network composed of four residual networks connected in series, a context feature extraction network composed of multiple dilated convolutional layers, a feature fusion network, and a detection head module;

[0143] The regression unit 303 is used to obtain the detection results of the infrared weak target image based on the output of the detection head module.

[0144] In one embodiment of the present invention, the detection head module in the detection unit 302 includes a first detection head and a second detection head;

[0145] The training methods for the detection model include:

[0146] Obtain multiple labeled training samples;

[0147] Multiple training samples are input into the backbone network, and the four residual networks in the backbone network are used to perform first-stage feature extraction, second-stage feature extraction, third-stage feature extraction and fourth-stage feature extraction on each training sample respectively, so as to obtain the feature map after each stage of feature extraction.

[0148] The feature map extracted in the fourth stage is input into each dilated convolutional layer in the context feature extraction network to obtain the expanded feature map corresponding to each training sample.

[0149] Using a feature fusion network, based on the feature map after the third stage feature extraction and the expanded feature map, a first fusion feature map corresponding to each training sample is obtained. After obtaining the first fusion feature map, a second fusion feature map corresponding to each training sample is obtained based on the first fusion feature map and the feature map after the second stage feature extraction.

[0150] The first fused feature map and the second fused feature map are respectively input into the first detection head and the second detection head to obtain the output result of each training sample;

[0151] Based on the output of each training sample and the corresponding label, the network parameters of the neural network are adjusted until a detection model that meets the expectations is obtained.

[0152] In one embodiment of the present invention, in the detection unit 302, the first detection head and the second detection head share the same set of network parameters; the first detection head and the second detection head train the network parameters using a grouping normalization method.

[0153] In one embodiment of the present invention, in the detection unit 302, both the first detection head and the second detection head include a classification branch, a regression branch and a centrality branch;

[0154] When executing the process of inputting the first fused feature map and the second fused feature map into the first detection head and the second detection head respectively to obtain the output result of each training sample, the following is performed:

[0155] The first fused feature map and the second fused feature map are respectively input into the classification branch of the first detection head and the classification branch of the second detection head to obtain the classification result of each pixel in the first fused feature map and the second fused feature map;

[0156] The first fused feature map is input into the regression branch and centrality branch of the first detection head, and the second fused feature map is input into the regression branch and centrality branch of the second detection head, respectively, to obtain the regression results of the first fused feature map and the regression results of the second fused feature map.

[0157] In one embodiment of the present invention, in the detection unit 302, the regression branch and the centrality branch share the same set of convolutional layers.

[0158] In one embodiment of the present invention, the output of the detection head module in the detection unit 302 includes the classification and regression results of the first target feature map, and the classification and regression results of the second target feature map; the classification and regression results of the first target feature map are the output results of the first target feature map obtained by using the first detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model; the classification and regression results of the second target feature map are the output results of the second target feature map obtained by using the second detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model.

[0159] Regression unit 303, when executing the detection results of the infrared weak target image based on the output of the detection head module, is used to perform:

[0160] Each pixel in the first target feature map and the second target feature map is mapped back to the infrared weak target image, so as to determine the initial target based on the classification results corresponding to the first target feature map and the second target feature map respectively;

[0161] Based on the regression results corresponding to the first target feature map and the second target feature map, generate candidate boxes for each initial target;

[0162] Calculate the distance from the center pixel of each initial target in the infrared weak target image to the corresponding candidate box, and filter each initial target according to the distance to obtain the infrared weak targets;

[0163] Calculate the normalized distance from the center pixel of each infrared weak target in the infrared weak target image to the corresponding candidate box, and then filter the candidate boxes corresponding to the infrared weak targets based on the normalized distance to obtain the detection boxes of the infrared weak targets.

[0164] In one embodiment of the present invention, in the regression unit 303, the distance is calculated using the following formula:

[0165]

[0166]

[0167]

[0168]

[0169] In the formula, the distance is (l * ,t * ,r * ,b * ), l * t * r * and b * These represent the distances from the center pixel of the initial target in the infrared image of small targets to the left, top, right, and bottom edges of the corresponding candidate bounding box, respectively. The coordinates of the top-left corner of the candidate box. The coordinates of the bottom right corner of the candidate box;

[0170] The normalized distance is calculated using the following formula:

[0171]

[0172] In the formula, Centerness is the normalized distance, l * t * r * and b * These represent the left, top, right, and bottom distances of the center pixel of the infrared weak target in the infrared weak target image to the corresponding candidate box.

[0173] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on a detection device for infrared weak targets. In other embodiments of the present invention, a detection device for infrared weak targets may include more or fewer components than illustrated, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0174] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0175] This invention also provides a computing device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a method for detecting infrared weak targets according to any embodiment of this invention.

[0176] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform a method for detecting infrared weak targets according to any embodiment of this invention.

[0177] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0178] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0179] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0180] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0181] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0182] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0183] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting weak infrared targets, characterized in that, include: Acquire images of the weak infrared targets to be detected; The infrared weak target image is input into a pre-trained detection model; wherein the detection model is trained based on a preset neural network, the preset neural network including a backbone network composed of four residual networks connected in series, a context feature extraction network composed of multiple dilated convolutional layers, a feature fusion network, and a detection head module; Based on the output of the detection head module, the detection result of the infrared weak target image is obtained; The detection head module includes a first detection head and a second detection head; The training methods for the detection model include: Obtain multiple labeled training samples; Multiple training samples are input into the backbone network, and the four residual networks in the backbone network are used to perform first-stage feature extraction, second-stage feature extraction, third-stage feature extraction and fourth-stage feature extraction on each training sample respectively, so as to obtain the feature map after each stage of feature extraction; The feature map extracted in the fourth stage is input into each dilated convolutional layer in the context feature extraction network to obtain the expanded feature map corresponding to each training sample. Using the feature fusion network, based on the feature map extracted in the third stage and the expanded feature map, a first fusion feature map corresponding to each training sample is obtained. After obtaining the first fusion feature map, a second fusion feature map corresponding to each training sample is obtained based on the first fusion feature map and the feature map extracted in the second stage. The first fused feature map and the second fused feature map are respectively input into the first detection head and the second detection head to obtain the output result of each training sample; Based on the output of each training sample and the label corresponding to each training sample, the network parameters of the neural network are adjusted until a detection model that meets the expectations is obtained. The output of the detection head module includes the classification and regression results of the first target feature map, and the classification and regression results of the second target feature map. The classification and regression results of the first target feature map are the output results of the first target feature map obtained by using the first detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model. The classification and regression results of the second target feature map are the output results of the second target feature map obtained by using the second detection head in the detection model to process the infrared weak target image through the feature fusion network in the detection model. The step of obtaining the detection result of the infrared weak target image based on the output result of the detection head module includes: Each pixel in the first target feature map and the second target feature map is mapped back into the infrared weak target image, so as to determine the initial target based on the classification results corresponding to the first target feature map and the second target feature map, respectively. Based on the regression results corresponding to the first target feature map and the second target feature map, a candidate box is generated for each of the initial targets; Calculate the distance from the center pixel of each initial target in the infrared weak target image to the corresponding candidate box, so as to filter each initial target according to the distance and obtain infrared weak targets; Calculate the normalized distance from the center pixel of each infrared weak target in the infrared weak target image to the corresponding candidate box, so as to filter the candidate boxes corresponding to the infrared weak target according to the normalized distance and obtain the detection box of the infrared weak target.

2. The method according to claim 1, characterized in that, The first detection head and the second detection head share the same set of network parameters; the first detection head and the second detection head train the network parameters using a group normalization method.

3. The method according to claim 1 or 2, characterized in that, Both the first detection head and the second detection head include classification branches, regression branches, and centrality branches; The step of inputting the first fused feature map and the second fused feature map into the first detection head and the second detection head respectively to obtain the output result of each training sample includes: The first fused feature map and the second fused feature map are respectively input into the classification branch of the first detection head and the classification branch of the second detection head to obtain the classification result of each pixel in the first fused feature map and the second fused feature map; The first fused feature map is input into the regression branch and the centrality branch of the first detection head, respectively, and the second fused feature map is input into the regression branch and the centrality branch of the second detection head, respectively, to obtain the regression results of the first fused feature map and the regression results of the second fused feature map.

4. The method according to claim 3, characterized in that, The regression branch and the centrality branch share the same set of convolutional layers.

5. The method according to claim 1, characterized in that, The distance is calculated using the following formula: In the formula, the distance is , , , and These are the distances from the center pixel of the initial target in the infrared weak target image to the left, top, right, and bottom of the corresponding candidate box, respectively. Let be the coordinates of the upper left corner of the candidate box. The coordinates of the lower right corner of the candidate box; The normalized distance is calculated using the following formula: In the formula, The normalized distance, , , and These are the left, top, right, and bottom distances of the center pixel of the infrared weak target in the infrared weak target image to the corresponding candidate box.

6. A detection device for weak infrared targets, used to implement the method as described in any one of claims 1-5, characterized in that, include: The acquisition unit is used to acquire images of the infrared weak targets to be detected; The detection unit is used to input the infrared weak target image into a pre-trained detection model; wherein the detection model is trained based on a preset neural network, and the preset neural network includes a backbone network composed of four residual networks connected in series, a context feature extraction network composed of multiple dilated convolutional layers, a feature fusion network, and a detection head module; The regression unit is used to obtain the detection result of the infrared weak target image based on the output result of the detection head module.

7. A computing device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-5.

Images