Infrared small target detection method and device, computing device, and storage medium

Abstract

The present application relates to the field of infrared detection technology, and particularly relates to an infrared small target detection method and device, a computing device and a storage medium. The method comprises: acquiring an infrared small target image to be detected; inputting the infrared small target image into a first detection model generated by pre-training; wherein the first detection model is trained based on a first neural network and a second detection model generated by training a second neural network, the first neural network and the second neural network both comprise a feature fusion module, a prediction module and a feature extraction module formed by a plurality of sub-modules in series; each sub-module of the second neural network comprises a plurality of residual networks connected in sequence; the number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network; and a detection result of the infrared small target image is obtained. The present application can reduce the parameter quantity of the infrared small target detection model while not reducing the detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of infrared detection technology, and in particular to a method, apparatus, computing device, and storage medium for detecting small infrared targets. Background Technology

[0002] Infrared detection technology has the advantage of being unaffected by environmental factors, and its applications are becoming increasingly widespread in military and civilian fields. Infrared small target detection, as an important branch of infrared image detection, plays a crucial role in early target warning and drone detection.

[0003] With the development of deep learning, deep neural networks have been applied to infrared small target detection, greatly improving the detection accuracy. However, to cope with complex learning tasks, the network models of existing infrared small target detection methods are often designed to be large and complex. This results in these models requiring a large amount of computing resources for training and deployment, making them difficult to apply in embedded and mobile devices.

[0004] Therefore, there is an urgent need for a new method for detecting small infrared targets. Summary of the Invention

[0005] To address the issue that existing infrared small target detection methods require significant computational resources for network model training and deployment, this invention provides an infrared small target detection method, apparatus, computing device, and storage medium.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting small infrared targets, comprising:

[0007] Acquire images of small infrared targets to be detected;

[0008] The infrared small target image is input into a pre-trained first detection model. The first detection model is trained based on a first neural network and a second detection model trained by a second neural network. Both the first and second neural networks include a feature fusion module, a prediction module, and a feature extraction module composed of multiple cascaded sub-modules. Each sub-module of the second neural network includes several cascaded residual networks. The number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network.

[0009] The detection results of the infrared small target image are obtained.

[0010] Preferably, the number of sub-modules in the first neural network and the number of sub-modules in the second neural network are the same;

[0011] The training methods for the first detection model include:

[0012] Obtain multiple labeled training samples;

[0013] Each training sample is input into the first neural network and the second detection model respectively, so that each sub-module of the first neural network and each sub-module of the second detection model can be used to extract features from each training sample, thereby obtaining the feature extraction results of each sub-module in the first neural network and the second detection model.

[0014] The feature extraction results of each sub-module in the first neural network are input into the feature fusion module of the first neural network to obtain a feature fusion map corresponding to each sub-module in the first neural network. The feature extraction results of each sub-module in the second detection model are input into the feature fusion module of the second detection model to obtain a feature fusion map corresponding to each sub-module in the second detection model.

[0015] The feature fusion maps corresponding to the first sub-module in the first neural network and the second detection model are respectively input into the prediction modules of the first neural network and the second detection model to obtain the prediction results of the first neural network and the second detection model for each training sample;

[0016] For each training sample, the network parameters of the first neural network are adjusted for the first time based on the feature fusion map of the current training sample in the first neural network and the second detection model and the prediction result. Then, the network parameters of the first neural network are adjusted for the second time based on the prediction result of the first neural network for the current training sample and the label corresponding to the current training sample, until the first detection model that meets the expectations is obtained.

[0017] Preferably, the feature fusion module of the first neural network and the second neural network includes multiple feature fusion sub-modules;

[0018] Each feature fusion map in the first neural network and the second detection model is obtained in the following way:

[0019] Obtain the feature extraction results of the deepest sub-module in the first neural network and the second detection model to obtain the feature fusion map corresponding to the deepest sub-module;

[0020] For each submodule except the deepest submodule, the feature extraction result of the current submodule and the feature fusion map corresponding to the deeper submodule connected in series with the current submodule are input into the corresponding feature fusion submodule to obtain the feature fusion map corresponding to each submodule except the deepest submodule.

[0021] Preferably, before obtaining the feature fusion map corresponding to the deepest sub-module, the method further includes: upsampling the feature extraction results of the deepest sub-modules in the first neural network and the second detection model respectively to obtain the feature fusion map corresponding to the deepest sub-modules in the first neural network and the second detection model.

[0022] Before obtaining the feature fusion map corresponding to each sub-module except the deepest sub-module, the method further includes: upsampling the output of each feature fusion sub-module to obtain the feature fusion map corresponding to each sub-module except the deepest sub-module in the first neural network and the second detection model.

[0023] Preferably, the first adjustment of the network parameters of the first neural network based on each feature fusion map of the current training sample in the first neural network and the second detection model and the prediction result includes:

[0024] The feature fusion maps corresponding to the first sub-module in the first neural network and the second detection model are respectively input to the corresponding softmax layer, and the cross-entropy loss function is used to calculate the difference between the feature fusion map corresponding to the first sub-module of the first neural network and the feature fusion map corresponding to the first sub-module of the second detection model, so as to make the first adjustment of the network parameters of the first sub-module of the first neural network.

[0025] For each submodule other than the first submodule, the feature fusion map corresponding to the current submodule in the second detection model and the feature fusion map corresponding to the shallower submodule connected in series with the current submodule are spliced ​​together to make the network parameters of each submodule other than the first submodule in the first neural network for the first adjustment.

[0026] Based on the prediction results of the first neural network and the second detection model for the current training samples, the network parameters of the prediction module of the first neural network are adjusted for the first time.

[0027] Preferably, before performing the splicing operation on the feature fusion map corresponding to the current sub-module in the second detection model and the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module, the method further includes: downsampling the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module.

[0028] Preferably, each softmax layer is calculated according to the following formula:

[0029]

[0030] In the formula, q i Let T be the output of each neuron in the softmax layer, T be the temperature parameter, Z be the input of each neuron in the softmax layer, and i and j be variables.

[0031] Secondly, embodiments of the present invention also provide an infrared small target detection device, comprising:

[0032] The acquisition unit is used to acquire an image of the small infrared target to be detected.

[0033] An input unit is used to input the infrared small target image into a pre-trained first detection model; wherein the first detection model is trained based on a first neural network and a second detection model trained by a second neural network, and both the first neural network and the second neural network include a feature fusion module, a prediction module, and a feature extraction module composed of multiple sub-modules connected in series; each sub-module of the second neural network includes several cascaded residual networks; the number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network;

[0034] The output unit is used to obtain the detection results of the infrared small target image.

[0035] Thirdly, embodiments of the present invention also provide a computing device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of this specification.

[0036] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.

[0037] This invention provides a method, apparatus, computing device, and storage medium for detecting small infrared targets. First, the image of the small infrared target to be detected is input into a pre-trained first detection model. In this scheme, the first detection model is trained based on a first neural network and a second detection model trained by a second neural network. Both the first and second neural networks include a feature fusion module, a prediction module, and a feature extraction module composed of multiple cascaded sub-modules. Each sub-module of the second neural network includes several cascaded residual networks. The number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network. This scheme utilizes the information contained in the second detection model to guide the training of the first neural network. Because the second neural network is deeper and more complex than the first neural network, the resulting first detection model, while lightweight, can also learn the deeper "knowledge" of the second detection model, achieving the same detection accuracy as the second detection model. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of an infrared small target detection method provided in an embodiment of the present invention;

[0040] Figure 2 This is a hardware architecture diagram of a computing device provided in an embodiment of the present invention;

[0041] Figure 3 This is a structural diagram of an infrared small target detection device provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0043] As mentioned earlier, in order to cope with complex learning tasks, the network models of existing infrared small target detection methods are often designed to be large and complex. This results in the need for a lot of computing resources to train and deploy these models, making them difficult to apply in embedded and mobile devices.

[0044] To address the aforementioned technical issues, the inventors could consider transferring knowledge from a complex, deep second detection model to a smaller first detection model, thereby improving the accuracy of the lightweight first detection model. Therefore, this solution can reduce the number of parameters in the infrared small target detection model without lowering detection accuracy, thus solving the problem that existing infrared small target detection methods require significant computational resources for both training and deployment of their network models.

[0045] The following describes the specific implementation of the above concept.

[0046] Please refer to Figure 1 This invention provides a method for detecting small infrared targets, the method comprising:

[0047] Step 100: Obtain an image of the small infrared target to be detected;

[0048] Step 102: Input the infrared small target image into the pre-trained first detection model; wherein, the first detection model is trained based on a first neural network and a second detection model trained by a second neural network, and both the first and second neural networks include a feature fusion module, a prediction module, and a feature extraction module composed of multiple sub-modules connected in series; each sub-module of the second neural network includes several cascaded residual networks; the number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network;

[0049] Step 104: Obtain the detection results of the infrared small target image.

[0050] In this embodiment of the invention, the infrared small target image to be detected is first input into a pre-trained first detection model. In this scheme, the first detection model is trained based on a first neural network and a second detection model trained by a second neural network. Both the first and second neural networks include a feature fusion module, a prediction module, and a feature extraction module composed of multiple cascaded sub-modules. Each sub-module of the second neural network includes several cascaded residual networks. The number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network. This scheme utilizes the information contained in the second detection model to guide the training of the first neural network. Because the second neural network is more deep and complex than the first neural network, the resulting first detection model, while lightweight, can also learn the deeper "knowledge" of the second detection model, achieving the same detection accuracy as the second detection model.

[0051] For steps 100 and 102:

[0052] In some implementations, the number of submodules in the first neural network is the same as the number of submodules in the second neural network;

[0053] The training method for the first detection model may include steps S1-S5:

[0054] S1: Obtain multiple labeled training samples;

[0055] S2: Input each training sample into the first neural network and the second detection model respectively, so as to use each sub-module of the first neural network and each sub-module of the second detection model to extract features from each training sample, and obtain the feature extraction results of each sub-module in the first neural network and the second detection model.

[0056] S3: Input the feature extraction results of each sub-module in the first neural network into the feature fusion module of the first neural network to obtain a feature fusion map corresponding to each sub-module in the first neural network. Input the feature extraction results of each sub-module in the second detection model into the feature fusion module of the second detection model to obtain a feature fusion map corresponding to each sub-module in the second detection model.

[0057] S4: Input the feature fusion map corresponding to the first sub-module in the first neural network and the second detection model into the prediction module of the first neural network and the second detection model respectively to obtain the prediction results of the first neural network and the second detection model for each training sample;

[0058] S5: For each training sample, based on the feature fusion map and prediction results of the current training sample in the first neural network and the second detection model, the network parameters of the first neural network are adjusted for the first time. Based on the prediction results of the first neural network for the current training sample and the label corresponding to the current training sample, the network parameters of the first neural network are adjusted for the second time until the first detection model that meets the expectations is obtained.

[0059] For example, both the first and second neural networks include a feature fusion module, a prediction module, and a feature extraction module consisting of three cascaded sub-modules. Each sub-module of the second neural network includes three cascaded residual networks, while each sub-module of the lightweight first neural network includes only one residual network. The three sub-modules of the second neural network are denoted as stage1, stage2, and stage3; the three sub-modules of the first neural network are denoted as layers1, layers2, and layers3.

[0060] The training method of the first detection model will be illustrated below.

[0061] In step S2, each training sample is input into the first neural network and the second detection model respectively, so that each sub-module of the first neural network and each sub-module of the second detection model can be used to extract features from each training sample. The feature extraction results of the three sub-modules in the first neural network and the feature extraction results of the three sub-modules in the second detection model can be obtained.

[0062] In step S3, the feature extraction results of each sub-module in the first neural network are input into the feature fusion module of the first neural network to obtain a feature fusion map corresponding to each sub-module in the first neural network, namely the feature fusion maps corresponding to layers1, layers2 and layers3; the feature extraction results of each sub-module in the second detection model are input into the feature fusion module of the second detection model to obtain a feature fusion map corresponding to each sub-module in the second detection model, namely the feature fusion maps corresponding to stage1, stage2 and stage3.

[0063] In step S4, the feature fusion map corresponding to layer1 in the first neural network is input into the prediction module of the first neural network to obtain the prediction result of the first neural network for each training sample; similarly, the feature fusion map corresponding to stage1 in the second detection model is input into the prediction module of the second detection model to obtain the prediction result of the second detection model for each training sample.

[0064] In step S5, for each training sample, the following steps are performed: the network parameters of the first neural network are adjusted for the first time based on the feature fusion map and prediction results of the current training sample in the first neural network and the second detection model; and the network parameters of the first neural network are adjusted for the second time based on the prediction results of the first neural network for the current training sample and the label corresponding to the current training sample, until the first detection model that meets the expectations is obtained.

[0065] In this embodiment, the intermediate features of the second detection model, namely each feature fusion map and prediction result, are used as training supervision signals for the first neural network. The network parameters of each layer of the first neural network are adjusted for the first time. Then, the network parameters of the first neural network are adjusted for the second time based on the prediction results of the first neural network and the labels corresponding to the training samples. Although the first detection model has fewer network parameters than the second detection model, it can quickly learn the "knowledge" of the second detection model, which greatly saves the training and application costs of the first detection model.

[0066] It should be noted that the network parameters of the second detection model are fixed. By inputting training samples into the second detection model, the feature extraction and fusion results and prediction results of each layer of the second detection model are obtained. These results are then compared with the feature extraction and fusion results and prediction results of the corresponding layers of the first neural network. This allows the first neural network to quickly learn the network parameters that have already been trained in the second detection model. It can be understood that compared to the first neural network, which relies solely on the labels of the training samples for training, this approach can significantly improve the training speed and detection accuracy of the first detection model.

[0067] In some implementations, the feature fusion module of the first neural network and the second neural network includes multiple feature fusion sub-modules;

[0068] Each feature fusion map in the first neural network and the second detection model is obtained in the following way:

[0069] Obtain the feature extraction results of the deepest sub-module in the first neural network and the second detection model to obtain the feature fusion map corresponding to the deepest sub-module;

[0070] For each submodule except the deepest submodule, the feature extraction result of the current submodule and the feature fusion map corresponding to the deeper submodule connected in series with the current submodule are input into the corresponding feature fusion submodule to obtain the feature fusion map corresponding to each submodule except the deepest submodule.

[0071] For example, since the feature extraction modules of the first neural network and the second neural network listed in this embodiment both include 3 sub-modules, the feature fusion modules of the first neural network and the second neural network can each include 2 feature fusion sub-modules.

[0072] In step S3, each feature fusion map in the first neural network and the second detection model is obtained as follows:

[0073] The feature extraction results of the deepest sub-modules in the first neural network and the second detection model, namely the feature extraction results of layers3 and stage3, are directly used as the feature fusion map corresponding to layers3 and stage3.

[0074] Next, the process of obtaining the feature fusion maps corresponding to layers1, layers2, stage1, and stage2 will be explained.

[0075] The feature extraction results of layers2 and the feature fusion map corresponding to layers3 in the first neural network are input into the first feature fusion submodule corresponding to the first neural network to obtain the feature fusion map corresponding to layers2; similarly, the feature extraction results of stage2 and the feature fusion map corresponding to stage3 in the second detection model are input into the first feature fusion submodule corresponding to the second detection model to obtain the feature fusion map corresponding to stage2.

[0076] Similarly, the feature extraction results of layers1 and the feature fusion map corresponding to layers2 in the first neural network are input into the second feature fusion submodule corresponding to the first neural network to obtain the feature fusion map corresponding to layers1; the feature extraction results of stage1 and the feature fusion map corresponding to stage2 in the second detection model are input into the second feature fusion submodule corresponding to the second detection model to obtain the feature fusion map corresponding to stage1.

[0077] In this embodiment, the proposed ACM module is used as a feature fusion sub-module to fuse the relationships between contexts, which can enhance the interaction of deep and shallow semantic information in the first neural network and the second detection model.

[0078] In some implementations, before obtaining the feature fusion map corresponding to the deepest sub-module, the method further includes: upsampling the feature extraction results of the deepest sub-module in the first neural network and the second detection model respectively to obtain the feature fusion map corresponding to the deepest sub-module in the first neural network and the second detection model.

[0079] Before obtaining the feature fusion map corresponding to each submodule except the deepest submodule, the process further includes: upsampling the output of each feature fusion submodule to obtain the feature fusion map corresponding to each submodule except the deepest submodule in the first neural network and the second detection model.

[0080] In this embodiment, in step S3, the feature extraction results of layers3 and stage3 are upsampled respectively, and the upsampled results are used as the feature fusion map corresponding to layers3 and the feature fusion map corresponding to stage3.

[0081] Then, the feature extraction results of layers 2 and the corresponding feature fusion map of layers 3 in the first neural network are input into the first feature fusion submodule of the first neural network. The feature map output by the first feature fusion submodule is then upsampled to obtain the feature fusion map corresponding to layers 2. The feature extraction results of layers 1 and the corresponding feature fusion map of layers 2 in the first neural network are input into the second feature fusion submodule. The feature map output by the second feature fusion submodule is then upsampled to obtain the feature fusion map corresponding to layers 1. Similarly, the upsampled feature fusion maps corresponding to stage 1 and stage 2 in the second detection model can be obtained.

[0082] In this embodiment, upsampling is performed before obtaining the feature fusion maps corresponding to each submodule. On the one hand, this increases the resolution of each feature fusion map, which is beneficial for the first adjustment of the network parameters of the first neural network. On the other hand, it increases the size of the deep feature fusion map so as to match the size of the deep and shallow feature maps when performing feature fusion through the feature fusion submodule.

[0083] In some implementations, step S5, "adjusting the network parameters of the first neural network for the first time based on each feature fusion map and prediction result of the current training samples in the first neural network and the second detection model," may include the following steps H1-H3:

[0084] H1: Input the feature fusion maps corresponding to the first sub-module in the first neural network and the second detection model into the corresponding softmax layers respectively, and use the cross-entropy loss function to calculate the difference between the feature fusion map corresponding to the first sub-module of the first neural network and the feature fusion map corresponding to the first sub-module of the second detection model, so as to make the first adjustment of the network parameters of the first sub-module of the first neural network.

[0085] H2: For each submodule other than the first submodule, the feature fusion map corresponding to the current submodule in the second detection model and the feature fusion map corresponding to the shallower submodule connected in series with the current submodule are spliced ​​together to make the first adjustment of the network parameters of each submodule in the first neural network other than the first submodule.

[0086] H3: Based on the prediction results of the first neural network and the second detection model for the current training samples, the network parameters of the prediction module of the first neural network are adjusted for the first time.

[0087] For example, in step H1, the feature fusion map corresponding to the first neural network and the first sub-module, i.e., the feature fusion map corresponding to layers1, is input into the first softmax layer of the first neural network, and the feature fusion map corresponding to the second detection model stage1 is input into the first softmax layer of the second detection model. Then, the difference between the feature fusion map corresponding to layers1 and the feature fusion map corresponding to stage1 is calculated using the cross-entropy loss function, and fed back to the layers1 and second feature fusion sub-modules of the first neural network to make the first adjustment to the network parameters of the layers1 and second feature fusion sub-modules of the first neural network.

[0088] In step H2, the feature fusion map corresponding to stage 2 and stage 1 in the second detection model are concatenated, and the concatenated feature tensor is then input into the second softmax layer of the second detection model. Meanwhile, the feature fusion map corresponding to layers 2 of the first neural network is input into the second softmax layer of the first neural network. The difference between the feature tensor and the feature fusion map corresponding to layers 2 is calculated using the cross-entropy loss function and fed back to layers 2 and the first feature fusion submodule of the first neural network for the first adjustment of their network parameters. Similarly, the feature fusion map corresponding to stage 3 and stage 2 in the second detection model are concatenated, and the concatenated feature tensor is input into the third softmax layer of the second detection model. Meanwhile, the feature fusion map corresponding to layers 3 of the first neural network is input into the third softmax layer of the first neural network. The difference between the feature tensor and the feature fusion map corresponding to layers 3 is calculated using the cross-entropy loss function and fed back to layers 3 of the first neural network for the first adjustment of their network parameters.

[0089] In step H3, the prediction result output by the prediction module of the first neural network is input into the fourth softmax layer of the first neural network, and the prediction result output by the prediction module of the second detection model is input into the fourth softmax layer of the second detection model. Then, the difference between the prediction result of the first neural network and the prediction result of the second detection model is calculated using the cross-entropy loss function and fed back to the prediction module of the first neural network to make the first adjustment to the network parameters of the prediction module of the first neural network.

[0090] In this embodiment, directly using the intermediate layers of the second detection model to transfer knowledge to the corresponding layers in the first neural network is a strong constraint. This restricts the first neural network to only learning the representations of specific intermediate layers in the second detection model for a particular block. However, in reality, each block in the first neural network should be able to learn the intermediate layer representations of non-corresponding blocks in the second detection model. Furthermore, the intermediate layer representations of the second detection model contain not only useful information but also useless information. Simply having the first neural network mimic the intermediate layer representations of the second detection model will result in learning useless information as well.

[0091] Therefore, in this embodiment, for each sub-module of the second detection model, the feature fusion map corresponding to the current sub-module is not only transferred to the corresponding sub-module of the first neural network for knowledge transfer, but also passed to the next-level sub-module for knowledge transfer to the next sub-module of the first neural network. Here, before performing the concatenation operation in step H2, 1x1 convolution is used for dimensionality reduction. On the one hand, this matches the feature dimensions of the first neural network; on the other hand, the convolution provides the first neural network with the possibility of selection: that is, the first neural network can choose which layers of the second detection model, and can also selectively learn useful information in the second detection model at the model optimization level, while rejecting the learning of useless information. Through more relaxed restrictions and more connections, the first neural network has more learning choices and better performance.

[0092] In some implementations, before concatenating the feature fusion map corresponding to the current sub-module in the second detection model and the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module, the method further includes: downsampling the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module.

[0093] In this embodiment, before using the concate operation to connect and form a feature tensor, downsampling can be used to reduce the size of the upper-level feature fusion map to a suitable value, so as to better stitch together feature fusion maps of different levels.

[0094] In some implementations, each softmax layer is calculated according to the following formula:

[0095]

[0096] In the formula, q i Let T be the output of each neuron in the softmax layer, T be the temperature parameter, Z be the input of each neuron in the softmax layer, and i and j be variables.

[0097] Before the prediction module and each sub-module use the cross-entropy loss function, each softmax layer activates the input feature map. When training the first detection model, a higher temperature parameter is used to make the distribution of the softmax output soft enough, so that the softmax output of the first detection model can be closer to that of the second detection model.

[0098] Regarding step 104:

[0099] After obtaining the first detection model that meets the expectations, its network parameters are fixed. The temperature parameter T of the softmax layer in the first detection model is set to 1. Then, after the infrared small target image to be detected is input into the first detection model, the prediction module of the first detection model can output the detection result of the infrared small target image.

[0100] like Figure 2 , Figure 3 As shown, this embodiment of the invention provides a device for detecting small infrared targets. The device embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, as... Figure 2 The diagram shown is a hardware architecture diagram of a computing device housing an infrared small target detection device provided in an embodiment of the present invention. (Except for...) Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 3 As shown, a device in a logical sense is formed by the CPU of its computing device reading the corresponding computer program from non-volatile memory into memory and running it. This embodiment provides a small infrared target detection device, including:

[0101] Acquisition unit 301 is used to acquire an image of the infrared small target to be detected;

[0102] The input unit 302 is used to input an infrared small target image into a pre-trained first detection model; wherein, the first detection model is trained based on a first neural network and a second detection model trained by a second neural network, and both the first neural network and the second neural network include a feature fusion module, a prediction module, and a feature extraction module composed of multiple sub-modules connected in series; each sub-module of the second neural network includes several cascaded residual networks; the number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network;

[0103] The output unit 303 is used to obtain the detection results of infrared small target images.

[0104] In one embodiment of the present invention, the number of sub-modules in the first neural network and the number of sub-modules in the second neural network in the input unit 302 are the same;

[0105] When performing training to generate the first detection model, it is used for:

[0106] Obtain multiple labeled training samples;

[0107] Each training sample is input into the first neural network and the second detection model respectively, so that each sub-module of the first neural network and each sub-module of the second detection model can be used to extract features from each training sample, thereby obtaining the feature extraction results of each sub-module in the first neural network and the second detection model.

[0108] The feature extraction results of each sub-module in the first neural network are input into the feature fusion module of the first neural network to obtain a feature fusion map corresponding to each sub-module in the first neural network. The feature extraction results of each sub-module in the second detection model are input into the feature fusion module of the second detection model to obtain a feature fusion map corresponding to each sub-module in the second detection model.

[0109] The feature fusion maps corresponding to the first sub-module in the first neural network and the second detection model are input into the prediction modules of the first neural network and the second detection model, respectively, to obtain the prediction results of the first neural network and the second detection model for each training sample.

[0110] For each training sample, the network parameters of the first neural network are adjusted for the first time based on the feature fusion map and prediction results of the current training sample in the first neural network and the second detection model. Then, the network parameters of the first neural network are adjusted for the second time based on the prediction results of the first neural network for the current training sample and the label corresponding to the current training sample, until the first detection model that meets the expectations is obtained.

[0111] In one embodiment of the present invention, the feature fusion module of the first neural network and the second neural network in the input unit 302 includes multiple feature fusion sub-modules;

[0112] Each feature fusion map in the first neural network and the second detection model is obtained in the following way:

[0113] Obtain the feature extraction results of the deepest sub-module in the first neural network and the second detection model to obtain the feature fusion map corresponding to the deepest sub-module;

[0114] For each submodule except the deepest submodule, the feature extraction result of the current submodule and the feature fusion map corresponding to the deeper submodule connected in series with the current submodule are input into the corresponding feature fusion submodule to obtain the feature fusion map corresponding to each submodule except the deepest submodule.

[0115] In one embodiment of the present invention, before the input unit 302 obtains the feature fusion map corresponding to the deepest sub-module, it is further used to upsample the feature extraction results of the deepest sub-module in the first neural network and the second detection model respectively to obtain the feature fusion map corresponding to the deepest sub-module in the first neural network and the second detection model.

[0116] Before obtaining the feature fusion map corresponding to each submodule except the deepest submodule, it is also used to upsample the output of each feature fusion submodule to obtain the feature fusion map corresponding to each submodule except the deepest submodule in the first neural network and the second detection model.

[0117] In one embodiment of the present invention, when the input unit 302 performs the first adjustment of the network parameters of the first neural network based on each feature fusion map and prediction result in the first neural network and the second detection model using the current training samples, it is specifically used to perform:

[0118] The feature fusion maps corresponding to the first sub-module in the first neural network and the second detection model are respectively input into the corresponding softmax layer, and the cross-entropy loss function is used to calculate the difference between the feature fusion map corresponding to the first sub-module of the first neural network and the feature fusion map corresponding to the first sub-module of the second detection model, so as to make the first adjustment of the network parameters of the first sub-module of the first neural network.

[0119] For each submodule except the first submodule, the feature fusion map corresponding to the current submodule in the second detection model and the feature fusion map corresponding to the shallower submodule connected in series with the current submodule are spliced ​​together to make the network parameters of each submodule except the first submodule in the first neural network for the first adjustment.

[0120] Based on the prediction results of the first neural network and the second detection model for the current training samples, the network parameters of the prediction module of the first neural network are adjusted for the first time.

[0121] In one embodiment of the present invention, before performing the operation of splicing the feature fusion map corresponding to the current sub-module in the second detection model and the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module, the input unit 302 is further configured to downsample the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module.

[0122] In one embodiment of the present invention, in the input unit 302, each softmax layer is calculated according to the following formula:

[0123]

[0124] In the formula, q i Let T be the output of each neuron in the softmax layer, T be the temperature parameter, Z be the input of each neuron in the softmax layer, and i and j be variables.

[0125] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on an infrared small target detection device. In other embodiments of the present invention, an infrared small target detection device may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0126] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0127] This invention also provides a computing device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a method for detecting small infrared targets according to any embodiment of this invention.

[0128] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform a method for detecting small infrared targets according to any embodiment of this invention.

[0129] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0130] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0131] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0132] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0133] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0135] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting small infrared targets, characterized in that, include: Acquire images of small infrared targets to be detected; The infrared small target image is input into a pre-trained first detection model. The first detection model is trained based on a first neural network and a second detection model trained by a second neural network. Both the first and second neural networks include a feature fusion module, a prediction module, and a feature extraction module composed of multiple cascaded sub-modules. Each sub-module of the second neural network includes several cascaded residual networks. The number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network. The detection results of the infrared small target image are obtained; The number of sub-modules in the first neural network is the same as the number of sub-modules in the second neural network; The training methods for the first detection model include: Obtain multiple labeled training samples; Each training sample is input into the first neural network and the second detection model respectively, so that each sub-module of the first neural network and each sub-module of the second detection model can be used to extract features from each training sample, thereby obtaining the feature extraction results of each sub-module in the first neural network and the second detection model. The feature extraction results of each sub-module in the first neural network are input into the feature fusion module of the first neural network to obtain a feature fusion map corresponding to each sub-module in the first neural network. The feature extraction results of each sub-module in the second detection model are input into the feature fusion module of the second detection model to obtain a feature fusion map corresponding to each sub-module in the second detection model. The feature fusion maps corresponding to the first sub-module in the first neural network and the second detection model are respectively input into the prediction modules of the first neural network and the second detection model to obtain the prediction results of the first neural network and the second detection model for each training sample; For each training sample, the network parameters of the first neural network are adjusted for the first time based on the feature fusion map of the current training sample in the first neural network and the second detection model and the prediction result. Then, the network parameters of the first neural network are adjusted for the second time based on the prediction result of the first neural network for the current training sample and the label corresponding to the current training sample, until the first detection model that meets the expectations is obtained.

2. The method according to claim 1, characterized in that, The feature fusion modules of the first neural network and the second neural network include multiple feature fusion sub-modules; Each feature fusion map in the first neural network and the second detection model is obtained in the following way: Obtain the feature extraction results of the deepest sub-module in the first neural network and the second detection model to obtain the feature fusion map corresponding to the deepest sub-module; For each submodule except the deepest submodule, the feature extraction result of the current submodule and the feature fusion map corresponding to the deeper submodule connected in series with the current submodule are input into the corresponding feature fusion submodule to obtain the feature fusion map corresponding to each submodule except the deepest submodule.

3. The method according to claim 2, characterized in that, Before obtaining the feature fusion map corresponding to the deepest sub-module, the method further includes: upsampling the feature extraction results of the deepest sub-modules in the first neural network and the second detection model respectively to obtain the feature fusion map corresponding to the deepest sub-modules in the first neural network and the second detection model. Before obtaining the feature fusion map corresponding to each sub-module except the deepest sub-module, the method further includes: upsampling the output result of each feature fusion sub-module to obtain the feature fusion map corresponding to each sub-module except the deepest sub-module in the first neural network and the second detection model.

4. The method according to claim 1, characterized in that, The first adjustment of the network parameters of the first neural network based on each feature fusion map of the current training samples in the first neural network and the second detection model and the prediction result includes: The feature fusion maps corresponding to the first sub-module in the first neural network and the second detection model are respectively input to the corresponding softmax layer, and the cross-entropy loss function is used to calculate the difference between the feature fusion map corresponding to the first sub-module of the first neural network and the feature fusion map corresponding to the first sub-module of the second detection model, so as to make the first adjustment of the network parameters of the first sub-module of the first neural network. For each submodule other than the first submodule, the feature fusion map corresponding to the current submodule in the second detection model and the feature fusion map corresponding to the shallower submodule connected in series with the current submodule are spliced ​​together to make the network parameters of each submodule other than the first submodule in the first neural network for the first adjustment. Based on the prediction results of the first neural network and the second detection model for the current training samples, the network parameters of the prediction module of the first neural network are adjusted for the first time.

5. The method according to claim 4, characterized in that, Before performing the concatenation operation between the feature fusion map corresponding to the current sub-module in the second detection model and the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module, the method further includes: downsampling the feature fusion map corresponding to the shallower sub-module connected in series with the current sub-module.

6. The method according to claim 4, characterized in that, Each of the softmax layers is calculated according to the following formula: In the formula, q i Let T be the output of each neuron in the softmax layer, T be the temperature parameter, Z be the input of each neuron in the softmax layer, and i and j be variables.

7. A detection device for small infrared targets, used to implement the method as described in any one of claims 1-6, characterized in that, include: The acquisition unit is used to acquire an image of the small infrared target to be detected. An input unit is used to input the infrared small target image into a pre-trained first detection model; wherein the first detection model is trained based on a first neural network and a second detection model trained by a second neural network, and both the first neural network and the second neural network include a feature fusion module, a prediction module, and a feature extraction module composed of multiple sub-modules connected in series; each sub-module of the second neural network includes several cascaded residual networks; the number of residual networks in each sub-module of the first neural network is less than the number of residual networks in each sub-module of the second neural network; The output unit is used to obtain the detection results of the infrared small target image.

8. A computing device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-6.

Images