Model training method and device, infrared weak and small target detection method and device and electronic equipment

Abstract

The invention discloses a model training method and an infrared weak and small target detection method, a trained model takes a Yolov3 model as a framework, and a detection layer corresponding to a large-size feature map in the Yolov3 model is cut off, so that the model focuses on small and medium targets of an image; a channel attention mechanism and a space attention mechanism are further utilized, shallow layer features extracted from a backbone network of a Yov3 model are regarded as effective features, the shallow layer features are utilized to guide middle layer features, the model selectively pays attention to more important features when weak and small targets are detected, effective features of other layers are embedded, detail information of images is effectively utilized, and the detection accuracy of the weak and small targets is improved. The fitting capability and the detection performance of the model are enhanced; in addition, a joint loss function is established, the scale sensitivity of weak and small target position regression on similarity measurement is relieved, multiple measurement modes make up each other, a position prediction frame is guided from multiple angles, and the accuracy of prediction of the model on the final position of the target is optimized.

Description

Technical Field

[0001] The present invention belongs to image processing technology, and in particular relates to a model training method, an infrared weak small target detection method, a device and an electronic device. Background Art

[0002] Infrared imaging technology uses infrared light instead of natural light to generate images. Infrared imaging systems are widely used due to their continuous working ability and high anti-interference ability. Among them, infrared weak target detection plays a vital role in computer vision tasks. The "small" of infrared weak targets is mainly manifested in the small number of pixels occupied by the target in the image, which is due to the long detection distance; "weak" is mainly manifested in the weak signal intensity of the target and the low contrast with the background, mainly due to atmospheric attenuation. Severe background interference, fuzzy shape of the target and other factors seriously restrict the performance of infrared weak target detection. How to accurately and comprehensively detect infrared weak targets is still a problem that needs to be solved urgently.

[0003] At present, most of the infrared weak small target detection technologies can be divided into multi-frame based and single-frame based. The multi-frame based detection algorithm takes multiple time-series infrared images as input, combines the prior information of motion trajectory continuity, and fuses the time domain and spatial domain information of multiple frames to complete target detection. This makes it unsuitable for non-continuous frame scenes and has a limited scope of application. Secondly, multi-frame image analysis is also prone to high algorithm computational complexity and poor real-time performance. The single-frame based detection algorithm only needs single-frame image information to complete target detection, which can make up for the disadvantages of the multi-frame based detection algorithm. It can be roughly divided into model-driven traditional methods and data-driven deep learning methods. However, most detection models also increase redundant features while increasing the network level. The features of small targets are easily submerged in deep features, and the features with higher task relevance cannot be fully utilized. In addition, the size of infrared weak small targets is too small, and they are more affected by the same predicted position offset and are more sensitive to position deviation. Summary of the invention

[0004] Based on this, the present invention aims to propose a model training method and an infrared dim small target detection method, which make full use of the more representative features of infrared images to guide other prediction layer features, reduce the sensitivity of infrared dim small targets to position deviations through a joint loss function, and make position prediction more accurate.

[0005] In a first aspect, the present invention provides a model training method, wherein a model trained by the method is used for infrared dim small target detection, comprising:

[0006] Get training infrared images with true labels;

[0007] An infrared target detection model is established, which includes a guided attention module and an improved Yolov3 model. The guided attention module includes a channel attention mechanism and a spatial attention mechanism, and the detection network of the improved Yolov3 model has only two detection layers.

[0008] The training infrared image is input into the infrared target detection model for model training. The training infrared image is passed through the backbone network in the improved Yolov3 model to obtain shallow features, intermediate features and deep features. The shallow features and intermediate features are passed through the guided attention module to obtain fused features. The fused features and deep features are passed through the detection network of the improved Yolov3 model to obtain the prediction results.

[0009] A joint loss function is constructed based on the real labels and prediction results of the training infrared images. The model parameters are adjusted with the goal of minimizing the loss function to obtain a trained infrared target detection model.

[0010] Furthermore, the feature layer and its prediction layer corresponding to the last layer of feature maps are deleted in the improved Yolov3 model, so that the detection network of the improved Yolov3 model has only two prediction layers corresponding to the shallow features and the middle layer features respectively.

[0011] Furthermore, the shallow layer features and the intermediate layer features are guided by the attention module to obtain fused features including:

[0012] The shallow features are extracted through the channel attention mechanism and the spatial attention mechanism to obtain the channel feature weight and the spatial feature weight;

[0013] Divide the intermediate layer features into two groups along the channel and multiply them with the channel feature weight and the spatial feature weight respectively to obtain the first enhanced feature and the second enhanced feature;

[0014] The first strengthening feature and the second strengthening feature are connected along the channel to form a fusion feature.

[0015] Furthermore, the spatial attention mechanism includes the first convolution layer, the horizontal pooling layer, the vertical pooling layer, the second convolution layer, the normalization layer, the activation layer, and the feature adaptation module in sequence along the direction of feature input;

[0016] The shallow feature is resized by the first convolution layer to obtain the first feature. The first feature is respectively passed through the horizontal pooling layer and the vertical pooling layer to obtain the first spatial aggregation feature and the second spatial aggregation feature. The second spatial aggregation feature is transposed and spliced ​​with the first spatial aggregation feature to obtain the spliced ​​feature. The spliced ​​feature is sequentially passed through the second convolution layer, the normalization layer, and the activation layer to obtain the second feature. The second feature is split into the first spatial attention feature and the second spatial attention feature in the horizontal and vertical directions. The first spatial attention feature and the second spatial attention feature are respectively input into the feature adaptation module for adaptive adjustment to obtain the first spatial feature weight and the second spatial feature weight.

[0017] Furthermore, the channel attention mechanism includes the third convolutional layer, the global pooling layer, and the multi-layer perceptron in the direction of feature input;

[0018] The shallow features are resized by the third convolutional layer to obtain the third features. The third features are squeezed and aggregated by the global pooling layer to obtain the global features. The global features are adaptively adjusted by the multi-layer perceptron to obtain the channel feature weights.

[0019] Furthermore, the shallow and intermediate features are calculated by guiding the attention module to obtain the fusion features as follows:

[0020]

[0021] in, represents fusion features, CAT represents feature concatenation, and They represent two sets of features split along the channel of the intermediate layer features, SA is the spatial attention mechanism, and CA is the channel attention mechanism. Represents shallow features, and represents the spatial feature weight, Represents the channel feature weight.

[0022] Furthermore, the joint loss function includes a position regression loss function, which jointly uses the IoU metric, the NWD metric, and the image center point distance metric.

[0023] Furthermore, the joint loss function also includes confidence loss and classification loss.

[0024] Furthermore, the position regression loss function is calculated as follows:

[0025]

[0026] in, represents the position regression loss, H / W represents the height / width of the final prediction image, and M represents the number of anchor boxes. Indicates whether there is an object in the i-th cell and whether the j-th anchor box in the i-th cell should predict the target, ρ represents the Euclidean distance, and λ represents the balance coefficient used to balance the IoU metric and the NWD metric. ( ) / ( ) represents the (horizontal center position, vertical center position, width, height) of the predicted / true box. Represents the NWD metric.

[0027] In a second aspect, the present invention provides an infrared small target detection method, comprising:

[0028] Acquire the infrared image to be detected;

[0029] The infrared image to be detected is input into the infrared target detection model trained by the model training method of the first aspect to perform weak target detection to obtain the target detection result.

[0030] In a third aspect, the present invention provides a model training device, comprising:

[0031] A training image acquisition module, used to acquire training infrared images with real labels;

[0032] A model building module is used to build an infrared target detection model including a guided attention module and an improved Yolov3 model, wherein the guided attention module includes a channel attention mechanism and a spatial attention mechanism, and the detection network of the improved Yolov3 model has only two detection layers;

[0033] The model training module is used to input the training infrared image into the infrared target detection model for model training. The training infrared image is passed through the backbone network in the improved Yolov3 model to obtain shallow features, intermediate features and deep features. The shallow features and intermediate features are passed through the guided attention module to obtain fused features. The fused features and deep features are passed through the detection network of the improved Yolov3 model to obtain prediction results.

[0034] The loss calculation update module is used to construct a joint loss function based on the real labels and prediction results of the training infrared images, adjust the model parameters with the goal of minimizing the loss function, and obtain a trained infrared target detection model.

[0035] In a fourth aspect, the present invention provides an infrared small target detection device, comprising:

[0036] An infrared image acquisition module is used to acquire the infrared image to be detected;

[0037] The target detection module is used to input the infrared image to be detected into the infrared target detection model trained by the model training method of the first aspect to perform weak target detection and obtain the target detection result.

[0038] In a fifth aspect, the present invention provides an electronic device, comprising: a memory and a processor;

[0039] The memory is used to store programs;

[0040] The processor is used to call the program stored in the memory to execute the model training method provided by the above-mentioned first aspect embodiment and / or any possible implementation method combined with the first aspect embodiment, or to execute the infrared weak small target detection method provided by the above-mentioned second aspect embodiment.

[0041] In a sixth aspect, the present invention further provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the model training method provided in the above-mentioned first aspect embodiment and / or any possible implementation method in combination with the first aspect embodiment, or executes the infrared weak small target detection method provided in the above-mentioned second aspect embodiment.

[0042] Compared with the existing infrared small target deep learning detection model, the present invention has the following beneficial effects:

[0043] The invention provides a model training method and an infrared weak small target detection method. The trained model takes the Yolov3 model as a framework, considers the full utilization of effective features in an image, cuts off the detection layer corresponding to a large-size feature map in the Yolov3 model, and makes the model focus more on small and medium-sized targets in the image; proposes a guiding attention module, which includes a channel attention mechanism and a spatial attention mechanism, regards shallow features extracted from a backbone network of the Yolov3 model as effective features, uses shallow features to guide middle-layer features, makes the model selectively focus on more important features when detecting weak small targets, and embeds effective features of other layers, effectively utilizes the detailed information of the image, and enhances the fitting ability and detection performance of the model; in addition, the invention also proposes to establish a joint loss function, alleviates the scale sensitivity of weak small target position regression in similarity measurement, and multiple measurement methods complement each other, guides the position prediction frame from multiple angles, and is conducive to optimizing the accuracy of the model in predicting the final position of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 It is the network architecture of the infrared target detection model provided in the embodiment of the present application;

[0046] Figure 2 A flow chart of an infrared target detection model training method provided in an embodiment of the present application;

[0047] Figure 3 This is the original Yolov3 model architecture diagram used in the infrared target detection model of the embodiment of the present application;

[0048] Figure 4 It is a heat map-like visualization of the feature map extracted by the original Yolov3 model backbone network in the embodiment of the present application;

[0049] Figure 5 This is a diagram of the spatial attention mechanism architecture provided by an embodiment of the present application;

[0050] Figure 6 This is a channel attention mechanism architecture diagram provided in an embodiment of the present application;

[0051] Figure 7 This is a comparison chart of IoU metric calculations for target images of different sizes provided in an embodiment of the present application;

[0052] Figure 8 A flowchart of an infrared small target detection method provided in an embodiment of the present application;

[0053] Fig. 9 A structural diagram of a model training device provided in an embodiment of the present application;

[0054] Fig.10 A structural diagram of an infrared small target detection device provided in an embodiment of the present application;

[0055] Fig.11 This is a diagram of the electronic device architecture provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] like Figure 1 As shown, it shows the network structure of the infrared target detection model proposed in an embodiment of the present application, including a guided attention module and an improved Yolov3 model, wherein the guided attention module includes a channel attention mechanism and a spatial attention mechanism, and the detection network of the improved Yolov3 model has only two detection layers.

[0058] The input infrared image enters the backbone network of the Yolov3 model to extract feature maps of different scales. The shallow features and the middle layer features are passed through the guided attention module to obtain fused features. In the guided attention module, the shallow features are used to guide the middle layer features to embed effective information. The deep features extracted by the backbone network and the fused features enter the detection network together. The present invention removes the last layer of features and the corresponding detection layer in the original Yolov3 model to enhance the feature representation capability, and uses the remaining two detection layers to predict small targets. The prediction results and the real labels of the images are measured for similarity through a joint loss function, and the measurement deviation guides model learning and adjustment until the model converges.

[0059] See also Figure 2 An embodiment of the present application provides a method based on Figure 1 The model training method of the network architecture is used for infrared dim small target detection, and the model trained by this method includes the following steps:

[0060] Step S210: Obtain a training infrared image with a true label.

[0061] Step S220. Establish an infrared target detection model including a guided attention module and an improved Yolov3 model, wherein the guided attention module includes a channel attention mechanism and a spatial attention mechanism, and the detection network of the improved Yolov3 model has only two detection layers.

[0062] Step S230. Input the training infrared image into the infrared target detection model for model training. The training infrared image is passed through the backbone network in the improved Yolov3 model to obtain shallow features, intermediate features and deep features. The shallow features and intermediate features are passed through the guided attention module to obtain fused features. The fused features and deep features are passed through the detection network of the improved Yolov3 model to obtain prediction results.

[0063] Step S240. Construct a joint loss function based on the real labels and prediction results of the training infrared images, adjust the model parameters with the goal of minimizing the loss function, and obtain a trained infrared target detection model.

[0064] The above embodiment provides a model training method, which takes the Yolov3 model as the framework, considers the full utilization of effective features in the image, and cuts off the detection layer corresponding to the large-size feature map in the Yolov3 model, so that the model focuses more on small and medium-sized targets in the image; proposes a guided attention module, which includes a channel attention mechanism and a spatial attention mechanism, regards the shallow features extracted from the backbone network of the Yolov3 model as effective features, and uses the shallow features to guide the intermediate layer features, so that the model selectively focuses on more important features when detecting weak targets, and embeds effective features of other layers, effectively utilizes the detailed information of the image, and enhances the fitting ability and detection performance of the model; in addition, the present invention also proposes to establish a joint loss function to alleviate the scale sensitivity of the weak target position regression in the similarity measurement, and multiple measurement methods complement each other, and guide the position prediction frame from multiple angles, which is conducive to optimizing the accuracy of the model's prediction of the final position of the target.

[0065] In further embodiments, see Figure 3 and Figure 4 , which illustrates an improvement process of the Yolov3 model in the present application, deleting the feature layer and its prediction layer corresponding to the last layer of feature map, so that the detection network of the improved Yolov3 model has only two prediction layers corresponding to the shallow layer features and the middle layer features respectively.

[0066] In a convolutional neural network, when the network becomes deeper and more complex, while improving the model fitting potential, it will also bring more redundant features. Redundant features not only bring more computational complexity, but may even reduce model performance. Therefore, it is necessary to first select effective features that are strongly related to the task. Specifically, in this embodiment, the class activation heat map generated by the feature map of each layer extracted in the Yolov3 model backbone network is visualized to show where the center of gravity of the feature map of each scale in the model is for the image.

[0067] Specifically, this embodiment uses the Yolov3 model including three detection layers as the framework of the infrared target detection model. Figure 3 As shown, the original Yolov3 model can extract 6 feature maps of different resolutions at the backbone network, which are represented as , , , , , For infrared weak targets, the activation heat maps are generated for these six feature maps respectively as follows: Figure 4 shown.

[0068] Figure 4 Displays the input infrared image, the true label map, and the feature map ~ The input image has two targets, such as Figure 4 (b) As shown in the white pixel points in the red detection box, the input image also includes an obvious noise point with a similar appearance to the target, marked with a green box.

[0069] like Figure 4 (c) It is a class heat map corresponding to the feature map after the input image is convolved once, and the extracted features are not obvious; It is a shallow feature extracted by multiple residual blocks. It has a high response to both targets and noise points, and cannot distinguish between noise and targets. It shows the ability of high response to dual targets and low response to noise; and Compared to , lost the target on the right, It is more responsive to the underlying part of the image. Then both targets are lost. Figure 4 Various heat maps show that Features are more relevant to the task, Features are weakly related to the task.

[0070] Therefore, this embodiment chooses to convert the original Yolov3 model The above feature layers are pruned, these feature layers are considered redundant, and the feature layers are reduced The corresponding prediction layer changes from three prediction layers to two prediction layers, and the shallow features Considered as valid features, such as Figure 1 shown.

[0071] In a further embodiment, the attention module is guided to use shallow features that are more representative of weak targets to guide the attention of deep features used for prediction, so that effective shallow features guide prediction layer features, embed effective information, and enhance the model's representation ability.

[0072] In some embodiments, the spatial attention mechanism used includes a first convolution layer, a horizontal pooling layer, a vertical pooling layer, a second convolution layer, a normalization layer, an activation layer, and a feature adaptation module in sequence along the direction of feature input. The shallow feature is resized by the first convolution layer to obtain the first feature, the first feature is respectively passed through the horizontal pooling layer and the vertical pooling layer to obtain the first spatial aggregation feature and the second spatial aggregation feature, the second spatial aggregation feature is transposed and then spliced ​​with the first spatial aggregation feature to obtain the spliced ​​feature, the spliced ​​feature is sequentially passed through the second convolution layer, the normalization layer, and the activation layer to obtain the second feature, the second feature is split into the first spatial attention feature and the second spatial attention feature in the horizontal direction and the vertical direction, the first spatial attention feature and the second spatial attention feature are respectively input into the feature adaptation module for adaptive adjustment to obtain the first spatial feature weight and the second spatial feature weight.

[0073] Specifically, the spatial attention mechanism used in this embodiment is as follows: Figure 5 As shown, the shallow feature F low Enter the first convolutional layer (Conv) for size adjustment, and use the first convolutional layer to adjust the size of the guidance feature map to the same size as the guided feature map. Figure 1 The average pooling operation is then performed along the horizontal (X Avg Pool) and vertical (Y Avg Pool) directions to obtain spatial aggregation features in two directions. The features obtained by pooling along the vertical direction are then transposed and concatenated with the features obtained by pooling along the horizontal direction. The concatenated features obtain spatial perception in two directions. The concatenated features are input into the second convolutional layer (Conv), normalization layer (BN) and activation layer (Leaky ReLU) for feature adjustment. The adjusted features are split into adjusted horizontal and vertical spatial attention features. The process can be expressed as follows:

[0074]

[0075] in, / represents the adjusted horizontal and vertical spatial attention features, Split represents the split feature operation, δ represents the Leaky ReLU activation function, and BN represents the normalization operation. / represents the convolution operation, CAT represents the concat feature concatenation, / Represents an average pooling operation along the horizontal / vertical direction.

[0076] Then, the adjusted horizontal and vertical spatial attention features are / The convolution layer (Conv) and activation layer (Sigmoid) in the feature adaptive adjustment module are input respectively for feature adaptive adjustment again to obtain the final spatial perception attention features in the horizontal and vertical directions. The process can be expressed as follows:

[0077]

[0078]

[0079] in, / It represents the spatial perception attention feature finally obtained by the shallow feature through the spatial attention mechanism, σ represents the Sigmoid activation function, and W represents the convolution operation.

[0080] In a further embodiment, the channel attention mechanism used in this embodiment includes a third convolutional layer, a global pooling layer, and a multi-layer perceptron in sequence along the direction of feature input; the shallow feature is resized by the third convolutional layer to obtain the third feature, the third feature is squeezed and aggregated by the global pooling layer to obtain the global feature, and the global feature is adaptively adjusted by the multi-layer perceptron to obtain the channel feature weight.

[0081] Specifically, the channel attention mechanism used in this embodiment is as follows: Figure 6 As shown, the shallow feature F low Enter the convolutional layer (Conv) for size adjustment, and use the convolutional layer to adjust the size of the guiding feature map to the same size as the guided feature map. Figure 1 Then, the global pooling operation (Global Avg Pool) is used to squeeze and aggregate the features along the spatial dimension to obtain global information, and the multi-layer perceptron (MLP) is used to adaptively adjust the channel weights. The process can be defined as follows:

[0082]

[0083]

[0084] Among them, MLP represents multi-layer perceptron, GAP represents global average pooling operation, and W represents convolution operation; the multi-layer perceptron MLP consists of two fully connected layers and an activation layer, which models the correlation between feature channels and adaptively adjusts the feature channel weights.

[0085] The channel attention mechanism assigns a weight to each channel feature. Different weights represent different importance of features, which enables the network to focus on channel features with large weight values.

[0086] In a further embodiment, based on the spatial attention mechanism and channel attention mechanism described in the foregoing embodiments, the shallow features and the intermediate layer features of step S230 are guided by the attention module to obtain the fused features, which includes the following steps:

[0087] Step S231. Extract shallow features through channel attention mechanism and spatial attention mechanism respectively to obtain channel feature weight and spatial feature weight;

[0088] Step S232: Divide the intermediate layer features into two groups along the channel and multiply them by the channel feature weight and the spatial feature weight respectively to obtain the first enhanced feature and the second enhanced feature;

[0089] Step S233: Connect the first enhancement feature and the second enhancement feature along the channel to form a fusion feature.

[0090] Specifically, Figure 1 The characteristic conduction process is shown in the schematic diagram. The spatial feature weights and channel feature weights are extracted through the spatial attention mechanism and channel attention mechanism respectively, and then the deeper features are The channels are divided into two groups, which are multiplied by the extracted spatial feature weights and channel feature weights respectively to strengthen the more important features in the spatial and channel dimensions respectively. Finally, they are connected along the channels to form features with more detailed information guided by effective feature attention. The process can be expressed as follows:

[0091]

[0092] in, represents the fusion feature enhanced by the guided attention module, CAT represents concat feature concatenation, and Represents the guided intermediate layer features Two sets of features split along the channel, SA represents the spatial attention mechanism, and CA represents the channel attention mechanism. Represents the effective shallow features of the input , and represents a pair of spatially aware feature weights obtained through the spatial attention mechanism, Represents the channel feature weight extracted by the channel attention mechanism.

[0093] In a further embodiment, the joint loss function constructed in step S240 includes a position regression loss function, wherein the position regression loss function jointly uses the IoU metric, the NWD metric, and the image center point distance metric.

[0094] Specifically, due to different measurement methods, even if the predicted value and the true value are the same, different difference values ​​and different reverse gradient values ​​will be obtained, which will affect the iteration direction of the model. Small and weak targets are more sensitive to position offsets and require more dimensional difference measurements to fully consider the differences between the predicted box and the true box at different angles to improve the convergence of the model in terms of position regression.

[0095] like Figure 7 As shown in the figure, for a 3×3 target, when the center deviation between the predicted value and the true value is 1 pixel, the IoU metric value is 0.29; for a 5×5 target, when the center deviation is 1 pixel, the IoU metric value is 0.47. It can be seen that for targets of different scales, there is a large difference in their IoU metric values ​​for the same deviation, that is, the IoU metric is highly sensitive to scale, and the smaller the scale of the target, the more sensitive it is to the IoU metric, that is, when having the same position deviation, the smaller the target, the worse the IoU metric value between its predicted position and the true position.

[0096] The embodiment of the present application proposes a joint loss function, which jointly uses two different position regression metrics, namely the IoU metric and the normalized Gauss-Worthern distance (NWD metric), and the metric of the center point of the key position target box as the position regression loss function, to improve the accuracy of the model target position regression prediction value, and jointly uses multiple position loss functions to perform multiple similarity metrics on the predicted box and the true box instead of a single similarity metric, thereby reducing the scale sensitivity of small target position regression and achieving more accurate prediction of the small target position.

[0097] Specifically, the position regression loss function adds the Euclidean distance metric and NWD metric between the center point of the predicted box and the true box on the basis of the IoU loss. This not only highlights the key position of the predicted box - the center point, but also reduces the sensitivity of weak targets to targets.

[0098] The position regression loss function proposed in this embodiment is calculated as follows:

[0099]

[0100] in, represents the position regression loss, H / W represents the height / width of the final prediction image, and M represents the number of anchor boxes. Indicates whether there is an object in the i-th cell and whether the j-th anchor box in the i-th cell should predict the target, ρ represents the Euclidean distance, and λ represents the balance coefficient used to balance the IoU metric and the NWD metric. ( ) / ( ) represents the (horizontal center position, vertical center position, width, height) of the predicted / true box. Represents the NWD metric.

[0101] The NWD metric first models the location box as a two-dimensional Gaussian distribution, and measures the similarity of the two-dimensional Gaussian distribution by normalizing the Gauss-Wasserstein distance, which is calculated as follows:

[0102]

[0103] Among them, exp represents the exponential operation with base e, and C represents a constant related to the data set.

[0104] In a further embodiment, in addition to the position regression loss, the joint loss function of the embodiment of the present application also includes confidence loss and classification loss. The complete joint loss function is expressed as follows:

[0105]

[0106] in, represents the joint loss function, represents the classification loss, Represents the confidence loss.

[0107] The classification loss and confidence loss are calculated using cross entropy and are expressed as follows:

[0108]

[0109]

[0110] Among them, classes represents the target type, / ( / ) represents the true / predicted confidence (score), represents the cross entropy loss.

[0111] The prediction results of the constructed model include target classification and position regression. The target classification output is confidence (Confidence, C) and classification probability (probability, P). The position regression output is the horizontal / vertical coordinates, length, and width (x, y, w, h) of the center point of the target box.

[0112] Through the above model training method, a more accurate infrared target detection model can be obtained. Figure 8 A method for detecting infrared dim small targets using the model according to an embodiment of the present invention is described.

[0113] like Figure 8 As shown, an embodiment of the present invention provides an infrared small target detection method, comprising the following steps:

[0114] Step S810: Acquire the infrared image to be detected.

[0115] Step S820: Input the infrared image to be detected into the infrared target detection model trained as above to perform small target detection and obtain the target detection result.

[0116] The above disclosed embodiments describe in detail a model training method and an infrared weak small target detection method. The above disclosed method can be implemented using various forms of equipment. Therefore, the present invention also discloses a device corresponding to the above method. Specific embodiments are given below for detailed description.

[0117] like Fig. 9 As shown, one embodiment of the present invention provides a model training device, comprising:

[0118] The training image acquisition module 902 is used to acquire a training infrared image with a real label;

[0119] A model building module 904 is used to build an infrared target detection model including a guided attention module and an improved Yolov3 model, wherein the guided attention module includes a channel attention mechanism and a spatial attention mechanism, and the detection network of the improved Yolov3 model has only two detection layers;

[0120] The model training module 906 is used to input the training infrared image into the infrared target detection model for model training. The training infrared image is passed through the backbone network in the improved Yolov3 model to obtain shallow features, intermediate features and deep features. The shallow features and intermediate features are passed through the guided attention module to obtain fused features. The fused features and deep features are passed through the detection network of the improved Yolov3 model to obtain prediction results.

[0121] The loss calculation update module 908 is used to construct a joint loss function based on the real label and prediction result of the training infrared image, adjust the model parameters with the goal of minimizing the loss function, and obtain a trained infrared target detection model.

[0122] like Fig.10 As shown, an embodiment of the present invention provides an infrared small target detection device, comprising:

[0123] The infrared image acquisition module 1002 is used to acquire the infrared image to be detected;

[0124] The target detection module 1004 is used to input the infrared image to be detected into the trained infrared target detection model to perform weak target detection and obtain the target detection result.

[0125] The implementation principle and technical effects of the device provided in the above embodiments of the present application are the same as those of the above method embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the above method embodiments.

[0126] The following embodiments are described by taking the method applied to a computer device as an example. It can be understood that the computer device can be any device with computing and processing functions, and can be but not limited to a server or a personal laptop computer, etc. In one embodiment, the computer device can be an application server, and the application server can be a server for running an application to be tested.

[0127] See also Fig.11 , which shows a hardware block diagram of an electronic device, the electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0128] like Fig.11 As shown, the electronic device includes: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0129] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0130] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0131] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0132] Among them, the memory stores a program, and the processor can call the program stored in the memory, and the program is used to: implement each processing flow of the aforementioned model training method, or implement each processing flow of the aforementioned infrared weak target detection method.

[0133] An embodiment of the present invention also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the various processing flows of the aforementioned model training method are implemented, or the various processing flows of the aforementioned infrared weak small target detection method are implemented.

[0134] The above-mentioned embodiments have described the present invention in particular detail with respect to possible scenarios, and those skilled in the art will recognize that the present invention can be practiced through other embodiments. The specific naming of components, the capitalization of terms, attributes, data structures, or any other programming or structural aspects are not mandatory or important, and the mechanisms or features of the present invention may have different names, forms, or procedures. The system may be implemented by a combination of hardware and software (as described), entirely by hardware elements, or entirely by software elements. The specific division of functions between the various system components described herein is exemplary only and not mandatory; on the contrary, the functions performed by a single system component may be performed by multiple components, or the functions performed by multiple components may be performed by a single component.

[0135] Those skilled in the art should understand that the various steps of the above disclosed method can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0136] These computing device executable programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors, and these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0137] Certain aspects of the present invention include process steps and instructions described herein in the form of algorithms. It should be noted that the process steps and instructions of the present invention can be implemented in software, firmware and / or hardware, and when implemented by software, it can be downloaded, stored on different platforms used by various operating systems and operated from the platforms.

[0138] Those skilled in the art will understand that the structures shown in the accompanying drawings are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "possible design" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0140] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0141] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, characterized in that: The model trained by the method is used for infrared dim small target detection, including: Get training infrared images with true labels; Establishing an infrared target detection model including a guided attention module and an improved Yolov3 model, wherein the guided attention module includes a channel attention mechanism and a spatial attention mechanism, and the detection network of the improved Yolov3 model has only two detection layers; The training infrared image is input into the infrared target detection model for model training. The training infrared image is passed through the backbone network in the improved Yolov3 model to obtain shallow features, intermediate features and deep features. The shallow features and intermediate features are passed through the guided attention module to obtain fused features. The fused features and deep features are passed through the detection network of the improved Yolov3 model to obtain prediction results. A joint loss function is constructed according to the real labels and prediction results of the training infrared images, and the model parameters are adjusted with the goal of minimizing the loss function to obtain a trained infrared target detection model.

2. The method according to claim 1, characterized in that The feature layer and its prediction layer corresponding to the last layer of feature maps are deleted in the improved Yolov3 model, so that the detection network of the improved Yolov3 model has only two prediction layers corresponding to the shallow layer features and the middle layer features respectively.

3. The method according to claim 1, characterized in that The shallow layer features and the intermediate layer features are fused by the guided attention module to obtain the following features: The shallow features are respectively extracted by the channel attention mechanism and the spatial attention mechanism to obtain channel feature weights and spatial feature weights; Divide the intermediate layer features into two groups along the channel and multiply them by the channel feature weight and the spatial feature weight respectively to obtain a first enhanced feature and a second enhanced feature; The first reinforcement feature and the second reinforcement feature are connected along the channel to form the fusion feature.

4. The method according to claim 1, characterized in that: The spatial attention mechanism includes a first convolution layer, a horizontal pooling layer, a vertical pooling layer, a second convolution layer, a normalization layer, an activation layer, and a feature adaptation module in sequence along the direction of feature input; The shallow feature is resized by the first convolution layer to obtain the first feature, the first feature is respectively passed through the horizontal pooling layer and the vertical pooling layer to obtain the first spatial aggregation feature and the second spatial aggregation feature, the second spatial aggregation feature is transposed and then spliced ​​with the first spatial aggregation feature to obtain the spliced ​​feature, the spliced ​​feature is sequentially passed through the second convolution layer, the normalization layer, and the activation layer to obtain the second feature, the second feature is split into the first spatial attention feature and the second spatial attention feature in the horizontal direction and the vertical direction, the first spatial attention feature and the second spatial attention feature are respectively input into the feature adaptation module for adaptive adjustment to obtain the first spatial feature weight and the second spatial feature weight.

5. The method according to claim 1, characterized in that The channel attention mechanism includes a third convolutional layer, a global pooling layer, and a multi-layer perceptron in sequence along the direction of feature input; The shallow features are resized by the third convolutional layer to obtain the third features, the third features are squeezed and aggregated by the global pooling layer to obtain the global features, and the global features are adaptively adjusted by the multi-layer perceptron to obtain the channel feature weights.

6. The method according to claim 1, characterized in that The joint loss function includes a position regression loss function, and the position regression loss function jointly uses the IoU metric, the NWD metric, and the image center point distance metric.

7. A method for detecting infrared small targets, characterized in that: include: Acquire the infrared image to be detected; The infrared image to be detected is input into the infrared target detection model trained by the model training method described in any one of claims 1 to 6 to perform weak target detection to obtain a target detection result.

8. A model training device, characterized in that: include: A training image acquisition module, used to acquire training infrared images with real labels; A model building module is used to build an infrared target detection model including a guided attention module and an improved Yolov3 model, wherein the guided attention module includes a channel attention mechanism and a spatial attention mechanism, and the detection network of the improved Yolov3 model has only two detection layers; The model training module is used to input the training infrared image into the infrared target detection model for model training. The training infrared image is passed through the backbone network in the improved Yolov3 model to obtain shallow features, intermediate features and deep features. The shallow features and intermediate features are passed through the guided attention module to obtain fused features. The fused features and deep features are passed through the detection network of the improved Yolov3 model to obtain prediction results. The loss calculation update module is used to construct a joint loss function based on the real labels and prediction results of the training infrared images, adjust the model parameters with the goal of minimizing the loss function, and obtain a trained infrared target detection model.

9. An infrared small target detection device, characterized in that: include: An infrared image acquisition module is used to acquire the infrared image to be detected; The target detection module is used to input the infrared image to be detected into the infrared target detection model trained by the model training device as described in claim 8 to perform weak target detection and obtain the target detection result.

10. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to call the program stored in the memory to execute the model training method as described in any one of claims 1 to 6, or to execute the infrared weak target detection method as described in claim 7.

Images