An intelligent detection method for infrared small targets based on multi-frame difference

Abstract

This application discloses a method for intelligent infrared small target detection based on multi-frame difference, which relates to the field of sequential image detection technology. The method comprises: determining a current frame image to be detected in an infrared sequence image, and screening a number of image frames that meet a preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set; registering all image frames in the image frame set based on the current frame image to obtain a number of frame registration maps, and performing a frame difference operation on the current frame image with each of the registration maps to obtain a number of frame difference images; superimposing the several frame difference images in sequence order to obtain a difference image of the current frame image, and inputting the difference image into a pre-built semantic segmentation network to output a small target detection result. This application can improve the detection performance and accuracy of small target detection in infrared sequence images.

Description

Technical Field

[0001] The present invention relates to the technical field of sequence image detection, and in particular to an intelligent detection method for infrared small targets based on multi-frame difference. Background Art

[0002] With the development of infrared imaging technology, infrared small target detection technology can be used to detect drones and ships, as well as autonomous driving and pedestrian detection, so this technology has a wide range of applications. Infrared image sequences are composed of several single-frame images that are continuous in time. For infrared image sequences, in addition to the spatial domain information in the single-frame images, the target's motion characteristics in the time domain profile are also an important basis for target detection.

[0003] Traditional target detection is already very mature and generally goes through three steps: background suppression, threshold segmentation, and target detection. Traditional sequence small target detection processing methods include filtering based on time domain and spatial domain. Among them, spatial domain filtering has large performance differences for different background features, is not easy to handle complex backgrounds, has weak generalization ability, and often requires manual parameter adjustment for different targets and backgrounds; time domain filtering mainly uses the principle of frame difference method, and the detection results will have problems such as background misalignment, repeated calculation, and large errors.

[0004] Compared to traditional methods, deep learning methods are currently more widely used in the field of object detection. Deep learning can proactively optimize network parameters by learning image features through neural networks. With the development of deep learning, deep learning-based object detection technology has made significant progress. Existing deep learning networks primarily focus on single frames. However, due to the small size and weak energy of small objects, detecting small objects in infrared image sequences using existing deep learning methods suffers from high time costs, poor accuracy, and limited detection performance.

[0005] In summary, how to improve the detection performance and accuracy of small target detection in infrared sequence images is a problem that needs to be solved. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide an intelligent infrared small target detection method based on multi-frame differencing, which can improve the detection performance and accuracy of small targets in infrared sequence images. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses an intelligent detection method for infrared small targets based on multi-frame difference, comprising:

[0008] Determining a current frame image to be detected in the infrared sequence image, and selecting a number of image frames that meet a preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set;

[0009] registering all image frames in the image frame set based on the current frame image to obtain a plurality of frame registration images, and performing a frame difference operation between the current frame image and each of the registration images to obtain a plurality of frame difference images;

[0010] Several frame difference images are superimposed in sequence to obtain a differential image of the current frame image, and the differential image is input into a pre-built semantic segmentation network to output a small target detection result.

[0011] Optionally, registering all image frames in the image frame set based on the current frame image to obtain a plurality of frame registration maps includes:

[0012] Using the current frame image as a reference image for a registration operation;

[0013] All image frames in the image frame set are registered based on the reference image using a scale-invariant feature transformation algorithm to obtain a plurality of frame registration images.

[0014] Optionally, the process of performing a frame difference operation on the current frame image and each of the registration images to obtain a plurality of frame difference images further includes:

[0015] A target area in which a frame difference result between the current frame image and the registration image in a frame difference operation is a negative value is determined, and a grayscale value of the target area is set to zero.

[0016] Optionally, superimposing a plurality of the frame difference images in a sequential order to obtain a differential image of the current frame image includes:

[0017] A bitwise AND operation is performed on the pixel values ​​of a plurality of the frame difference images in a sequential order to obtain a differential image of the current frame image.

[0018] Optionally, the multi-frame difference-based infrared small target intelligent detection method further includes:

[0019] The semantic segmentation network is constructed in advance based on a downsampling model, an upsampling model, a spatial attention gate model and a loss compensation model.

[0020] Optionally, inputting the difference image into a pre-built semantic segmentation network to output a small object detection result includes:

[0021] Performing downsampling processing on the input differential image a preset number of times using the downsampling model to obtain a corresponding downsampled image;

[0022] Processing the downsampled image using the spatial attention gate model and the upsampling model to obtain a corresponding image processing result;

[0023] The image processing result and the mask image of the current frame image are processed using the loss compensation model to obtain a small target detection result.

[0024] Optionally, before determining the current frame image to be detected in the infrared sequence image, the method further includes:

[0025] Using a preset annotation tool to annotate all image frames in the infrared sequence image, and determining the bounding box of the target in each image frame based on the annotation results;

[0026] The area outside the boundary box is masked to obtain a mask image corresponding to each image frame.

[0027] Optionally, the using the loss compensation model to process the image processing result and the mask image of the current frame image to obtain a small target detection result includes:

[0028] Processing the image processing result using a preset activation function to obtain a target output result;

[0029] Obtaining a mask image of the current frame image, determining an IoU value between the mask image and the target output result using an IoU method, and constructing a loss function based on the IoU value;

[0030] The semantic segmentation network is iteratively trained using the loss function until a preset number of training times is met, and the current target output result is used as the small target detection result.

[0031] Optionally, the process of processing the downsampled image using the spatial attention gate model and the upsampling model to obtain a corresponding image processing result further includes:

[0032] Determining an upsampled image corresponding to the output of the upsampled model based on the feature dimension of the downsampled image currently being processed, and convolving the downsampled image and the upsampled image with corresponding convolution kernels to obtain corresponding first and second convolution results;

[0033] Linearly adding the first convolution result and the second convolution result, and processing the addition result using a ReLU activation function to obtain a first activation result;

[0034] Convolving the first activation result to obtain a third convolution result, and processing the third convolution result using a sigmoid activation function to obtain a second activation result;

[0035] Performing an upsampling process on the second activation result to obtain an attention weight coefficient, and weighting the downsampled image using the attention weight coefficient to obtain an attention weighted result;

[0036] The attention weighted result and the upsampling result are upsampled to obtain an updated upsampled image and output it, and the process jumps back to the step of determining the upsampled image output in the upsampling model based on the feature dimensions of the currently processed downsampled image, until all downsampled images are processed to obtain the corresponding image processing results.

[0037] It can be seen that the present application determines the current frame image to be detected in the infrared sequence image, and filters out a number of image frames that meet the preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set; based on the current frame image, all image frames in the image frame set are aligned to obtain a number of frame alignment maps, and the current frame image is subjected to frame difference operation with each of the alignment maps to obtain a number of frame difference images; several frame difference images are superimposed in sequence order to obtain a differential image of the current frame image, and the differential image is input into a pre-constructed semantic segmentation network to output a small target detection result. It can be seen that after determining the current frame image to be detected in the infrared sequence image, the present application first needs to screen out the image frames that meet the preset frame interval with the current frame image to obtain an image frame set, and then perform registration processing on all the image frames in the image frame set based on the current frame image to obtain several frame registration images, and then perform frame difference operations on the current frame image and each registration image respectively to obtain several frame difference images, and superimpose several frame difference images in sequence order to obtain a differential image of the current frame image. That is, the present application uses a multi-frame difference method based on multi-frame accumulation and designs registration in the frame difference algorithm, which can improve the signal-to-noise ratio of background suppression under various complex backgrounds, perform background suppression on the current frame image, make small target detection more accurate, and the multi-frame difference method can adapt to complex environments and is more robust than inter-frame difference. Finally, the difference image is fed into a pre-built semantic segmentation network to output small target detection results. Due to the use of multi-frame differencing as preprocessing, the signal-to-noise ratio of the difference image input to the semantic segmentation network is improved, and its features are sparse, which further increases the network's training speed, improves the semantic segmentation's temporal filtering capabilities, and enhances its adaptability to different scenarios. As a result, the above scheme improves the overall detection performance and accuracy of small target detection in infrared image sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0039] Figure 1 This is a flow chart of an intelligent detection method for infrared small targets based on multi-frame difference disclosed in this application;

[0040] Figure 2 This is a flowchart of a specific infrared small target intelligent detection method based on multi-frame difference disclosed in this application;

[0041] Figure 3 This is a flowchart of another specific infrared small target intelligent detection method based on multi-frame difference disclosed in this application;

[0042] Figure 4 This is a schematic diagram of a multi-frame difference-based intelligent detection process for infrared small targets disclosed in this application;

[0043] Figure 5 This is a schematic diagram of the upsampling, downsampling and attention gate processes in a semantic segmentation network disclosed in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] Currently, deep learning methods are more widely used in the field of target detection. Deep learning can actively optimize network parameters by learning the features of images through neural networks. With the development of deep learning, target detection technology based on deep learning has made great progress. Existing deep learning networks are mainly based on single frames. However, due to the small size and weak energy of small targets, the use of existing deep learning methods for small target detection in infrared sequence images has the problems of high time cost, poor accuracy, and limited detection performance. To this end, the embodiment of the present application discloses an intelligent detection method for infrared small targets based on multi-frame difference, which can improve the detection performance and accuracy of small target detection in infrared sequence images.

[0046] See also Figure 1 As shown, the embodiment of the present application discloses an intelligent detection method for infrared small targets based on multi-frame difference, which includes:

[0047] Step S11: determining a current frame image to be detected in the infrared sequence image, and selecting a number of image frames that meet a preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set.

[0048] In this embodiment, first, the infrared sequence image to be detected is obtained, and the current frame image to be detected in the infrared sequence image is determined. It can be understood that the single frame image being processed in the infrared sequence image is the current frame image, which is denoted by I e Then, several image frames I that meet the preset frame interval with the current frame image are selected from the infrared sequence image. i To obtain a set of image frames, that is, to find a number of image frames I with a step τ in the infrared sequence image i In a specific embodiment, in one embodiment, the step τ takes a fixed value of 5, and the preset frame intervals are 5, 10, 15, 20, etc., then the image frame I i is the total number of infrared images in the sequence with I e An image set consisting of single-frame images with an interval of n×5 frames, where n is 1, 2, 3, etc.

[0049] It should be pointed out that since the subsequent frame difference operation needs to be performed based on the current frame image and the filtered image frames, if only a distant frame is used for frame difference, the time interval is too long, and the relative motion within the background causes uncontrollable background clutter, triggering false alarms and reducing the detection accuracy; and if only adjacent frames are used for frame difference, since the drone target is far away from the aerial photography equipment and its normal movement speed is slow, the target will overlap, and the inter-frame subtraction will cause the overlapping part of the target to be cut off, reducing the detection rate. Therefore, this application sets the current frame image I e With image frame I i Having a step τ can reduce repeated features and reduce target overlap.

[0050] Step S12: registering all image frames in the image frame set based on the current frame image to obtain a plurality of frame registration images, and performing a frame difference operation on the current frame image and each of the registration images to obtain a plurality of frame difference images.

[0051] In this embodiment, after the current frame image and the image frame set are determined, based on the current frame image I e For all image frames I in the image frame set i Perform registration to obtain several frame registration images I' after registration is completed i Then the current frame image I e Respectively with each registration image I' i Perform a frame difference operation to obtain several frame difference images, that is, subtract the registration image from the current frame image to obtain the corresponding frame difference image.

[0052] Furthermore, the above-mentioned process of performing frame difference operations on the current frame image and each of the registration images to obtain a plurality of frame difference images also includes: determining a target area where the frame difference result between the current frame image and the registration image in the frame difference operation is a negative value, and setting the grayscale value of the target area to zero. It should be pointed out that infrared images measure the brightness of objects and are therefore grayscale images. The heat of a target is usually high, and the grayscale value in the image belongs to a larger category. The difference between the target and the background near the target is generally positive, and the subtracted side that is negative is the background. Therefore, the present application needs to determine the target area where the frame difference result is a negative value, and truncate the grayscale value of this area to 0, that is, set it to 0.

[0053] Step S13: Superimposing a plurality of the frame difference images in a sequential order to obtain a differential image of the current frame image, and inputting the differential image into a pre-built semantic segmentation network to output a small target detection result.

[0054] In this embodiment, a plurality of frame difference images are superimposed in a sequential order to obtain the current frame image I e From the above content, it can be seen that the present application adopts a multi-frame difference method based on multi-frame accumulation and designs registration in the frame difference algorithm, which can improve the signal-to-noise ratio of background suppression under various complex backgrounds, making small target detection more accurate, and the multi-frame difference method can adapt to complex environments and is more robust than inter-frame difference. In general, the above steps are to perform background suppression on the current frame image. The robustness of this multi-frame difference is better than the existing frame difference time domain filtering, and there is no need to manually adjust the filter parameters during the background suppression and registration process.

[0055] Finally, the difference image is fed into a pre-built semantic segmentation network for small object segmentation, which then outputs the small object detection results. The use of multi-frame difference preprocessing improves the signal-to-noise ratio (SNR) of the difference image fed into the semantic segmentation network, making its features sparser. This further speeds up the network's training, enhances the semantic segmentation's temporal filtering capabilities, and improves its adaptability to diverse scenarios.

[0056] It can be seen that the present application determines the current frame image to be detected in the infrared sequence image, and filters out a number of image frames that meet the preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set; based on the current frame image, all image frames in the image frame set are aligned to obtain a number of frame alignment maps, and the current frame image is subjected to frame difference operation with each of the alignment maps to obtain a number of frame difference images; several frame difference images are superimposed in sequence order to obtain a differential image of the current frame image, and the differential image is input into a pre-constructed semantic segmentation network to output a small target detection result. It can be seen that after determining the current frame image to be detected in the infrared sequence image, the present application first needs to screen out the image frames that meet the preset frame interval with the current frame image to obtain an image frame set, and then perform registration processing on all the image frames in the image frame set based on the current frame image to obtain several frame registration images, and then perform frame difference operations on the current frame image and each registration image respectively to obtain several frame difference images, and superimpose several frame difference images in sequence order to obtain a differential image of the current frame image. That is, the present application uses a multi-frame difference method based on multi-frame accumulation and designs registration in the frame difference algorithm, which can improve the signal-to-noise ratio of background suppression under various complex backgrounds, perform background suppression on the current frame image, make small target detection more accurate, and the multi-frame difference method can adapt to complex environments and is more robust than inter-frame difference. Finally, the difference image is fed into a pre-built semantic segmentation network to output small target detection results. Due to the use of multi-frame differencing as preprocessing, the signal-to-noise ratio of the difference image input to the semantic segmentation network is improved, and its features are sparse, which further increases the network's training speed, improves the semantic segmentation's temporal filtering capabilities, and enhances its adaptability to different scenarios. As a result, the above scheme improves the overall detection performance and accuracy of small target detection in infrared image sequences.

[0057] See also Figure 2 As shown, the embodiment of the present application discloses a specific infrared small target intelligent detection method based on multi-frame difference. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0058] Step S21: determining a current frame image to be detected in the infrared sequence image, and selecting a number of image frames that meet a preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set.

[0059] Step S22: using the current frame image as a reference image for registration operation, and registering all image frames in the image frame set based on the reference image using a scale-invariant feature transformation algorithm to obtain a plurality of frame registration images.

[0060] In this embodiment, when performing the registration operation, the current frame image is used as the reference image for the registration operation, and all image frames in the image frame set are registered based on the reference image using a scale-invariant feature transformation algorithm to obtain a plurality of frame registration maps. In other words, the present application registers the image frames based on the current frame as the standard. It is understandable that due to the influence of factors such as camera shake when capturing sequential images, the background between adjacent frames of the measured data may be rotated, displaced, or stretched, so it is necessary to register multiple frames of image I. i Take the current frame image I e Registration is performed based on the standard. Registration is also called inter-frame alignment, and the principle is to affine transform one image to align it with another image, so that the dynamic background becomes stationary again, and the extracted moving object is the target. In this step, the registration is performed using scale-invariant feature transform (SIFT). The principle of SIFT is to find feature points in space and extract the position, scale and rotation invariants of the feature points. It can be divided into four steps: scale space extreme value detection, key point positioning, direction determination, and feature point description. The SIFT method only needs to extract more than 3 invariant feature points to calculate the position and orientation of the object.

[0061] Step S23: performing a frame difference operation on the current frame image and each of the registration images to obtain a plurality of frame difference images.

[0062] Step S24: performing a bitwise AND operation on the pixel values ​​of several frame difference images in a sequential order to obtain a differential image of the current frame image, and inputting the differential image into a pre-built semantic segmentation network to output a small target detection result.

[0063] In this embodiment, when multiple frame difference images are superimposed, a bitwise AND operation is performed on the pixel values ​​of the frame difference images. It is understandable that the pixel data range of a grayscale image is an integer between 0 and 255, and the result of superimposing multiple pixels may be larger than the maximum representable pixel. To prevent the pixel size of the superimposed images from exceeding 255, this application does not directly add the corresponding positions of the image pixels, but instead performs a bitwise AND (&) operation on each image pixel value. For example, when more than three image frames are superimposed, the result is counted as 1 only when all the inputs of & are simultaneously 1.

[0064] For more specific processing procedures of the above steps S21 and S23, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0065] As can be seen, the embodiments of this application take into account factors such as camera shake when capturing sequential images, as well as the possibility of background rotation, displacement, and stretching between adjacent frames. Therefore, multiple frames need to be registered based on the current frame image. This process utilizes a scale-invariant feature transformation method. Furthermore, to prevent the pixel size of the superimposed images from exceeding 255, when superimposing multiple frame difference images, a bitwise AND operation is performed on the pixel values ​​of the frame difference images.

[0066] See also Figure 3 and Figure 4 As shown, the embodiment of the present application discloses a specific infrared small target intelligent detection method based on multi-frame difference. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0067] Step S31: determining a current frame image to be detected in the infrared sequence image, and selecting a number of image frames that meet a preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set.

[0068] In this embodiment, before determining the current frame image to be detected in the infrared sequence image, the above-mentioned process further includes: using a preset annotation tool to annotate all image frames in the infrared sequence image, and determining the bounding box of the target in each image frame based on the annotation results; masking the area outside the bounding box to obtain a mask image corresponding to each image frame. That is, before determining the current frame image, all image frames in the infrared sequence image have been annotated in advance to obtain a mask image for each image frame. Specifically, Darklabel is used as an annotation tool to manually segment the boundary of the target, and mask these areas outside the boundary to obtain the final mask image, wherein DarkLabel is a lightweight video annotation software.

[0069] Step S32: registering all image frames in the image frame set based on the current frame image to obtain a plurality of frame registration images, and performing a frame difference operation on the current frame image and each of the registration images to obtain a plurality of frame difference images.

[0070] Step S33: Superimpose several frame difference images in a sequential order to obtain a differential image of the current frame image, and determine a semantic segmentation network pre-constructed based on a downsampling model, an upsampling model, a spatial attention gate model and a loss compensation model.

[0071] In this embodiment, the preset semantic segmentation network is obtained by optimizing the existing Unet network. The Unet network includes a downsampling encoder composed of multiple convolutional layers and pooling layers, an upsampling decoder composed of multiple deconvolutional layers, and a skip-layer connection connecting the encoder and decoder with the same resolution layer feature map. When Unet performs feature fusion in the skip-layer connection, the features are superimposed in the channel dimension, reducing the resolution loss of upsampling. Figure 4 As shown in , the semantic segmentation network in this embodiment includes a downsampling model, an upsampling model, a spatial attention gate model, and a loss compensation model. Specifically, the downsampling model is a downsampling encoder composed of several convolutional layers and pooling layers; the upsampling model is composed of several deconvolutional layers; the spatial attention gate model serves as a jump connection between the downsampling model and the upsampling model, and has several jump layers. The loss compensation model combines the output of the upsampling model with the mask image of the current frame and finally outputs the small target detection result.

[0072] Step S34: using the downsampling model to perform downsampling processing on the input differential image for a preset number of times to obtain a corresponding downsampled image.

[0073] In this embodiment, a downsampling model is used to perform downsampling processing on the input differential image for a preset number of times to obtain a corresponding downsampled image.

[0074] Step S35: Using the spatial attention gate model and the upsampling model to process the downsampled image to obtain a corresponding image processing result.

[0075] In this embodiment, the spatial attention gate model and the upsampling model are used to process the downsampled image to obtain the corresponding image processing result. In a specific embodiment, the above process of using the spatial attention gate model and the upsampling model to process the downsampled image to obtain the corresponding image processing result also includes: determining the corresponding output upsampled image in the upsampling model based on the feature dimension of the currently processed downsampled image, and using the corresponding convolution kernel to convolve the downsampled image and the upsampled image respectively to obtain the corresponding first convolution result and second convolution result; linearly adding the first convolution result and the second convolution result, and processing the addition result using the ReLU activation function to obtain a first activation result; convolving the first activation result The third convolution result is obtained by multiplication, and the third convolution result is processed by the sigmoid activation function to obtain the second activation result; the second activation result is upsampled to obtain the attention weight coefficient, and the downsampled image is weighted by the attention weight coefficient to obtain the attention weighted result; the attention weighted result and the upsampled result are upsampled to obtain the updated upsampled image and output, and the step of determining the upsampled image output in the upsampling model based on the feature dimension of the currently processed downsampled image is jumped again until all downsampled images are processed to obtain the corresponding image processing result. This application solves the attention allocation problem in feature fusion by realizing upsampling and downsampling association in the semantic segmentation network and adding attention optimization. After adding the attention gate, the network's attention weight in the target area increases, the training time is about 70% of the original network, and it makes the network more accurate.

[0076] The specific operation process of the spatial attention gate model is as follows: assuming X l is the input feature map of one of the skip-layer connections in the spatial attention gate model, is the output of the pooling layer in the same layer as one of the deconvolution layers in the downsampling model, and g is the output of the previous deconvolution layer in the upsampling model. It is the skip-layer connection input feature map processed by the spatial attention gate model.

[0077] The specific steps are:

[0078] 1. X l Downsampling makes X l Same size as g;

[0079] 2. X l and g are convolved with kernel 1*1*number of convolutional layer channels to obtain and Among them, the dimensions of the two convolution kernels are the dimensions of the corresponding deconvolution layer, representing the X after the 1*1 convolution kernel dimension changes. land g;

[0080] 3. Yes and Do linear addition and then activate with ReLU function;

[0081] 4. Perform 1*1*1 convolution on the output of step 3 and then activate it with the sigmoid function;

[0082] 5. Upsample the output of step 4 to obtain X l The attention weight α, α multiplied by X l Get attention-weighted input

[0083] by Figure 5 As an example, the above process is illustrated. In this embodiment, the input is 15-dimensional in total, and five differential images are spliced, each with 3 dimensions. Since the grayscale image is processed, the 3 dimensions are the same; SA in the figure is self-attention. Assume that the downsampling model has three layers of convolutional layers and pooling layers, which can downsample the input 15-dimensional differential image three times to obtain feature maps of three scales of feature dimensions: 64, 128, and 256, respectively. The upsampling model has three deconvolution layers, which can upsample the feature dimension 64 to 15, 128 to 64, and 256 to 128, which are recorded as

[0084] First, As As g, the input spatial attention gate model goes through the above five steps and finally outputs the attention-weighted The dimension is 128, and the dimension is 128 and The upsampling of the deconvolution becomes the updated

[0085] Then, As As g, the input spatial attention gate model goes through the above five steps and finally outputs the attention-weighted The dimension is 64, and the dimension is 64 and The upsampling of the deconvolution becomes the updated

[0086] Finally, As g is upsampled to That is to say, the final image processing result is obtained.

[0087] The attention mechanism used in the examples of this application works as follows: It is modeled after the human visual attention mechanism, drawing on the human body's ability to focus on key areas of information through scanning. Because the attention mechanism is simple and generally does not increase model complexity, it is widely used in the field of computer vision.

[0088] The calculation formula of attention is as follows:

[0089]

[0090] Q is an element in the target, S is the input sequence, L s is the length of S, K is an element of S, V is the value of the corresponding element, R(Q, K) represents the relevance or similarity of the input element to the output element, and the weight of the input affects the degree of focus of attention. A(Q, S) represents the amount of attention the input sequence pays to the output element.

[0091] Through the attention gate, the network focuses on inputs that are highly correlated with the target output. Inputs with high correlation have large R and high weight on Q. If the value of R is in the range [0,1], it is considered soft attention; if the value of R is 0 or 1, it is considered hard attention. Soft attention is used in classification, segmentation, detection, generative models, video processing, etc.; hard attention is often used for image cropping.

[0092] Step S36: using the loss compensation model to process the image processing result and the mask image of the current frame image to obtain a small target detection result.

[0093] In this embodiment, the image processing result and the mask image of the current frame image are processed using a loss compensation model, and the small target detection result is finally output. In a specific embodiment, the above-mentioned process of using the loss compensation model to process the image processing result and the mask image of the current frame image to obtain the small target detection result includes: processing the image processing result using a preset activation function to obtain a target output result; obtaining the mask image of the current frame image, and using the IoU method to determine the IoU value between the mask image and the target output result, and constructing a loss function based on the IoU value; using the loss function to iteratively train the semantic segmentation network until the preset number of training times is met, and then using the current target output result as the small target detection result. That is, this application first uses a preset activation function to process the image processing result to obtain the target output result. The preset activation function can be specifically a sigmiod function, and then calculates the IoU value between the mask image and the output result, and uses 1-IoU as the loss function, so as to use the loss function to iteratively train the semantic segmentation network until the preset number of training times is met, and the current target output result is used as the small target detection result.

[0094] For a more specific processing procedure of the above step S32, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described again here.

[0095] It can be seen that the semantic segmentation network in the embodiment of the present application is obtained by optimizing the existing Unet network. Improvements are made to the feature extraction and feature fusion parts of the Unet network. A fully convolutional network based on spatial attention is proposed, and the multi-frame difference method is used for preprocessing to further improve the training speed of the network and improve the adaptability of the network. The semantic segmentation network specifically includes a downsampling model, an upsampling model, a spatial attention gate model, and a loss compensation model. Moreover, by realizing the association between upsampling and downsampling in the semantic segmentation network and adding attention optimization, the attention allocation problem in feature fusion is solved. After adding the attention gate, the network's attention weight in the target area increases, the training time is about 70% of the original network, and it makes the network more accurate.

[0096] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0097] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0099] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0100] The above is a detailed introduction to the infrared small target intelligent detection method based on multi-frame difference provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for intelligent detection of infrared small targets based on multi-frame difference, characterized in that: include: Determining a current frame image to be detected in the infrared sequence image, and selecting a number of image frames that meet a preset frame interval with the current frame image from the infrared sequence image to obtain an image frame set; registering all image frames in the image frame set based on the current frame image to obtain a plurality of frame registration images, and performing a frame difference operation between the current frame image and each of the registration images to obtain a plurality of frame difference images; Superimposing a plurality of the frame difference images in a sequential order to obtain a differential image of the current frame image, and inputting the differential image into a pre-built semantic segmentation network to output a small target detection result; Wherein, the method further includes: Preliminarily constructing the semantic segmentation network based on a downsampling model, an upsampling model, a spatial attention gate model, and a loss compensation model; The step of inputting the difference image into a pre-built semantic segmentation network to output a small object detection result includes: Performing downsampling processing on the input differential image a preset number of times using the downsampling model to obtain a corresponding downsampled image; Processing the downsampled image using the spatial attention gate model and the upsampling model to obtain a corresponding image processing result; Processing the image processing result using a preset activation function to obtain a target output result; Obtaining a mask image of the current frame image, determining an IoU value between the mask image and the target output result using an IoU method, and constructing a loss function based on the IoU value; The semantic segmentation network is iteratively trained using the loss function until a preset number of training times is met, and the current target output result is used as the small target detection result.

2. The infrared small target intelligent detection method based on multi-frame difference according to claim 1 is characterized in that: The registering all image frames in the image frame set based on the current frame image to obtain a plurality of frame registration maps includes: Using the current frame image as a reference image for a registration operation; All image frames in the image frame set are registered based on the reference image using a scale-invariant feature transformation algorithm to obtain a plurality of frame registration images.

3. The infrared small target intelligent detection method based on multi-frame difference according to claim 1 is characterized in that: The process of performing a frame difference operation on the current frame image and each of the registration images to obtain a plurality of frame difference images further includes: A target area in which a frame difference result between the current frame image and the registration image in a frame difference operation is a negative value is determined, and a grayscale value of the target area is set to zero.

4. The infrared small target intelligent detection method based on multi-frame difference according to claim 1 is characterized in that: The step of superimposing a plurality of the frame difference images in a sequential order to obtain a differential image of the current frame image includes: A bitwise AND operation is performed on the pixel values ​​of a plurality of the frame difference images in a sequential order to obtain a differential image of the current frame image.

5. The infrared small target intelligent detection method based on multi-frame difference according to claim 1 is characterized in that: Before determining the current frame image to be detected in the infrared sequence image, the method further includes: Using a preset annotation tool to annotate all image frames in the infrared sequence image, and determining the bounding box of the target in each image frame based on the annotation results; The area outside the boundary box is masked to obtain a mask image corresponding to each image frame.

6. The infrared small target intelligent detection method based on multi-frame difference according to claim 1 is characterized in that: The process of processing the downsampled image using the spatial attention gate model and the upsampling model to obtain a corresponding image processing result further includes: Determining an upsampled image corresponding to the output of the upsampled model based on the feature dimension of the downsampled image currently being processed, and convolving the downsampled image and the upsampled image with corresponding convolution kernels to obtain corresponding first and second convolution results; Linearly adding the first convolution result and the second convolution result, and processing the addition result using a ReLU activation function to obtain a first activation result; Convolving the first activation result to obtain a third convolution result, and processing the third convolution result using a sigmoid activation function to obtain a second activation result; Performing an upsampling process on the second activation result to obtain an attention weight coefficient, and weighting the downsampled image using the attention weight coefficient to obtain an attention weighted result; The attention weighted result and the upsampling result are upsampled to obtain an updated upsampled image and output it, and the process jumps back to the step of determining the upsampled image output in the upsampling model based on the feature dimensions of the currently processed downsampled image, until all downsampled images are processed to obtain the corresponding image processing results.

Images