Infrared image weak and small moving target detection method based on track discrimination network model

Abstract

The invention discloses an infrared image weak and small moving target detection method based on a track discrimination network model, and the method comprises the steps: processing an infrared image sequence, and obtaining a binary image; extracting positions and sizes of candidate targets; inputting positions and sizes of candidate targets in the same candidate track into the motion feature analysis model to obtain a motion feature map; performing screenshot in the original infrared image based on the positions and the sizes of the candidate targets to obtain a spatial domain image of each candidate target; inputting the spatial domain images of the candidate targets in the same candidate track into the time domain appearance feature analysis model to obtain a time domain appearance feature map; and splicing the motion feature map and the time-domain appearance feature map of the same candidate track, and inputting the spliced motion feature map and time-domain appearance feature map into a classification module to obtain the authenticity of the candidate track, thereby completing the detection of a real target. The method is applied to the field of target detection, the number of misinformation targets can be remarkably reduced, meanwhile, real targets cannot be remarkably lost, and the method has the advantages of being good in robustness, high in calculation efficiency and the like.

Description

Technical Field

[0001] The present invention relates to the field of object detection in deep learning and computer vision technologies, and specifically to an infrared image small and weak moving object detection method based on a trajectory discrimination network model. Background Art

[0002] Infrared object detection methods can achieve all-weather detection by utilizing the infrared radiation characteristics and passive imaging characteristics of objects. This technology has important application values in fields such as night monitoring and post-disaster rescue. However, when the imaging distance is far, this detection method will face the following challenges: (1) Low signal-to-noise ratio: The atmospheric scattering effect will cause the similarity between the object and the surrounding background to be very high; (2) Scarce object information: Compared with color images, infrared images have only a single channel and lack color information; (3) Small object size: Due to the far imaging distance, the object size is small, and it is difficult to extract texture information; (4) Complex background environment: The shooting scene often contains a large number of elements such as clouds, trees, buildings, etc., and there are many background interferences. Therefore, researching efficient and reliable infrared small object detection algorithms has many challenges.

[0003] Currently, infrared small object detection algorithms are mainly divided into single-frame object detection and multi-frame object detection. The main difference between the two lies in whether the time information of the object is utilized. Single-frame object detection realizes detection by extracting and analyzing the spatial domain features of the object and the background. Traditional algorithms usually utilize local contrast or image block modeling. Mainstream deep learning algorithms achieve efficient information interaction in the spatial domain through multi-scale feature fusion in the spatial domain or introducing a Transformer network. However, relying only on the scarce appearance information of the object will limit the further improvement of the algorithm performance. Multi-frame methods improve the performance of small object detection by associating multiple frames of images in the time series and utilizing the time features of the object. Existing multi-frame algorithms mainly utilize time motion information and time appearance information. Time motion information refers to the trajectory information formed by moving objects and background interferences in multiple frames of images, while time appearance information refers to the appearance changes of a certain image block in several consecutive frames.

[0004] Existing trajectory algorithms have a relatively single processing method for time-domain motion information. Most rely on prior motion hypothesis modeling, and then perform trajectory discrimination according to the model to achieve object detection. These models usually assume that the object is moving at a constant speed or has smooth trajectory characteristics, and it is difficult to process objects with large maneuverability, easily resulting in the loss of real objects. In addition, parameters need to be manually adjusted for different motion types of objects, and the adaptability is poor. Summary of the Invention

[0005] In view of the above deficiencies in the prior art, the present invention provides an infrared image small and weak moving target detection method based on a trajectory discrimination network model, which can effectively overcome the defects such as insufficient processing of time-domain information and weak algorithm robustness in the existing target detection methods.

[0006] To achieve the above object, the present invention provides an infrared image small and weak moving target detection method based on a trajectory discrimination network model, including the following steps: Step 1, obtain an infrared image sequence of the target, and process the infrared image sequence to obtain a binary image of each frame of the infrared image; Step 2, extract the position information and size information of each candidate target in each of the binary images; Step 3, arrange and combine the candidate targets in multiple consecutive frames of images to obtain a number of candidate trajectories; Step 4, input the position information and size information of the candidate targets in the same candidate trajectory in each frame of the image into the motion feature analysis model of the trajectory discrimination network model to obtain the motion feature map of the candidate trajectory; Step 5, take screenshots in the original infrared image based on the position information and size information of each candidate target to obtain the spatial domain images of each candidate target at each frame time; Step 6, input the spatial domain images of the candidate targets in the same candidate trajectory into the time-domain appearance feature analysis model of the trajectory discrimination network model to obtain the time-domain appearance feature map of the candidate trajectory; Step 7, splice the motion feature map and the time-domain appearance feature map of the same candidate trajectory, and then input them into the classification module of the trajectory discrimination network model to obtain the authenticity of the candidate trajectory and complete the detection of real targets.

[0007] In one embodiment, step 2 specifically includes: Use the coordinates of the centroid of the set of candidate target pixel points in the binary image as the position information of the candidate target; Obtain the size information of the candidate target according to the length and width values of the smallest target box that can contain the set of candidate target pixel points in the binary image.

[0008] In one embodiment, in step 4, the motion feature analysis model is an LSTM model, and the process of extracting the motion feature map is specifically as follows: Register the position information and size information of the candidate targets in the same candidate trajectory in each frame of the image to obtain the registered position and size information , where is the registered position information of the candidate target in the t + k -th frame of the image, is the registered position information of the candidate target in the t + kThe registered width dimension information in the frame image, is the candidate target in the t + k registered height dimension information in the frame image, and the subscript indicates that the candidate trajectory is composed of continuous N + 1 candidate targets; Input the registered position dimension information together into the LSTM model to obtain the motion feature map of the candidate trajectory.

[0009] In one embodiment, in step 5, the process of cropping the original infrared image based on the position information and dimension information of each candidate target is specifically as follows: Obtain the height dimension H and width dimension W of the candidate target, take the larger value L in the target dimensions (W, H), and take the coordinates of the position information of the candidate target as the center, and crop a square with side length L in the original infrared image, and ensure that the cropped image block can completely contain the candidate target pixels, that is, obtain the spatial domain image of the candidate target in the current frame of the original infrared image.

[0010] In one embodiment, in step 6, the extraction process of the temporal appearance feature map is as follows: Obtain the first-frame spatial domain image I of the candidate target in the same candidate trajectory; Use two-dimensional convolution in two layers to extract the target spatial domain feature map of image I respectively, and after the second layer of two-dimensional convolution, input the generated feature map into the coordinate attention module, and use two pooling kernels with spatial ranges (H, 1) or (1, W) for each channel respectively, and encode along the horizontal and vertical coordinates to generate the aggregated feature maps in the horizontal and vertical directions; Concatenate the aggregated feature maps in the horizontal and vertical directions and send them to the shared 1×1 convolution transformation function to generate the intermediate feature map; After splitting the intermediate feature map into two separate tensors along the spatial dimension, use the two tensors as attention weights respectively to obtain the output features of the coordinate attention mechanism; After stacking the output features corresponding to all the spatial domain images of the candidate targets in the same candidate trajectory, perform two three-dimensional convolution operations and then unfold them, that is, obtain the temporal appearance feature map of the candidate trajectory.

[0011] In one embodiment, step 7 specifically includes: Concatenate the motion feature map and the temporal appearance feature map of the same candidate trajectory into a multi-layer perceptron with three linear layers, and then obtain the probability of the candidate trajectory through the Sigmoid function processing; Determine whether the probability of the candidate trajectory exceeds the threshold: If so, determine that the candidate trajectory is a real trajectory, and output the target in the real trajectory as the real target; Otherwise, determine the candidate trajectory as noise.

[0012] Compared with the prior art, the present invention has the following beneficial technical effects: 1. The present invention utilizes the similarity of the appearance features of a moving target in consecutive multiple frames of images, and this appearance similarity does not exist in background interference, thereby being able to handle the problem of irregular trajectories. Based on this, by combining the motion features and appearance features of the moving target to judge the authenticity of the target trajectory, it can effectively overcome the defects such as insufficient processing of time-domain information and weak algorithm robustness in existing target detection methods; 2. In the preferred embodiment of the present invention, by introducing a coordinate attention module, the performance of the trajectory discrimination network model is effectively improved, and further the accuracy of detecting small and weak moving targets in infrared images is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the structures shown in these drawings.

[0014] Figure 1 It is a flowchart of a method for detecting small and weak moving targets in infrared images based on a trajectory discrimination network model in an embodiment of the present invention; Figure 2 It is an overall framework diagram of a trajectory discrimination network model in an embodiment of the present invention; Figure 3 It is a schematic diagram of a motion feature analysis model in an embodiment of the present invention; Figure 4 It is a schematic diagram of a time-domain appearance feature analysis model in an embodiment of the present invention; Figure 5 It is a schematic diagram of a coordinate attention module in an embodiment of the present invention.

[0015] The realization, functional characteristics and advantages of the object of the present invention will be further described in conjunction with the embodiments and with reference to the drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0017] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0018] This embodiment discloses an infrared image small and weak moving target detection method based on a trajectory discrimination network model. The trajectory discrimination network model mainly consists of a motion feature analysis model, a temporal appearance feature analysis model, a feature splicing module, and a classification module. Among them, the motion feature analysis model is used to extract the motion feature map of the target, the temporal appearance feature analysis model is used to extract the temporal appearance feature map of the target, the feature splicing module is used to splice the motion feature map and the temporal appearance feature map, and the classification module is used to predict the authenticity of the candidate target.

[0019] Reference Figure 1 , the infrared image small and weak moving target detection method based on the trajectory discrimination network model in this embodiment specifically includes the following steps: Step 1, obtain the infrared image sequence of the target, and perform binary processing on the infrared image sequence by using a single-frame deep learning network or an adaptive threshold algorithm, etc., to obtain the binary image of each frame of the infrared image; Step 2, extract the position information and size information of each candidate target in each binary image. The specific implementation process is as follows: Take the coordinates of the centroid of the set of candidate target pixel points in the binary image as the position information of the candidate target; Obtain the size information of the candidate target according to the length and width values of the smallest target box that can contain the set of candidate target pixel points in the binary image; Step 3, arrange and combine the candidate targets in multiple consecutive frames of images to obtain several candidate trajectories. Among them, arranging and combining the candidate targets in multiple consecutive frames of images means traversing all the trajectory combination methods. For example, in three consecutive frames of images, if there are 2 candidate targets (Target 1, Target 2) in the first frame of image, 3 candidate targets (Target 3, Target 4, Target 5) in the second frame of image, and 1 candidate target (Target 6) in the third frame of image, a total of 6 candidate trajectories can be obtained through arrangement and combination, which are: Candidate trajectory 1: Target 1 - Target 3 - Target 6; Candidate trajectory 2: Target 1 - Target 4 - Target 6; Candidate trajectory 3: Target 1 - Target 5 - Target 6; Candidate trajectory 4: Target 2 - Target 3 - Target 6; Candidate trajectory 5: Target 2 - Target 4 - Target 6; Candidate trajectory 6: Target 2 - Target 5 - Target 6; Step 4: Input the position information and size information of candidate targets in each frame of the image in the same candidate trajectory into the motion feature analysis model of the trajectory discrimination network model to obtain the motion feature map of the candidate trajectory; Step 5: Based on the position information and size information of each candidate target, capture screenshots in the original infrared image to obtain the spatial domain images of each candidate target at each frame moment. The specific implementation process is as follows: Obtain the height size H and width size W of the candidate target, and take the larger value L of the target sizes (W, H). Then, centered on the coordinate of the position information of the candidate target, intercept a square with a side length of L in the original infrared image, and ensure that the intercepted image block can completely contain the candidate target pixels, that is, obtain the spatial domain image of the candidate target in the original infrared image of the current frame; Step 6: Input the spatial domain images of candidate targets in the same candidate trajectory into the time-domain appearance feature analysis model of the trajectory discrimination network model to obtain the time-domain appearance feature map of the candidate trajectory; Step 7: After splicing the motion feature map and the time-domain appearance feature map of the same candidate trajectory, input them into the classification module of the trajectory discrimination network model to obtain the authenticity of the candidate trajectory and complete the detection of real targets.

[0020] In this embodiment, the motion feature analysis model is an LSTM model. The LSTM model is a variant of the recurrent neural network (RNN), which solves the problems of gradient explosion and disappearance during network training by adding a forget gate, an input gate, and an output gate.

[0021] Reference Figure 2 、 Figure 3 , the specific process of extracting the motion feature map is as follows: First, use the method in the prior art "ORB: An efficient alternative to sift or surf" to register the position information and size information of candidate targets in each frame of the image in the same candidate trajectory to obtain the registered position and size information , where is the registered position information of the candidate target in the t + k -th frame image, is the registered width size information of the candidate target in the t + k -th frame image, is the registered height size information of the candidate target in the t + k -th frame image. The subscript indicates that the candidate trajectory consists of continuous N + 1 candidate targets; Input the registered position and size information Input into the LSTM model together; In the LSTM model, the forget gate is used to determine whether to discard or retain the hidden variable information, that is: (1) Among them, is the output feature of the forget gate, is the weight matrix of the forget gate, is the hidden state at the previous moment, is the bias term of the forget gate, is the non-linear activation function; In the LSTM model, the input gate is used to generate the information of the hidden variable that needs to be updated, that is: (2) Among them, is the activation value of the input gate, is the weight matrix of the input gate, is the bias term of the input gate, is the candidate memory cell, is the weight matrix of the memory cell, is the bias term of the memory cell, is the updated memory cell at the current moment, is the updated memory cell at the previous moment, and tanh is the hyperbolic tangent activation function; In the LSTM model, the output gate is used to determine the output of the model, that is: (3) Among them, is the output feature of the output gate, is the weight matrix of the output gate, is the bias term of the output gate, is the hidden state at the previous moment; Input the position information and size information in each frame of the image in the same candidate trajectory into the LSTM together, and the motion feature map corresponding to the candidate trajectory can be obtained . Since the larger the frame number span, the higher the complexity of the target trajectory, the number of output channels of the LSTM model is proportional to the frame number span. For example, if the candidate trajectory contains 4 frames of images, and the number of unique output channels for each frame of image is 12, then the finally obtained motion feature map has a size of .

[0022] Refer to Figure 2 , Figure 4 , Figure 5 , the extraction process of the temporal appearance feature map is: First, obtain the first-frame airspace image of the candidate target in the same candidate trajectory, and use bicubic interpolation to uniformly magnify the intercepted image block to the standard size of 17 pixels × 17 pixels, denoted as image I; Use two layers of two-dimensional convolution to extract the target airspace feature map of image I. Assume that for a certain airspace image in the t-th frame Then, first obtain the feature map through the two-dimensional convolution of the first layer and then obtain the feature map through the two-dimensional convolution of the second layer That is: (4) Among them, and are convolution kernels with a traditional two-dimensional convolution stride of (3, 3, 3) and a padding value of 1; After the second layer of two-dimensional convolution, input the generated feature map into the coordinate attention module (CAM). Each channel of the coordinate attention module (CAM) uses two pooling kernels with spatial ranges (H, 1) or (1, W) respectively, and encodes along the horizontal and vertical coordinates. Therefore, the output of channel c with height h in the coordinate attention module (CAM) is: (5) Among them, is the pooled output feature of channel c in the vertical direction, and the output of the pooled output features of all channels in the vertical direction is the vertical aggregation feature map , p is the pixel position in the vertical direction, is the feature vector corresponding to the p-th column of the feature map with height h.

[0023] Similarly, the output of channel c with width w in the coordinate attention module (CAM) is: (6) Among them, is the output of channel c in the horizontal direction, and the output of the pooled output features of all channels in the horizontal direction is the horizontal aggregation feature map , is the pixel position in the horizontal direction, is the feature vector corresponding to the q-th row of the feature map with width w.

[0024] The above two transformations aggregate features along two spatial directions respectively, thus generating a pair of direction-aware feature maps, which is quite different from the compression operation of generating a single feature vector in the channel attention method. These two transformations also enable the coordinate attention module in this embodiment to capture long-range dependencies along one spatial direction and retain precise position information along the other spatial direction, thereby helping the network to more accurately locate the object of interest. Specifically, the horizontal and vertical aggregation feature maps generated according to formulas (5) and (6) are concatenated together and then sent to the shared 1×1 convolutional transformation function to generate an intermediate feature map, as follows: (7) Wherein, represents the concatenation operation along the spatial dimension, is the intermediate feature map, which is used to encode the spatial information in the horizontal and vertical directions, C is the control block, r is the reduction rate for controlling the block size; The intermediate feature map is split into two separate tensors and along the spatial dimension, and then two 1×1 convolutional transformations and are respectively used to convert and into tensors with the same number of channels as the input tensor , that is: (8) Wherein, is the Sigmoid function; To reduce the model complexity, this embodiment uses an appropriate scale to reduce the number of channels of and , and then the two tensors (9) are respectively used as attention weights to obtain the output features of the coordinate attention mechanism, as follows: y Wherein, relu is the Relu activation function, is the output feature, is the residual term, ,

[0025] Finally, two layers of three-dimensional convolution are used to process the changes of the target appearance in the time dimension to achieve the interactive matching of the target features between frames. By stacking the output features y of consecutive N + 1 frames to obtain , then perform two three-dimensional convolution operations, and then obtain the representation of the target temporal appearance feature through an unfolding operation , that is: (10) Among them, represents the tensor stacking operation, is the output feature of the first three-dimensional convolution, is the output feature of the second three-dimensional convolution.

[0026] In this embodiment, after splicing the motion feature map and the temporal appearance feature map of the same candidate trajectory, the classification module of the trajectory discrimination network model is input, and the authenticity of the candidate trajectory is obtained, and the implementation process of detecting the real target is specifically as follows: First, splice the motion feature map of the same candidate trajectory with the temporal appearance feature map into a multi-layer perceptron (MLP) with three linear layers, and then obtain the probability of the candidate trajectory through the Sigmoid function processing; Determine whether the probability of the candidate trajectory exceeds the threshold: If so, determine that the candidate trajectory is a real trajectory, and output the target in the real trajectory as the real target; Otherwise, determine that the candidate trajectory is noise.

[0027] The above are only the preferred embodiments of the present invention, and do not limit the protection scope of the present invention. Any equivalent structural transformation made under the inventive concept of the present invention, or direct / indirect application in other related technical fields is included in the protection scope of the present invention.

Claims

1. A method for detecting small moving targets in infrared images based on a trajectory discrimination network model, characterized in that: The steps include: Step 1, obtaining an infrared image sequence of the target, and processing the infrared image sequence to obtain a binary image of each frame of the infrared image; Step 2, extracting the position information and size information of each candidate target in each of the binary images; Step 3, arranging and combining the candidate targets in the continuous multi-frame images to obtain several candidate trajectories; Step 4, inputting the position information and size information of the candidate target in each frame image in the same candidate trajectory into the motion feature analysis model of the trajectory discrimination network model to obtain a motion feature map of the candidate trajectory; Step 5: based on the position information and size information of each candidate target, a screenshot is taken in the original infrared image to obtain a spatial domain image of each candidate target at each frame time; Step 6, inputting the spatial domain image of the candidate target in the same candidate trajectory into the temporal domain appearance feature analysis model of the trajectory discrimination network model to obtain the temporal domain appearance feature map of the candidate trajectory; In step 7, the motion feature map and the time-domain appearance feature map of the same candidate trajectory are concatenated and input into the classification module of the trajectory discrimination network model to obtain the authenticity of the candidate trajectory and complete the detection of the real target.

2. The method for detecting small moving targets in infrared images based on a trajectory discrimination network model according to claim 1 is characterized in that: Step 2 specifically includes: The coordinates of the centroid of the candidate target pixel set in the binary image in the image are used as the position information of the candidate target; The size information of the candidate target is obtained according to the length and width of the minimum target box that can contain the set of candidate target pixels in the binary image.

3. The method for detecting small moving targets in infrared images based on a trajectory discrimination network model according to claim 1 is characterized in that: In step 4, the motion feature analysis model is an LSTM model, and the extraction process of the motion feature map is specifically as follows: The position and size information of the candidate targets in the same candidate trajectory in each frame image are aligned to obtain the aligned position and size information. ,in, For the candidate target The position information after registration in the frame image, For the candidate target The width dimension information after registration in the frame image, For the candidate target The height size information after registration in the frame image, subscript Indicates that the candidate trajectory consists of N+1 consecutive candidate targets; The position and size information after registration Input them into the LSTM model together to obtain the motion feature map of the candidate trajectory.

4. The infrared image weak moving target detection method based on trajectory discrimination network model according to claim 1, 2 or 3, characterized in that: In step 5, the screenshot is taken in the original infrared image based on the position information and size information of each candidate target, specifically: Get the height dimension H and width dimension W of the candidate target, and take the larger value L of the target size (W, H), and cut a square with a side length of L in the original infrared image with the position information coordinates of the candidate target as the center, and ensure that the cut image block can completely contain the candidate target pixels, that is, get the spatial domain image of the candidate target in the original infrared image of the current frame.

5. The method for detecting small moving targets in infrared images based on a trajectory discrimination network model according to claim 4 is characterized in that: In step 6, the extraction process of the time domain appearance feature map is: Obtain the first frame of spatial domain image I of the candidate target in the same candidate trajectory; Use two layers of two-dimensional convolution to extract the target spatial domain feature map of image I respectively, and after the second layer of two-dimensional convolution, the generated feature map is input into the coordinate attention module. Each channel uses two pooling kernels of spatial range (H, 1) or (1, W) to encode along the horizontal and vertical coordinates to generate aggregated feature maps in the horizontal and vertical directions; The aggregated feature maps in the horizontal and vertical directions are concatenated and sent to a shared 1×1 convolution transformation function to generate an intermediate feature map. After splitting the intermediate feature map into two separate tensors along the spatial dimension, the two tensors are used as attention weights to obtain the output features of the coordinate attention mechanism; After stacking the output features corresponding to all spatial domain images of the candidate targets in the same candidate trajectory, two three-dimensional convolution operations are performed and then expanded to obtain the temporal appearance feature map of the candidate trajectory.

6. The infrared image weak moving target detection method based on trajectory discrimination network model according to claim 1, 2 or 3, characterized in that: Step 7 specifically includes: The motion feature map and the temporal appearance feature map of the same candidate trajectory are concatenated into a multi-layer perceptron with three linear layers, and then the probability of the candidate trajectory is obtained by Sigmoid function processing; Determine whether the probability of a candidate trajectory exceeds the threshold: If so, the candidate trajectory is determined to be the true trajectory, and the target in the true trajectory is output as the true target; Otherwise, the candidate trajectory is determined to be noise.

Images