Infrared Image Weak Moving Target Detection Method Based on Trajectory Discrimination Network Model

Abstract

The present invention discloses a method for detecting small and weak moving targets in infrared images based on a trajectory discrimination network model, including: processing an infrared image sequence to obtain a binary image; extracting the positions and sizes of candidate targets; inputting the positions and sizes of candidate targets in the same candidate trajectory into a motion feature analysis model to obtain a motion feature map; taking screenshots of the original infrared image based on the positions and sizes of the candidate targets to obtain the spatial domain images of each candidate target; inputting the spatial domain images of candidate targets in the same candidate trajectory into a temporal appearance feature analysis model to obtain a temporal appearance feature map; splicing the motion feature map and the temporal appearance feature map of the same candidate trajectory and then inputting them into a classification module to obtain the authenticity of the candidate trajectory, thereby completing the detection of real targets. The present invention is applied to the field of target detection, which can significantly reduce the number of false alarm targets, and at the same time will not significantly lose real targets, and has the advantages of good robustness and high computational efficiency.

Description

Technical Field

[0001] The present invention relates to the field of object detection in deep learning and computer vision technologies, and specifically to an infrared image small and weak moving object detection method based on a trajectory discrimination network model. Background Art

[0002] Infrared object detection methods can achieve all-weather detection by utilizing the infrared radiation characteristics and passive imaging characteristics of objects. This technology has important application values in fields such as night monitoring and post-disaster rescue. However, when the imaging distance is far, this detection method will face the following challenges:

[0003] (1) Low signal-to-noise ratio: The atmospheric scattering effect will result in a high similarity between the object and the surrounding background;

[0004] (2) Scarce object information: Compared with color images, infrared images have only a single channel and lack color information;

[0005] (3) Small object size: Due to the far imaging distance, the object size is small, and it is difficult to extract texture information;

[0006] (4) Complex background environment: The shooting scene often contains a large number of elements such as clouds, trees, and buildings, and there are many background interferences. Therefore, researching efficient and reliable infrared small object detection algorithms has many challenges.

[0007] Currently, infrared small object detection algorithms are mainly divided into single-frame object detection and multi-frame object detection. The main difference between the two lies in whether the time information of the object is utilized. Single-frame object detection realizes detection by extracting and analyzing the spatial domain features of the object and the background. Traditional algorithms usually utilize local contrast or image patch modeling. Mainstream deep learning algorithms achieve efficient information interaction in the spatial domain through multi-scale feature fusion in the spatial domain or introducing a Transformer network. However, relying solely on the scarce appearance information of the object will limit the further improvement of the algorithm performance. Multi-frame methods improve the performance of small object detection by associating multiple frames of images in the time series and utilizing the time features of the object. Existing multi-frame algorithms mainly utilize time motion information and time appearance information. Time motion information refers to the trajectory information formed by moving objects and background interferences in multiple frames of images, while time appearance information refers to the appearance changes of a certain image patch in several consecutive frames.

[0008] Existing trajectory algorithms are relatively single in the processing method of time-domain motion information. Most rely on prior motion hypothesis modeling, and then perform trajectory discrimination according to the model to achieve object detection. These models usually assume that the object is moving at a constant speed or has smooth trajectory characteristics, and it is difficult to handle objects with large maneuverability, which easily leads to the loss of real objects. In addition, parameters need to be manually adjusted for different motion types of objects, and the adaptability is poor. Summary of the Invention

[0009] In view of the deficiencies in the above-mentioned prior art, the present invention provides an infrared image small and weak moving target detection method based on a trajectory discrimination network model, which can effectively overcome the defects such as insufficient processing of time-domain information and weak algorithm robustness in the existing target detection methods.

[0010] To achieve the above object, the present invention provides an infrared image small and weak moving target detection method based on a trajectory discrimination network model, including the following steps:

[0011] Step 1, obtain an infrared image sequence of the target, and process the infrared image sequence to obtain a binary image of each frame of the infrared image;

[0012] Step 2, extract the position information and size information of each candidate target in each of the binary images;

[0013] Step 3, arrange and combine the candidate targets in multiple consecutive frames of images to obtain a number of candidate trajectories;

[0014] Step 4, input the position information and size information of the candidate targets in each frame of the image in the same candidate trajectory into the motion feature analysis model of the trajectory discrimination network model to obtain the motion feature map of the candidate trajectory;

[0015] Step 5, based on the position information and size information of each candidate target, take screenshots in the original infrared image to obtain the spatial domain images of each candidate target at each frame moment;

[0016] Step 6, input the spatial domain images of the candidate targets in the same candidate trajectory into the time-domain appearance feature analysis model of the trajectory discrimination network model to obtain the time-domain appearance feature map of the candidate trajectory;

[0017] Step 7, after splicing the motion feature map and the time-domain appearance feature map of the same candidate trajectory, input them into the classification module of the trajectory discrimination network model to obtain the authenticity of the candidate trajectory and complete the detection of real targets.

[0018] In one embodiment, step 2 specifically includes:

[0019] Take the coordinates of the centroid of the set of candidate target pixel points in the binary image in the image as the position information of the candidate target;

[0020] Obtain the size information of the candidate target according to the length and width values of the smallest target box that can contain the set of candidate target pixel points in the binary image.

[0021] In one embodiment, in step 4, the motion feature analysis model is an LSTM model, and the process of extracting the motion feature map is specifically as follows:

[0022] Register the position information and size information of candidate targets in each frame of image for the same candidate trajectory to obtain the registered position and size information. , where is the registered position information of the candidate target in the t + k -th frame image, is the registered width size information of the candidate target in the t + k -th frame image, is the registered height size information of the candidate target in the t + k -th frame image, and the subscript indicates that the candidate trajectory consists of N + 1 consecutive candidate targets;

[0023] Input the registered position and size information into the LSTM model together to obtain the motion feature map of the candidate trajectory.

[0024] In one embodiment, in step 5, the process of cropping the original infrared image based on the position information and size information of each candidate target is as follows:

[0025] Obtain the height size H and width size W of the candidate target, take the larger value L in the target size (W, H), and use the coordinate of the position information of the candidate target as the center to crop a square with side length L in the original infrared image, and ensure that the cropped image block can completely contain the candidate target pixels, that is, obtain the spatial domain image of the candidate target in the current frame of the original infrared image.

[0026] In one embodiment, in step 6, the extraction process of the temporal domain appearance feature map is as follows:

[0027] Obtain the first-frame spatial domain image I of the candidate target in the same candidate trajectory;

[0028] Use two layers of two-dimensional convolution to extract the target spatial domain feature map of image I respectively, and after the second layer of two-dimensional convolution, input the generated feature map into the coordinate attention module, and use two pooling kernels with spatial ranges (H, 1) or (1, W) for each channel to encode along the horizontal and vertical coordinates to generate the aggregated feature maps in the horizontal and vertical directions;

[0029] Concatenate the aggregated feature maps in the horizontal and vertical directions and send them to the shared 1×1 convolution transformation function to generate the intermediate feature map;

[0030] After splitting the intermediate feature map along the spatial dimension into two separate tensors, use the two tensors as attention weights respectively to obtain the output features of the coordinate attention mechanism;

[0031] After stacking the output features corresponding to all the airspace images of the candidate targets in the same candidate trajectory, perform two three-dimensional convolution operations and then unfold them to obtain the temporal appearance feature map of the candidate trajectory.

[0032] In one embodiment, step 7 specifically includes:

[0033] Concatenate the motion feature map and the temporal appearance feature map of the same candidate trajectory into a multi-layer perceptron with three linear layers, and then obtain the probability of the candidate trajectory through the processing of the Sigmoid function;

[0034] Determine whether the probability of the candidate trajectory exceeds the threshold:

[0035] If so, determine that the candidate trajectory is a real trajectory, and output the target in the real trajectory as a real target;

[0036] Otherwise, determine that the candidate trajectory is noise.

[0037] Compared with the prior art, the present invention has the following beneficial technical effects:

[0038] 1. The present invention utilizes the similarity of the appearance features of moving targets in consecutive multiple frames of images, and this appearance similarity does not exist in background interference, so as to be able to handle the problem of irregular trajectories. Based on this, the motion features and appearance features of moving targets are combined to judge the authenticity of the target trajectory, which can effectively overcome the defects such as insufficient processing of temporal information and weak algorithm robustness in the existing target detection methods;

[0039] 2. In the preferred solution of the present invention, by introducing a coordinate attention module, the performance of the trajectory discrimination network model is effectively improved, and further the detection accuracy of small and weak moving targets in infrared images is improved. Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the structures shown in these drawings without creative efforts.

[0041] Figure 1 It is a flowchart of the method for detecting small and weak moving targets in infrared images based on the trajectory discrimination network model in the embodiments of the present invention;

[0042] Figure 2 It is the overall framework diagram of the trajectory discrimination network model in the embodiments of the present invention;

[0043] Figure 3It is the schematic diagram of the motion feature analysis model in the embodiment of the present invention;

[0044] Figure 4 It is the schematic diagram of the time-domain appearance feature analysis model in the embodiment of the present invention;

[0045] Figure 5 It is the schematic diagram of the coordinate attention module in the embodiment of the present invention.

[0046] The realization, functional features and advantages of the object of the present invention will be further described in conjunction with the embodiments with reference to the accompanying drawings. Specific embodiments

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0048] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement it. When the combination of technical solutions appears to be contradictory or unable to be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] This embodiment discloses an infrared image small and weak moving target detection method based on a trajectory discrimination network model. The trajectory discrimination network model mainly consists of a motion feature analysis model, a time-domain appearance feature analysis model, a feature splicing module and a classification module. Among them, the motion feature analysis model is used to extract the motion feature map of the target, the time-domain appearance feature analysis model is used to extract the time-domain appearance feature map of the target, the feature splicing module is used to splice the motion feature map and the time-domain appearance feature map, and the classification module is used to predict the authenticity of the candidate target.

[0050] Refer to Figure 1 , the infrared image small and weak moving target detection method based on the trajectory discrimination network model in this embodiment specifically includes the following steps:

[0051] Step 1, obtain the infrared image sequence of the target, and perform binary processing on the infrared image sequence by using a single-frame deep learning network or an adaptive threshold algorithm, etc., to obtain the binary image of each frame of the infrared image;

[0052] Step 2, extract the position information and size information of each candidate target in each binary image, and its specific implementation process is as follows:

[0053] Take the coordinates of the centroid of the set of candidate target pixel points in the binary image in the image as the position information of the candidate target;

[0054] Obtain the size information of the candidate target according to the length and width values of the smallest target box that can contain the set of candidate target pixel points in the binary image;

[0055] Step 3: Arrange and combine the candidate targets in consecutive multiple frames of images to obtain a number of candidate trajectories. Among them, arranging and combining the candidate targets in consecutive multiple frames of images means traversing all trajectory combination methods. For example, in three consecutive frames of images, if there are 2 candidate targets (Target 1, Target 2) in the first frame of image, 3 candidate targets (Target 3, Target 4, Target 5) in the second frame of image, and 1 candidate target (Target 6) in the third frame of image, a total of 6 candidate trajectories can be obtained through arrangement and combination, which are respectively:

[0056] Candidate Trajectory 1: Target 1 - Target 3 - Target 6;

[0057] Candidate Trajectory 2: Target 1 - Target 4 - Target 6;

[0058] Candidate Trajectory 3: Target 1 - Target 5 - Target 6;

[0059] Candidate Trajectory 4: Target 2 - Target 3 - Target 6;

[0060] Candidate Trajectory 5: Target 2 - Target 4 - Target 6;

[0061] Candidate Trajectory 6: Target 2 - Target 5 - Target 6;

[0062] Step 4: Input the position information and size information of the candidate targets in each frame of image in the same candidate trajectory into the motion feature analysis model of the trajectory discrimination network model to obtain the motion feature map of the candidate trajectory;

[0063] Step 5: Based on the position information and size information of each candidate target, take a screenshot in the original infrared image to obtain the spatial domain image of each candidate target at each frame moment. The specific implementation process is as follows:

[0064] Obtain the height size H and width size W of the candidate target, and take the larger value L in the target size (W, H). Then, with the coordinate of the position information of the candidate target as the center, intercept a square with a side length of L in the original infrared image, and ensure that the intercepted image block can completely contain the candidate target pixels, that is, obtain the spatial domain image of the candidate target in the original infrared image of the current frame;

[0065] Step 6: Input the spatial domain images of the candidate targets in the same candidate trajectory into the time-domain appearance feature analysis model of the trajectory discrimination network model to obtain the time-domain appearance feature map of the candidate trajectory;

[0066] Step 7: After splicing the motion feature map and the temporal appearance feature map of the same candidate trajectory, input them into the classification module of the trajectory discrimination network model to obtain the authenticity of the candidate trajectory and complete the detection of real targets.

[0067] In this embodiment, the motion feature analysis model is an LSTM model. The LSTM model is a variant of the recurrent neural network (RNN), which solves the problems of gradient explosion and disappearance during network training by adding a forget gate, an input gate, and an output gate.

[0068] Reference Figure 2 , Figure 3 , the process of extracting the motion feature map is specifically as follows:

[0069] First, use the method in the prior art "ORB: An efficient alternative to sift or surf" to register the position information and size information of the candidate target in each frame of the same candidate trajectory to obtain the registered position and size information , where is the registered position information of the candidate target in the t + k -th frame image, is the registered width size information of the candidate target in the t + k -th frame image, is the registered height size information of the candidate target in the t + k -th frame image, and the subscript indicates that the candidate trajectory consists of continuous N + 1 candidate targets;

[0070] Input the registered position and size information into the LSTM model together;

[0071] In the LSTM model, the forget gate is used to decide whether to discard or retain the hidden variable information, that is:

[0072] (1)

[0073] where is the output feature of the forget gate, is the weight matrix of the forget gate, is the hidden state of the previous moment, is the bias term of the forget gate, is the non-linear activation function;

[0074] In the LSTM model, the input gate is used to generate the information of the hidden variable to be updated, that is:

[0075] (2)

[0076] Among them, is the input gate activation value, is the input gate weight matrix, is the input gate bias term, is the candidate memory cell, is the weight matrix of the memory cell, is the bias term of the memory cell, is the updated memory cell at the current moment, is the updated memory cell at the previous moment, and tanh is the hyperbolic tangent activation function;

[0077] In the LSTM model, the output gate is used to determine the output of the model, that is:

[0078] (3)

[0079] Among them, is the output feature of the output gate, is the output gate weight matrix, is the output gate bias term, is the hidden state at the previous moment;

[0080] Input the position information and size information in each frame of the image in the same candidate trajectory into the LSTM together, and the motion feature map corresponding to the candidate trajectory can be obtained . Since the greater the frame number span, the higher the complexity of the target trajectory, the number of output channels of the LSTM model is proportional to the frame number span. For example, if the candidate trajectory contains 4 frames of images, and the number of unique output channels for each frame of the image is 12, then the finally obtained motion feature map The size is .

[0081] Refer to Figure 2 , Figure 4 , Figure 5 , the extraction process of the temporal appearance feature map is as follows:

[0082] First, obtain the first-frame spatial image of the candidate target in the same candidate trajectory, and use the bicubic interpolation method to uniformly enlarge the intercepted image block to the standard size of 17 pixels × 17 pixels, denoted as image I;

[0083] Use two layers of two-dimensional convolution to extract the target spatial feature map of image I. Assume that the t-th frame of a certain spatial image Then first obtain the feature map through the two-dimensional convolution of the first layer, and then obtain the feature map through the two-dimensional convolution of the second layer, that is: ​

[0084] (4)

[0085] Among them, and are traditional two-dimensional convolution kernels with a stride of (3, 3, 3) and a padding value of 1;

[0086] After the second layer of two-dimensional convolution, the generated feature map is input into the Coordinate Attention Module (CAM). Each channel of the Coordinate Attention Module (CAM) uses two pooling kernels with spatial ranges (H, 1) or (1, W) respectively, and encodes along the horizontal and vertical coordinates. Therefore, the output of channel c with height h in the Coordinate Attention Module (CAM) is:

[0087] (5)

[0088] Among them, is the pooled output feature of channel c in the vertical direction, and the output of the pooled output features of all channels in the vertical direction is the aggregated feature map in the vertical direction , p is the pixel position in the vertical direction, is the feature vector corresponding to the p-th column of the feature map with height h.

[0089] Similarly, the output of channel c with width w in the Coordinate Attention Module (CAM) is:

[0090] (6)

[0091] Among them, is the output of channel c in the horizontal direction, and the output of the pooled output features of all channels in the horizontal direction is the aggregated feature map in the horizontal direction , is the pixel position in the horizontal direction, is the feature vector corresponding to the q-th row of the feature map with width w.

[0092] The above two transformations aggregate features along two spatial directions respectively, thus generating a pair of direction-aware feature maps, which is quite different from the compression operation of generating a single feature vector in the channel attention method. These two transformations also enable the Coordinate Attention Module in this embodiment to capture long-range dependencies along one spatial direction and retain precise position information along the other spatial direction, thereby helping the network to more accurately locate the object of interest. Specifically, the aggregated feature maps in the horizontal and vertical directions generated according to formulas (5) and (6) are concatenated together and then sent to the shared 1×1 convolution transformation function to generate the intermediate feature map, which is:

[0093] (7)

[0094] Among them, represents a connection operation along the spatial dimension, is the intermediate feature map, which is used to encode the spatial information in the horizontal and vertical directions, C is the control block, r is the reduction rate of the control block size;

[0095] After splitting the intermediate feature map into two separate tensors along the spatial dimension and two 1×1 convolutional transformations are adopted and are respectively used to convert and into tensors with the same number of channels as the input tensor That is:

[0096] (8)

[0097] Among them, is the Sigmoid function;

[0098] In order to reduce the model complexity, an appropriate scale is used to reduce the number of channels, and then the two tensors and are respectively used as attention weights to obtain the output features of the coordinate attention mechanism, which are:

[0099] (9)

[0100] Among them, relu is the Relu activation function, y is the output feature, is the residual term, , are respectively the horizontal and vertical coordinate positions corresponding to the feature map.

[0101] Finally, two layers of three-dimensional convolutions are used to process the changes in the target appearance in the time dimension to achieve the interactive matching of the target features between frames. By stacking the output features y of consecutive N+1 frames to obtain , and then performing two three-dimensional convolution operations, and then obtaining the representation of the target temporal appearance feature through the unfolding operation, that is:

[0102] (10)

[0103] Among them, represents the tensor stacking operation, is the output feature of the first three-dimensional convolution, is the output feature of the second three-dimensional convolution.

[0104] In this embodiment, after splicing the motion feature map and the temporal appearance feature map of the same candidate trajectory, the classification module of the trajectory discrimination network model is input to obtain the authenticity of the candidate trajectory and complete the implementation process of real target detection. Specifically:

[0105] First, splice the motion feature map of the same candidate trajectory with the temporal appearance feature map into a multi-layer perceptron (MLP) with three linear layers, and then obtain the probability of the candidate trajectory through the Sigmoid function;

[0106] Determine whether the probability of the candidate trajectory exceeds the threshold:

[0107] If so, determine that the candidate trajectory is a real trajectory, and output the target in the real trajectory as a real target;

[0108] Otherwise, determine that the candidate trajectory is noise.

[0109] The above is only the preferred embodiment of the present invention, and does not limit the protection scope of the present invention. Any equivalent structural transformation made under the inventive concept of the present invention by using the content of the specification and drawings of the present invention, or directly / indirectly applied to other related technical fields, is included in the protection scope of the present invention.

Claims

1. An infrared image weak moving target detection method based on a trajectory discrimination network model, characterized in that, It includes the following steps: Step 1: Obtain the infrared image sequence of the target, and process the infrared image sequence to obtain the binary image of each frame of the infrared image; Step 2: Extract the position information and size information of each candidate target in each of the binary images; Step 3: Arrange and combine the candidate targets in multiple consecutive frames of images to obtain a number of candidate trajectories; Step 4: Input the position information and size information of the candidate targets in each frame of the same candidate trajectory into the motion feature analysis model of the trajectory discrimination network model to obtain the motion feature map of the candidate trajectory; Step 5: Based on the position information and size information of each candidate target, take a screenshot in the original infrared image to obtain the spatial domain image of each candidate target at each frame moment, specifically: Obtain the height size H and width size W of the candidate target, and take the larger value L of the target size (W, H), and take the coordinates of the position information of the candidate target as the center, and intercept a square with a side length of L in the original infrared image, and ensure that the intercepted image block can completely contain the candidate target pixels, that is, obtain the spatial domain image of the candidate target in the original infrared image of the current frame; Step 6: Input the spatial domain images of the candidate targets in the same candidate trajectory into the time-domain appearance feature analysis model of the trajectory discrimination network model to obtain the time-domain appearance feature map of the candidate trajectory, specifically including: Obtain the first-frame spatial domain image I of the candidate targets in the same candidate trajectory; Use two layers of two-dimensional convolution to extract the target spatial domain feature map of the image I respectively, and after the second layer of two-dimensional convolution, input the generated feature map into the coordinate attention module, and use two pooling kernels with spatial ranges (H, 1) or (1, W) for each channel respectively, and encode along the horizontal and vertical coordinates to generate the aggregated feature maps in the horizontal and vertical directions; After splicing the aggregated feature maps in the horizontal and vertical directions, send them to the shared 1×1 convolution transformation function to generate the intermediate feature map; After splitting the intermediate feature map into two separate tensors along the spatial dimension, use the two tensors as attention weights respectively to obtain the output features of the coordinate attention mechanism; After stacking the output features corresponding to all the spatial domain images of the candidate targets in the same candidate trajectory, perform two three-dimensional convolution operations and then unfold them to obtain the time-domain appearance feature map of the candidate trajectory; Step 7: After splicing the motion feature map and the time-domain appearance feature map of the same candidate trajectory, input them into the classification module of the trajectory discrimination network model to obtain the authenticity of the candidate trajectory and complete the detection of real targets.

2. The method for detecting a weak and small moving target in an infrared image based on a trajectory discrimination network model according to claim 1, wherein, Step 2 specifically includes: Take the coordinates of the centroid of the set of candidate target pixel points in the binary image as the position information of the candidate target; Obtain the size information of the candidate target according to the length and width values of the smallest target box that can contain the set of candidate target pixel points in the binary image.

3. The method for detecting a small and weak moving target in an infrared image based on a trajectory discrimination network model according to claim 1, wherein In Step 4, the motion feature analysis model is an LSTM model, and the extraction process of the motion feature map is specifically: Register the position information and size information of candidate targets in each frame of image for the same candidate trajectory to obtain the registered position and size information , where is the registered position information of the candidate target in the t + k -th frame of image, is the registered width size information of the candidate target in the t + k -th frame of image, is the registered height size information of the candidate target in the t + k -th frame of image, and the subscript indicates that the candidate trajectory consists of N + 1 consecutive candidate targets; Input the registered position and dimension information into the LSTM model together to obtain the motion feature map of the candidate trajectory.

4. The method for detecting a small and weak moving target in an infrared image based on a trajectory discrimination network model according to claim 1 or 2 or 3, characterized in that, Step 7 specifically includes: Splice the motion feature map and the time-domain appearance feature map of the same candidate trajectory into a multi-layer perceptron with three linear layers, and then process through the Sigmoid function to obtain the probability of the candidate trajectory; Determine whether the probability of the candidate trajectory exceeds the threshold: If so, determine that the candidate trajectory is a true trajectory, and output the target in the true trajectory as the true target; Otherwise, determine that the candidate trajectory is noise.

Images