Methods, devices, and storage media for small target detection and tracking in infrared images
By preprocessing infrared images and combining a U-shaped target detection network with Hungarian algorithm matching and Kalman filter tracking, the problem of detecting and tracking weak targets in infrared images is solved, achieving higher detection accuracy and tracking precision.
Patent Information
- Application Number
- CN202410187179.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-20
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-02-20
AI Technical Summary
Traditional infrared image-based methods for detecting weak targets have poor performance in scenarios with complex backgrounds and low signal-to-noise ratios, making it difficult to effectively detect and track weak targets at long distances, especially affected by noise and clutter interference.
Preprocessing is performed using frame-by-frame input sequence infrared images. The signal-to-noise ratio is enhanced using a second-order Laplacian operator and LBP. Target detection is performed using a U-shaped target detection network. A Hungarian algorithm is used to match the tracker, and a pipeline filter improved by Kalman filtering is used for target selection and tracking.
It improves the detection accuracy and tracking precision of small targets in infrared images, enhances the local signal-to-noise ratio, effectively eliminates background noise interference, and improves F1 score, recall, and precision.
Smart Images

Figure CN118015313B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of military tracking, and more particularly to a method, device, and storage medium for small target detection and tracking in infrared images. Background Technology
[0002] In recent years, with the rapid development of military technology, the power of military weapons has been greatly enhanced. Fighter jets and precision-guided missiles play a crucial role on the battlefield, but also pose a serious threat to security. In modern warfare, detecting targets at a sufficiently long distance is essential for defense systems to have enough reaction time. The greater the distance, the smaller the target's image area on an infrared imaging system, making it more susceptible to noise and clutter interference, thus increasing the difficulty of detection and tracking. Furthermore, using infrared imaging presents many challenges, such as the lack of crucial information about the target's texture, size, and structure due to the long distance of the detection equipment. Additionally, the low contrast of infrared imaging and potential background obstruction (clouds, ground features) and other interference can cause weak target information to be buried under background clutter and noise.
[0003] Traditional infrared image weak target detection methods typically rely on manual parameter selection and tuning, such as target segmentation thresholds and target detection sensitivity, and use prior knowledge, such as assuming small targets follow a Gaussian distribution and that the image background space is uniform. These methods perform poorly in scenes with complex backgrounds and low signal-to-noise ratios. Currently, traditional methods are being replaced by deep learning methods. Summary of the Invention
[0004] To address the problems of noise and clutter interference in infrared imaging system detection and tracking, this invention provides a method, device, and storage medium for small target detection and tracking in infrared images. The method mainly includes the following steps: S1. Input the sequence of infrared images frame by frame as the original images, and perform two preprocessing steps respectively; S2. Input the preprocessed image and the original image together into the U-shaped target detection network to obtain the detected target; S3. Use the Hungarian algorithm to match the detected target and the tracker to obtain the matching results; S4. Filter valid targets from the matching results; S5. Apply a pipe filter improved with Kalman filtering to the valid targets and output the target detection image.
[0005] Furthermore, the preprocessing includes a second-order Laplacian operator and LBP, performed simultaneously and separately, to enhance the signal-to-noise ratio of the image; The expression for the two-dimensional discrete Laplace operator is:
[0006] In the expression, (x, y) represents the position of a pixel in the image, and f(x, y) represents the intensity of the pixel in the image. The result is calculated using the second-order Laplacian operator; the Laplacian operator emphasizes high-frequency changes in the image and produces a strong response at the image edges, thereby highlighting the edge features of the original image; LBP is used to obtain the texture features of an image. The basic calculation formula is as follows:
[0007]
[0008] in, Representative field pixel The result of comparing the intensity with that at the center point c; Represents the center pixel. Neighboring pixels are pixels within a square area centered on the center point. Represents the center point in the image Value is to The result after converting to decimal.
[0009] Furthermore, the U-shaped target detection network consists of a two-layer structure of encoder-Swin transformer module-decoder group, independent decoders, and classifiers. The two-layer structure is interconnected through the encoder and the last decoder in the decoder group.
[0010] The encoder structure consists of a convolutional layer followed by a normalization layer, and then a ReLU activation function. The convolutional layer uses a 3x3 kernel with a stride of 2 for downsampling to reduce the spatial size of the feature map. The normalization layer performs batch normalization on the output of the convolutional layer to accelerate the training process and improve the stability of the model. Finally, the output of the first convolutional layer is non-linearly activated by the ReLU activation function.
[0011] The Swin transformer module includes Linear Embedding, Patch Merging, the SwinTransformer Block, residual connections, and a layer normalization module (LN). The Swin Transformer Block consists of a multi-head self-attention module (MSA) based on a shift window and a multilayer perceptron (MLP). In the first stage, the input feature map is divided into tensors of fixed size. Each tensor block is flattened and converted into an embedding vector through a 3x3 convolutional layer. The number of feature channels is adjusted by Linear Embedding, and then the vector enters the Swin Transformer Block. LN normalizes each MSA and MLP module. In the subsequent three stages, Linear Embedding is replaced with Patch Merging for downsampling to reduce the spatial resolution of the feature map. The same operation is repeated thereafter.
[0012] The decoder uses a subpixel convolution method. The specific process is as follows: the input channels are rearranged according to the upsampling factor. Each channel is divided into blocks of size equal to the upsampling factor, and the blocks are rearranged into new channels, so that the feature map of each channel is divided into multiple sub-maps. The result after channel rearrangement is reshaped, and a convolution operation is performed on the reshaped result to extract higher-level features. The channels of the convolution result are merged to obtain a high-resolution feature map.
[0013] The specific operation process of the classifier is as follows: The feature map is input into the first convolutional layer with a kernel size of 3x3, padding of 1, and stride of 1. Then, batch normalization is performed on the output of the first convolutional layer, and ReLU activation is performed on the batch normalized result. The ReLU activated feature map is then fed into the second convolutional layer with a kernel size of 3x3, padding of 1, and stride of 1. Finally, the small target heatmap of the original image is obtained. The center position of the heatmap is obtained through threshold segmentation as the specific location of the small target in the image.
[0014] Furthermore, step S2 specifically includes: S21. Input the preprocessed image and the original image into the first layer encoder, extract feature information at different depths and filter out some background noise to highlight the target, and use the output of the first layer encoder as the input of the second layer encoder to obtain the output result; S22. The output of the encoder is used as the input of the Swin transformer module connected to the current layer to integrate global and local information. S23. The encoder group decodes the output information of the Swin transformer module in the order of connection, gradually transforming the low-resolution abstract feature mapping into high-resolution features that match the input image. The input of the last decoder in the current layer encoder group is the output of the connected current layer decoder and the last decoder in the second layer encoder group. Finally, its output is input into an independent decoder outside the two-layer structure to obtain high-resolution features. S24. The classifier classifies the high-resolution features extracted by the independent decoder to obtain the location of small targets in the image.
[0015] Furthermore, step S3 specifically includes: Select the positions of small targets in the images of the sequence and the previous frame image, and use the Hungarian matching algorithm to match the same target in the two images: S31. Create a two-dimensional matrix as the cost matrix, where the rows of the cost matrix represent the detected targets in the current frame, the columns represent the tracked targets in the previous frame, and the elements record the cost associated with the detected targets in the current frame and the detected targets in the previous frame. The cost uses Euclidean distance. Initialize a matrix of the same size as the cost matrix with all elements equal to 0 as the label matrix. S32. In the cost matrix, for each row, find the minimum value of that row and subtract that minimum value from all elements of that row; S33. In the cost matrix, for each column, find the minimum value of that column and subtract that minimum value from all elements of that column; S34. Use the fewest possible horizontal and vertical lines to pass through all zero elements in the cost matrix; S35. Set the marker matrix elements at the same positions as the positions crossed by the straight line in the corresponding cost matrix to 1; S36. Check if the entire marker matrix is 1. If so, the algorithm ends and the matching result is the row and column target corresponding to the 0 element in the cost matrix; otherwise, proceed to the next step. S37. Find the smallest element in the cost matrix that is not set to 1, and denote it as m. Then subtract m from all elements in the cost matrix that are not crossed by a straight line, and add m to all intersections in the cost matrix. S38. Return to step S34 and continue execution.
[0016] Furthermore, the matching results fall into three categories: unmatched trackers, unmatched targets, and matched targets.
[0017] Furthermore, step S4 specifically includes: S41. For unmatched trackers, determine whether the tracker's loss age is greater than the threshold. If so, delete the tracker; otherwise, retain the tracker, predict its location, and increase its loss age. S42. For a matched target, determine whether the target is within the range of the tracker's pipeline filter. If it is, it is determined to be a valid target; otherwise, it is determined to be an interfering target. Create a new tracker to track the target and repeat steps S3-S4. S43. For unmatched targets, create a new tracker to track the target, and repeat steps S3-S4. Furthermore, step S5 specifically includes: Use a valid target to calculate the parameters of the updated Kalman filter and use the Kalman filter to predict the center position and pipe size of the updated pipe filter. Then add the target to the pipe buffer of the tracker and determine whether the pipe buffer of each tracker has reached the threshold. If so, pop up the target output detection image. The specific operation process of the model is as follows: S51. Initialize the state estimation of the Kalman filter. And error covariance matrix ; The target's state variables are position and velocity, expressed as follows: ,in and It represents the position of the target on the horizontal and vertical axes of the k-th frame of the image. and It refers to the target's velocity along the horizontal and vertical axes; S52. Based on the state transition matrix Predicted state The formula is as follows:
[0018] in, For the predicted state estimation; State estimation at time k-1; S53, Prediction Error Covariance Matrix :
[0019] in, It is process noise; It is the error covariance matrix at time k-1; S54. Calculate Kalman gain :
[0020] in, It is a measurement matrix; It measures noise; S55. Based on the observation data at the current moment Updated state estimate:
[0021] in, These are actual measured values; It is an updated state estimate; S56. Update the error covariance matrix :
[0022] S57. Repeat steps S52-S56 to continuously estimate the state. In the formula The first two items To predict the center of the pipeline Predict the radius of the pipeline It can be expressed as the following formula:
[0023] A storage medium storing instructions and data for implementing a method for small target detection and tracking in infrared images.
[0024] A computer device includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a method for small target detection and tracking in infrared images.
[0025] The beneficial effects of the technical solution provided by this invention are as follows: This invention uses frame-by-frame input sequence infrared images as the original images, performs two preprocessing steps, feeds the preprocessed images and the original images together into a U-shaped target detection network to obtain the detection targets, uses the Hungarian algorithm to match the detection targets and the tracker to obtain the matching results, filters valid targets from the matching results, and applies a pipeline filter improved from Kalman filtering to the valid targets, popping out the target output detection image. This enhances the local signal-to-noise ratio of the infrared image, eliminates background noise interference, and accurately detects and tracks targets in the infrared image. This method provides a new approach for the detection and tracking of weak targets in infrared images. Attached Figure Description
[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram illustrating the steps of a method for small target detection and tracking in infrared images according to an embodiment of the present invention; Figure 2 This is a flowchart of a method for small target detection and tracking in infrared images according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the Laplacian operator enhancement contrast in an embodiment of the present invention; Figure 4 This is a schematic diagram of LBP texture encoding comparison in an embodiment of the present invention; Figure 5 This is a schematic diagram of the U-shaped target detection network framework in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the principle of the pipeline filter in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the principle of the improved pipeline filter in an embodiment of the present invention; Figure 8 This is a schematic diagram of the results of the weak target detection and tracking method using sequential infrared images in an embodiment of the present invention; Figure 9 This is a schematic diagram of the hardware device working in an embodiment of the present invention. Detailed Implementation
[0027] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0028] Embodiments of the present invention provide a method, device, and storage medium for small target detection and tracking in infrared images.
[0029] Please refer to Figure 1 and Figure 2 , combined Figure 1 and Figure 2 The method for small target detection and tracking in infrared images according to an embodiment of the present invention specifically includes the following steps: The first step is to input a sequence of infrared images frame by frame as the raw images, and then perform two different preprocessing steps.
[0030] The image preprocessing section is mainly used to enhance the signal-to-noise ratio of small targets in infrared images to facilitate subsequent detection. Two image processing methods are used in this section: the second-order Laplacian operator and LBP (Local Binary Pattern) to highlight the edge information and obtain the texture features of the image.
[0031] The second-order Laplacian operator is often used to detect edge and texture variations in images. In discrete images,
[0032] In the expression, (x, y) represents the position of a pixel in the image, and f(x, y) represents the intensity of the pixel in the image. This is the result calculated using the second-order Laplacian operator. The formula represents the difference between a given pixel and its four surrounding pixels. The Laplacian value of that pixel is obtained by weighted summing four times the value of the center point.
[0033] This form of the Laplacian operator emphasizes high-frequency changes in the image, thus producing a stronger response at image edges. A comparison of the before and after processing is shown below. Figure 3 As shown.
[0034] Layered Backpropagation (LBP) is a commonly used method for extracting texture features from images. LBP primarily describes the spatial structure of local textures in an image. It is relatively robust to changes in illumination and grayscale, and therefore has been widely applied in many image processing tasks, such as face recognition and texture classification. The basic calculation formula for LBP is as follows:
[0035]
[0036] in, Representative field pixel The result of comparing the intensity with that at the center point c; Represents the center pixel. Neighboring pixels are pixels within a square area centered on the center point. Represents the center point in the image Value is to The result after converting to decimal.
[0037] The binary pattern obtained by binarization The data is combined into a binary number, then converted to its corresponding decimal number to obtain the LBP value. By performing operations on each pixel in the image, the texture features of the image can be obtained. A comparison before and after processing is provided. Figure 4 As shown.
[0038] The second step is to feed the preprocessed image and the original image together into the U-shaped target detection network to obtain the target to be detected.
[0039] The U-shaped target detection network is mainly used to detect the location of weak targets in an image by combining the preprocessed image and the original image. The specific network structure is as follows: Figure 5 As shown, it consists of a two-layer structure of encoder-Swin transformer module-decoder group, independent decoder and classifier. The two-layer structure is interconnected through the encoder and the last decoder of the decoder group.
[0040] The encoder extracts feature information at different depths and filters background noise to highlight the target, thus better extracting semantic and feature information from the input image. The Swin transformer module integrates both global and local information, crucial for contextual understanding in object detection tasks. The local window mechanism allows the model to focus on local features, while the integration of global information enhances the target's contextual awareness. The decoder transforms low-resolution abstract feature maps into high-resolution feature maps that match the input image. This helps the network better understand the local structure and details of the image. The classifier classifies the image features extracted by the decoder to determine the location of small targets in the image.
[0041] The encoder structure consists of a convolutional layer followed by a normalization layer, and then a ReLU activation function. The convolutional layer uses a 3x3 kernel with a stride of 2 for downsampling to reduce the spatial size of the feature map. The normalization layer performs batch normalization on the output of the convolutional layer to accelerate the training process and improve the stability of the model. Finally, the output of the first convolutional layer is non-linearly activated by the ReLU activation function.
[0042] The Swin transformer module includes Linear Embedding, Patch Merging, the SwinTransformer Block, residual connections, and a layer normalization module (LN). The Swin Transformer Block consists of a multi-head self-attention module (MSA) based on a shift window and a multilayer perceptron (MLP). In the first stage, the input feature map is divided into tensors of fixed size. Each tensor block is flattened and converted into an embedding vector through a 3x3 convolutional layer. The number of feature channels is adjusted by Linear Embedding, and then the vector enters the Swin Transformer Block. LN normalizes each MSA and MLP module. In the subsequent three stages, Linear Embedding is replaced with Patch Merging for downsampling to reduce the spatial resolution of the feature map. The same operation is repeated thereafter.
[0043] The decoder uses a subpixel convolution method. The specific process is as follows: the input channels are rearranged according to the upsampling factor. Each channel is divided into blocks of size equal to the upsampling factor, and the blocks are rearranged into new channels, so that the feature map of each channel is divided into multiple sub-maps. The result after channel rearrangement is reshaped, and a convolution operation is performed on the reshaped result to extract higher-level features. The channels of the convolution result are merged to obtain a high-resolution feature map.
[0044] The specific operation process of the classifier is as follows: The feature map is input into the first convolutional layer with a kernel size of 3x3, padding of 1, and stride of 1. Then, batch normalization is performed on the output of the first convolutional layer, and ReLU activation is performed on the batch normalized result. The ReLU activated feature map is then fed into the second convolutional layer with a kernel size of 3x3, padding of 1, and stride of 1. Finally, the small target heatmap of the original image is obtained. The center position of the heatmap is obtained through threshold segmentation as the specific location of the small target in the image.
[0045] The inputs and outputs of each module in the overall network structure diagram are shown in Table 1.
[0046] Table 1. Tensor shapes for the inputs and outputs of each module.
[0047] The third step is to use the Hungarian algorithm to match the detected target and the tracker to obtain the matching results.
[0048] Select the positions of small targets in the images of the sequence and the previous frame image, and use the Hungarian matching algorithm to match the same target in the two images: Step 1: Create a two-dimensional matrix called the cost matrix. The rows of the cost matrix represent the detected targets in the current frame, the columns represent the tracked targets in the previous frame, and the elements record the cost associated with the detected targets in the current frame and the detected targets in the previous frame. The cost uses Euclidean distance. Initialize a matrix of the same size as the cost matrix with all elements equal to 0 as the label matrix.
[0049] Step 2: For each row in the cost matrix, find the minimum value of that row and subtract that minimum value from all elements of that row.
[0050] Step 3: For each column in the cost matrix, find the minimum value of that column and subtract that minimum value from all elements in that column.
[0051] Step four: Use the fewest possible horizontal and vertical lines to pass through all the zero elements in the cost matrix.
[0052] Step 5: Set the marker matrix elements at the same positions as the locations crossed by the straight line in the corresponding cost matrix to 1.
[0053] Step 6: Check if the entire marker matrix is 1. If so, the algorithm ends and the matching result is the row and column target corresponding to the 0 element in the cost matrix; otherwise, proceed to step 7.
[0054] Step 7: Find the smallest element in the cost matrix that is not set to 1, denoted as m. Then, subtract m from all elements in the cost matrix that are not crossed by a straight line, and add m to all intersections in the cost matrix.
[0055] Step eight: Return to step four and continue.
[0056] The matching results are divided into three categories: unmatched trackers, unmatched targets, and matched targets.
[0057] The fourth step is to filter valid targets from the matching results.
[0058] For unmatched trackers, determine whether the tracker's loss age is greater than the threshold. If so, delete the tracker; otherwise, keep the tracker, predict its location, and increase its loss age. For a matched target, determine whether the target is within the range of the tracker's pipeline filter. If it is, it is considered a valid target; otherwise, it is considered an interfering target. Create a new tracker to track the target and repeat the matching and filtering process, i.e., steps three and four.
[0059] For unmatched targets, create a new tracker to track the target and repeat the matching and filtering process.
[0060] The fifth step involves applying a pipe filter improved from the Kalman filter to the valid targets, which then outputs the target detection image.
[0061] The pipe filter is a simple noise filter, essentially a spatiotemporal filter. The basic idea of this algorithm is to further distinguish between real and false targets by utilizing the locations of candidate targets in a sequence of images. Its schematic diagram is shown below. Figure 6 As shown, the position of the detected target in the k-th frame is... ,by Draw a circle with a manually defined radius and a central point. Extend the area of the circle to the subsequent M frames to form a pipeline. If the target is detected in every frame of the pipeline, the target is determined to be a real target; otherwise, it is a false target.
[0062] Traditional pipeline filtering algorithms use a fixed pipeline center position and an artificially determined pipeline radius. This leads to a problem: when the target's speed changes rapidly, it may move outside the pipeline's range, resulting in false positives.
[0063] Therefore, this paper proposes a dynamic pipeline filtering algorithm, such as... Figure 7 As shown, a Kalman filter is introduced to predict the target's location in the next frame. This location is used as the center of the pipeline filter for that frame, and the radius of the pipeline is set according to the object's motion speed. If the target's spatial position changes significantly between adjacent frames, the radius should be increased appropriately to prevent the target from being removed as noise; conversely, if the target's spatial position does not change significantly between adjacent frames, the radius should be adaptively decreased to prevent noise from remaining within the pipeline. This achieves more accurate pipeline filtering.
[0064] Use a valid target to calculate the parameters of the updated Kalman filter and use the Kalman filter to predict the center position and pipe size of the updated pipe filter. Then add the target to the pipe buffer of the tracker and determine whether the pipe buffer of each tracker has reached the threshold. If so, pop up the target output detection image. The specific operation process of the model is as follows: Step 1: Initialize the state estimation of the Kalman filter. And error covariance matrix ; The target's state variables are position and velocity, expressed as follows: ,in and It represents the position of the target on the horizontal and vertical axes of the k-th frame of the image. and It refers to the target's velocity on the horizontal and vertical axes.
[0065] Step 2, based on the state transition matrix Predicted state The formula is as follows:
[0066] in, For the predicted state estimation; This is the state estimate at time k-1.
[0067] Step 3, Prediction Error Covariance Matrix :
[0068] in, It is process noise; It is the error covariance matrix at time k-1.
[0069] Step 4: Calculate the Kalman gain :
[0070] in, It is a measurement matrix; It measures noise.
[0071] Step 5: Based on the observation data at the current moment Updated state estimate:
[0072] in, These are actual measured values; It is an updated state estimate.
[0073] Step 6: Update the error covariance matrix :
[0074] Step 7: Repeat steps 2 through 6 to continuously estimate the state.
[0075] In the formula The first two items To predict the center of the pipeline Predict the radius of the pipeline It can be expressed as the following formula:
[0076] Please refer to Figure 8 , Figure 8 This diagram illustrates the results of a weak target detection and tracking method for sequential infrared images. In a continuous sequence of frames, small targets are accurately detected, and the same target is accurately correlated between frames. Experimental results show that, using the same publicly available dataset MFIRST and the same experimental equipment, compared to traditional infrared image weak target detection and tracking methods, this method improves the F1 score by 10.9%, the recall score by 16.7%, and the precision score by 7.9%.
[0077] Please see Figure 9 , Figure 9 This is a schematic diagram of the hardware device operation according to an embodiment of the present invention. The hardware device specifically includes: a computer device 401, a processor 402, and a storage medium 403.
[0078] A computer device 401: The computer device 401 implements the method for small target detection and tracking in infrared images.
[0079] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the method for small target detection and tracking in infrared images.
[0080] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the method for small target detection and tracking in infrared images.
[0081] The beneficial effects of this invention are as follows: This invention uses frame-by-frame input of sequential infrared images as the original images, performs two preprocessing steps, and then feeds both the preprocessed and original images into a U-shaped target detection network to obtain the detection targets. The Hungarian algorithm is used to match the detected targets with the tracker to obtain the matching results. Valid targets are then selected from the matching results, and a pipeline filter improved from Kalman filtering is applied to the valid targets to output the target detection image. This enhances the local signal-to-noise ratio of the infrared image, eliminates background noise interference, and accurately detects and tracks targets in the infrared image. This method provides a new approach for the detection and tracking of weak targets in infrared images.
[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for small target detection and tracking in infrared images, characterized in that, The specific steps include: S1. Input the sequence of infrared images frame by frame as the original images, and perform two preprocessing steps respectively; the preprocessing includes the second-order Laplacian operator and LBP, which are performed simultaneously to enhance the signal-to-noise ratio of the images; S2. The preprocessed image and the original image are fed into the U-shaped target detection network to obtain the detected target. The U-shaped target detection network consists of a two-layer structure of encoder-Swin transformer module-decoder group, independent decoder and classifier. The two-layer structure is interconnected through the encoder and the last decoder of the decoder group. Step S2 is as follows: S21. Input the preprocessed image and the original image into the first layer encoder, extract feature information at different depths and filter out some background noise to highlight the target, and use the output of the first layer encoder as the input of the second layer encoder to obtain the output result; S22. The output of the encoder is used as the input of the Swin transformer module connected to the current layer to integrate global and local information. S23. The encoder group decodes the output information of the Swin transformer module in the order of connection, gradually transforming the low-resolution abstract feature mapping into high-resolution features that match the input image. The input of the last decoder in the current layer encoder group is the output of the connected current layer decoder and the last decoder in the second layer encoder group. Finally, its output is input into an independent decoder outside the two-layer structure to obtain high-resolution features. S24. The classifier classifies the high-resolution features extracted by the independent decoder to obtain the location of small targets in the image; S3. Use the Hungarian algorithm to match the detected target and the tracker to obtain the matching results; Step S3 is as follows: The positions of small targets in the images of the sequence and the previous frame are selected. The Hungarian matching algorithm is then used to match the same target in the two images. The specific process is as follows: S31. Create a two-dimensional matrix as the cost matrix, where the rows of the cost matrix represent the detected targets in the current frame, the columns represent the tracked targets in the previous frame, and the elements record the cost associated with the detected targets in the current frame and the detected targets in the previous frame. The cost uses Euclidean distance. Initialize a matrix of the same size as the cost matrix with all elements equal to 0 as the label matrix. S32. In the cost matrix, for each row, find the minimum value of that row and subtract that minimum value from all elements of that row; S33. In the cost matrix, for each column, find the minimum value of that column and subtract that minimum value from all elements of that column; S34. Use the fewest possible horizontal and vertical lines to pass through all zero elements in the cost matrix; S35. Set the marker matrix elements at the same positions as the positions crossed by the straight line in the corresponding cost matrix to 1; S36. Check if the entire label matrix is 1. If yes, the algorithm ends; otherwise, proceed to the next step. S37. Let m be the smallest element in the cost matrix at the same position where the marker matrix is not set to 1. Then subtract m from all elements in the cost matrix that are not crossed by a straight line, and add m to all intersections in the cost matrix. S38. Return to step S34 and continue execution; S4. Filter valid targets from the matching results; Step S4 is as follows: S41. For unmatched trackers, determine whether the tracker's loss age is greater than the threshold. If so, delete the tracker; otherwise, keep the tracker, predict the tracker's location, and increase the tracker's loss age. S42. For a matched target, determine whether the target is within the range of the tracker's pipeline filter. If it is, it is determined to be a valid target; otherwise, it is determined to be an interfering target. Create a new tracker to track the target and repeat steps S3-S4. S43. For unmatched targets, create a new tracker to track the target, and repeat steps S3-S4. S5. Apply a pipe filter improved with Kalman filtering to the effective targets and pop up the target output detection image; The parameters of the updated Kalman filter are calculated using a valid target, and the Kalman filter is used to predict the center position and pipe size of the updated pipe filter. The target is then added to the pipe buffer of the tracker. It is determined whether the pipe buffer of each tracker has reached the threshold. If so, the target output detection image is popped out.
2. The method for small target detection and tracking in infrared images as described in claim 1, characterized in that, The expression for the two-dimensional discrete Laplace operator is: In the expression, (x, y) represents the position of a pixel in the image, and f(x, y) represents the intensity of the pixel in the image. The result is calculated using the second-order Laplace operator; The Laplacian operator emphasizes high-frequency changes in an image and produces a strong response at image edges, thereby highlighting the edge features of the original image. LBP is used to obtain the texture features of an image. The basic calculation formula is as follows: in, Representative field pixel The result of comparing the intensity with that at the center point c; Represents the center pixel. Neighboring pixels are pixels within a square area centered on the center point. Represents the center point in the image Value is to The result after converting to decimal.
3. The method for small target detection and tracking in infrared images as described in claim 1, characterized in that, The encoder structure consists of a convolutional layer followed by a normalization layer, and then a ReLU activation function. The convolutional layer uses a 3x3 kernel with a stride of 2 for downsampling to reduce the spatial size of the feature map. The normalization layer performs batch normalization on the output of the convolutional layer to accelerate the training process and improve the stability of the model. Finally, the output of the first convolutional layer is non-linearly activated by the ReLU activation function. The Swin transformer module includes Linear Embedding, Patch Merging, Swin Transformer Block, Residual Connections, and Layer Normalization (LN) modules. The Swin Transformer Block consists of a multi-head self-attention module (MSA) based on a shift window and a multilayer perceptron (MLP). In the first stage, the input feature map is divided into tensors of fixed size. Each tensor block is flattened and converted into an embedding vector through a 3x3 convolutional layer. The number of feature channels is adjusted by Linear Embedding, and then the input enters the Swin Transformer Block module. LN normalizes each MSA module and MLP module. In the subsequent three stages, Linear Embedding is replaced with Patch Merging for downsampling to reduce the spatial resolution of the feature map. The same operation is repeated thereafter. The decoder uses a subpixel convolution method. The specific process is as follows: the input channels are rearranged according to the upsampling factor. Each channel is divided into blocks of size equal to the upsampling factor, and the blocks are rearranged into new channels, so that the feature map of each channel is divided into multiple sub-maps. The result after channel rearrangement is reshaped, and a convolution operation is performed on the reshaped result to extract higher-level features. The channels of the convolution result are merged to obtain a high-resolution feature map. The specific operation process of the classifier is as follows: The feature map is input into the first convolutional layer with a kernel size of 3x3, padding of 1, and stride of 1. Then, batch normalization is performed on the output of the first convolutional layer, and ReLU activation is performed on the batch normalized result. The ReLU activated feature map is then fed into the second convolutional layer with a kernel size of 3x3, padding of 1, and stride of 1. Finally, the small target heatmap of the original image is obtained. The center position of the heatmap is obtained through threshold segmentation as the specific location of the small target in the image.
4. The method for small target detection and tracking in infrared images as described in claim 1, characterized in that, The matching result is divided into three cases: unmatched trackers, unmatched targets, and matched targets; the matched targets are the row and column targets corresponding to the 0 elements in the cost matrix after the algorithm ends.
5. The method for small target detection and tracking in infrared images as described in claim 1, characterized in that, Step S5 is as follows: The specific operation process of the model is as follows: S51. Initialize the state estimation of the Kalman filter. And error covariance matrix ; The target's state variables are position and velocity, expressed as follows: ,in and It represents the position of the target on the horizontal and vertical axes of the k-th frame of the image. and It refers to the target's velocity along the horizontal and vertical axes; S52. Based on the state transition matrix Predicted state The formula is as follows: in, For the predicted state estimation; State estimation at time k-1; S53, Prediction Error Covariance Matrix : in, It is process noise; It is the error covariance matrix at time k-1; S54. Calculate Kalman gain : in, It is a measurement matrix; It measures noise; S55. Based on the observation data at the current moment Updated state estimate: in, These are actual measured values; It is an updated state estimate; S56. Update the error covariance matrix : S57. Repeat steps S52-S56 to continuously estimate the state. In the formula The first two items To predict the center of the pipeline Predict the radius of the pipeline It can be expressed as the following formula:
6. A storage medium, characterized in that: The storage medium stores instructions and data to implement the method for small target detection and tracking in infrared images as described in any one of claims 1 to 5.
7. A computer device, characterized in that: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the method for small target detection and tracking in infrared images as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Roadside occlusion scene vehicle robust tracking method for intelligent vehicle-road system
CN112581498A
Infrared small object multi-target tracking method and storage medium
CN116245917A
