Infrared small target tracking method and system based on local tracker

Abstract

The application discloses an infrared small target tracking method and system based on local trackers, and the method comprises the following steps: step (1), generating filters corresponding to each convolution layer in a pre-trained multi-convolution layer network; step (2), generating a plurality of local trackers, each local tracker comprising a multi-layer convolution layer network, a plurality of filters, a fusion module and a maximum value solving module; step (3), training the local trackers to obtain trained local trackers; and step (4), dividing a region to be identified into a plurality of local regions, and deploying the trained local trackers in different local regions to track targets, wherein, in the tracking process, if a local target is located, the local tracker will follow the target to move, and the target will be tracked in subsequent frames until the target cannot be tracked any more. The application can realize long-term tracking of frequently discontinuous moving targets.

Description

Technical Field

[0001] This invention relates to the field of infrared recognition technology, and in particular to an infrared small target tracking method and system based on a local tracker. Background Technology

[0002] Infrared weak point target tracking starts with the target's position in an initial frame from an infrared image sequence. Then, the target's state during motion is estimated, and its position in the infrared scene is predicted in subsequent frames to achieve tracking. Because infrared weak target tracking algorithms require high stability, they primarily employ generative and discriminative model-based methods.

[0003] The key challenge in long-term infrared target tracking lies in the difficulty of tracking discontinuous target motion caused by out-of-view or occlusion. Existing long-term tracking methods follow two typical strategies. The first strategy uses a local tracker for smooth tracking, and then uses another re-detector to detect the target when it is lost. While it can utilize temporal context, such as historical appearance and target location, a potential limitation of this strategy is that the local tracker tends to incorrectly identify nearby distractions as targets, rather than activating the re-detector when the real target is out of view. The other long-term tracking strategy is to perform global tracking of the target across the entire image, rather than local tracking based on previous results. However, existing global tracking strategies do not effectively utilize temporal context; most algorithms search for targets within a local region, making them unable to handle the problem of frequent discontinuous target movement in long-term tracking tasks. Summary of the Invention

[0004] Purpose of the invention: This invention addresses the problems existing in the prior art by providing an infrared small target tracking method and system based on a local tracker that can track the frequent and discontinuous movement of targets over long periods.

[0005] Technical solution: The infrared small target tracking method based on a local tracker described in this invention includes:

[0006] Step (1): Generate filters corresponding to each convolutional layer in the network based on the pre-trained multi-convolutional layer network;

[0007] Step (2): Generate multiple local trackers. Each local tracker includes a multi-layer convolutional network, multiple filters, a fusion module, and a maximum value solving module. The multi-layer convolutional network is used to process the input image frame and output the corresponding feature map on each convolutional layer. Each filter is connected to the corresponding convolutional layer of the multi-layer convolutional network and is used to filter the feature map output by the corresponding convolutional layer. The fusion module is connected to all filters and is used to fuse the outputs of all filters to obtain a fused feature map. The maximum value solving module is connected to the fusion module and is used to find the maximum value in the fused feature map and output the position corresponding to the maximum value as the target position.

[0008] Step (3): Train the local tracker to obtain a trained local tracker;

[0009] Step (4): Divide the region to be identified into multiple local regions and deploy the trained local trackers in different local regions for target tracking. During the tracking process, if the local tracker locates the target, the local tracker will follow the target and continue to track the target in subsequent frames until the target can no longer be tracked.

[0010] Furthermore, step (1) specifically includes:

[0011] Step (1-1): Acquire several infrared images and generate corresponding Gaussian labels;

[0012] Step (1-2): Input the infrared image into the pre-trained multi-convolutional network. Each convolutional layer of the multi-convolutional network generates a feature map.

[0013] Steps (1-3): Based on the Gaussian labels of the infrared images and the feature maps of each convolutional layer, the corresponding filters are generated using the following formula:

[0014]

[0015] In the formula, W k Let Y = F(y) represent the filter corresponding to the k-th convolutional layer, Y = F(y) represent the discrete Fourier transform of the Gaussian label y, and F() represent the discrete Fourier transform. k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform of λ, where λ represents the regularization parameter. Let represent the quadratic norm, and W represent the matrix formed by all filters.

[0016] Furthermore, the filter achieves filtration using the following formula:

[0017] P k =F -1 (Xk ·W k )

[0018] In the formula, P k F represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer. -1 () denotes the inverse discrete Fourier transform, X k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform, F() represents the discrete Fourier transform, W k This represents the filter corresponding to the k-th convolutional layer.

[0019] Furthermore, the fusion module implements fusion using the following formula:

[0020]

[0021]

[0022] st∑q ij =1

[0023] In the formula, Q represents the fused feature map, and P k This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer, where n represents the number of filters and KL represents the Kullback-Leibler divergence. P represents k The element in the i-th row and j-th column, q ij This represents the element in the i-th row and j-th column of Q.

[0024] Furthermore, the filter is updated using the following formula:

[0025]

[0026] In the formula, W t k , This represents the filter corresponding to the k-th convolutional layer at time t and time t-1, where γ is the learning rate. This represents the filter update amount, ⊙ represents element-wise multiplication, and Y represents the discrete Fourier transform of the Gaussian label y. Let t represent the infrared image acquired at time t, and λ represent the regularization parameter.

[0027] The infrared small target tracking system based on local trackers described in this invention includes multiple local trackers deployed in different local regions. Each local tracker is trained and includes a multi-layer convolutional network, multiple filters, a fusion module, and a maximum value calculation module. The multi-layer convolutional network processes the input image frame and outputs corresponding feature maps at each convolutional layer. Each filter is connected to the corresponding convolutional layer of the multi-layer convolutional network and filters the feature maps output by that layer. The fusion module connects to all filters and fuses their outputs to obtain a fused feature map. The maximum value calculation module is connected to the fusion module and calculates the maximum value in the fused feature map, outputting the position corresponding to the maximum value as the target position. The filters are generated based on a pre-trained multi-layer convolutional network. During tracking, if a local tracker locates a target, it follows the target and continues tracking it in subsequent frames until the target is no longer detected.

[0028] Furthermore, the filter is generated through the following steps:

[0029] Acquire several infrared images and generate corresponding Gaussian labels;

[0030] The infrared image is input into a pre-trained multi-convolutional network, and each convolutional layer of the multi-convolutional network generates a feature map.

[0031] The Gaussian labels based on infrared images and the feature maps of each convolutional layer are used to generate the corresponding filters using the following formula:

[0032]

[0033] In the formula, W k Let Y = F(y) represent the filter corresponding to the k-th convolutional layer, Y = F(y) represent the discrete Fourier transform of the Gaussian label y, and F() represent the discrete Fourier transform. k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform of λ, where λ represents the regularization parameter. Let represent the quadratic norm, and W represent the matrix formed by all filters.

[0034] Furthermore, the filter achieves filtration using the following formula:

[0035] P k =F -1 (X k ·W k )

[0036] In the formula, P k F represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer.-1 () denotes the inverse discrete Fourier transform, X k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform, F() represents the discrete Fourier transform, W k This represents the filter corresponding to the k-th convolutional layer.

[0037] Furthermore, the fusion module implements fusion using the following formula:

[0038]

[0039]

[0040] st∑q ij =1

[0041] In the formula, Q represents the fused feature map, and P k This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer, where n represents the number of filters and KL represents the Kullback-Leibler divergence. P represents k The element in the i-th row and j-th column, q ij This represents the element in the i-th row and j-th column of Q.

[0042] Furthermore, the filter is updated using the following formula:

[0043]

[0044] In the formula, W t k , This represents the filter corresponding to the k-th convolutional layer at time t and time t-1, where γ is the learning rate. This represents the filter update amount, ⊙ represents element-wise multiplication, and Y represents the discrete Fourier transform of the Gaussian label y. Let t represent the infrared image acquired at time t, and λ represent the regularization parameter.

[0045] Beneficial effects: Compared with the prior art, the significant advantage of this invention is that it equips the local tracker with a global re-detection function to detect the target, and switches between local tracking and global detection based on prior information, which can handle the problem of frequent and discontinuous target movement in long-term tracking tasks. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the infrared small target tracking method based on a local tracker provided by the present invention;

[0047] Figure 2This is a schematic diagram of the local tracker in this invention;

[0048] Figure 3 This is a schematic diagram of target recognition according to the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Example 1

[0051] This embodiment provides an infrared small target tracking method based on a local tracker, such as... Figure 1 As shown, it includes the following steps:

[0052] Step (1): Generate the filter corresponding to each convolutional layer in the network based on the pre-trained multi-convolutional layer network.

[0053] Specifically, the multi-convolutional network is VGG-Net. The steps for generating filters using VGG-Net are as follows: Step (1-1): Acquire several infrared images and generate corresponding Gaussian labels; Step (1-2): Input the infrared images into the pre-trained multi-convolutional network, where each convolutional layer generates a feature map; Step (1-3): Based on the Gaussian labels of the infrared images and the feature maps of each convolutional layer, generate the corresponding filters using the following formula:

[0054]

[0055] In the formula, W k Let Y = F(y) represent the filter corresponding to the k-th convolutional layer, Y = F(y) represent the discrete Fourier transform of the Gaussian label y, and F() represent the discrete Fourier transform. k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform of λ, where λ represents the regularization parameter. Let represent the quadratic norm, and W represent the matrix formed by all filters.

[0056] Step (2): Generate multiple local trackers. Each local tracker includes a multi-layer convolutional network, multiple filters, a fusion module, and a maximum value solving module. The multi-layer convolutional network is used to process the input image frame and output the corresponding feature map on each convolutional layer. Each filter is connected to the corresponding convolutional layer of the multi-layer convolutional network and is used to filter the feature map output by the corresponding convolutional layer. The fusion module is connected to all filters and is used to fuse the outputs of all filters to obtain a fused feature map. The maximum value solving module is connected to the fusion module and is used to find the maximum value in the fused feature map and output the position corresponding to the maximum value as the target position.

[0057] The structure of the local tracker is as follows: Figure 2 As shown, the multi-layer convolutional network is the pre-trained VGG-Net described above. The filter is generated in step (1) and is implemented using the following formula:

[0058] P k =F -1 (X k ·W k )

[0059] In the formula, P k F represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer. -1 () denotes the inverse discrete Fourier transform, X k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform, F() represents the discrete Fourier transform, W k This represents the filter corresponding to the k-th convolutional layer.

[0060] The fusion module uses the following formula to achieve fusion:

[0061]

[0062]

[0063] st∑q ij =1

[0064] In the formula, Q represents the fused feature map, and P k This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer, where n represents the number of filters and KL represents the Kullback-Leibler divergence. P represents k The element in the i-th row and j-th column, q ij This represents the element in the i-th row and j-th column of Q. In fact, each feature map P... kThey can all be viewed as a probability graph, which consists of a probability distribution. This probability distribution represents the probability that position (i, j) becomes the center of the target. Assume the filtered feature map set A = {p} 1，2 p 1，3 , ···, p 2，3 , ···, P n-1，n}, which contains n(n-1) / 2 feature maps with relatively low noise. Now, our goal is to fuse these filtered response maps. The function above can be modified as follows:

[0065]

[0066] The solution to the equation can be obtained using the Lagrange multiplier method, and the final result is as follows:

[0067]

[0068] The equation is the average of all filtered probabilities, which is to strengthen the final result by using a weighted sum of all filtered feature maps.

[0069] The maximum value solving module finds the maximum value in the fused feature map. Use the position corresponding to the maximum value as the target position.

[0070] To adapt the filter to the target appearance, the filter is updated over time using the following formula:

[0071]

[0072] In the formula, W t k , This represents the filter corresponding to the k-th convolutional layer at time t and time t-1, where γ is the learning rate. This represents the filter update amount, ⊙ represents element-wise multiplication, and Y represents the discrete Fourier transform of the Gaussian label y. Let t represent the infrared image acquired at time t, and λ represent the regularization parameter.

[0073] Step (3): Train the local tracker to obtain a trained local tracker.

[0074] During training, several samples, including infrared images and corresponding target locations as labels, are used. The training method adopts existing methods and will not be described in detail here.

[0075] Step (4): Divide the region to be identified into multiple local regions, and deploy the trained local trackers in different local regions for target tracking. During the tracking process, if a local tracker locates the target, it follows the target's movement and continues to track the target in subsequent frames until the target can no longer be tracked. The tracked target is as follows: Figure 3 As shown.

[0076] When the target's motion is smooth, the active local tracker (the local tracker that has been localized to the target) can track the target frame by frame, forming an active local tracking stream. With this process, it is easy to transfer and utilize temporal context from multiple historical frames to perform local tracking within a frame. On the other hand, when the target moves discontinuously due to occlusion or disappearance, although the active local tracker may lose track of the target, another local tracker nearby can take over tracking to localize the target, thus enabling long-term tracking tasks with frequent discontinuous target movement.

[0077] Example 2

[0078] This embodiment provides an infrared small target tracking system based on local trackers. The system includes multiple local trackers deployed in different local regions. Each local tracker is trained and includes a multi-layer convolutional network, multiple filters, a fusion module, and a maximum value calculation module. The multi-layer convolutional network processes the input image frame and outputs corresponding feature maps at each convolutional layer. Each filter is connected to the corresponding convolutional layer of the multi-layer convolutional network and filters the feature maps output by that layer. The fusion module connects to all filters and fuses their outputs to obtain a fused feature map. The maximum value calculation module connects to the fusion module and calculates the maximum value in the fused feature map, outputting the position corresponding to the maximum value as the target position. The filters are generated based on the pre-trained multi-layer convolutional network. During tracking, if a local tracker locates a target, it follows the target and continues tracking it in subsequent frames until the target is no longer tracked.

[0079] Furthermore, the filter is generated through the following steps:

[0080] Acquire several infrared images and generate corresponding Gaussian labels;

[0081] The infrared image is input into a pre-trained multi-convolutional network, and each convolutional layer of the multi-convolutional network generates a feature map.

[0082] Based on the Gaussian labels of the infrared images and the feature maps of each convolutional layer, the corresponding filters are generated using the following formula:

[0083]

[0084] In the formula, W k Let Y = F(y) represent the filter corresponding to the k-th convolutional layer, Y = F(y) represent the discrete Fourier transform of the Gaussian label y, and F() represent the discrete Fourier transform. k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform of λ, where λ represents the regularization parameter. Let represent the quadratic norm, and W represent the matrix formed by all filters.

[0085] Furthermore, the filter achieves filtration using the following formula:

[0086] P k =F -1 (X k ·W k )

[0087] In the formula, P k F represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer. -1 () denotes the inverse discrete Fourier transform, X k =F(x) k ) represents the feature map x output by the k-th convolutional layer. k The discrete Fourier transform, F() represents the discrete Fourier transform, W k This represents the filter corresponding to the k-th convolutional layer.

[0088] Furthermore, the fusion module implements fusion using the following formula:

[0089]

[0090]

[0091] st∑q ij =1

[0092] In the formula, Q represents the fused feature map, and P k This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer, where n represents the number of filters and KL represents the Kullback-Leibler divergence. P represents k The element in the i-th row and j-th column, q ij This represents the element in the i-th row and j-th column of Q.

[0093] Furthermore, the filter is updated using the following formula:

[0094]

[0095] In the formula, W t k , This represents the filter corresponding to the k-th convolutional layer at time t and time t-1, where γ is the learning rate. This represents the filter update amount, ⊙ represents element-wise multiplication, and Y represents the discrete Fourier transform of the Gaussian label y. Let t represent the infrared image acquired at time t, and λ represent the regularization parameter.

[0096] It is worth noting that in the embodiments of the above-mentioned determining device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

Claims

1. A method for tracking small infrared targets based on a local tracker, characterized in that... The method includes: Step (1): Generate filters corresponding to each convolutional layer in the network based on the pre-trained multi-convolutional layer network; Step (2): Generate multiple local trackers. Each local tracker includes a multi-convolutional layer network, multiple filters, a fusion module, and a maximum value solving module. The multi-convolutional layer network is used to process the input image frame and output the corresponding feature map on each convolutional layer. Each filter is connected to the corresponding convolutional layer of the multi-convolutional layer network and is used to filter the feature map output by the corresponding convolutional layer. The fusion module is connected to all filters and is used to fuse the outputs of all filters to obtain a fused feature map. The maximum value solving module is connected to the fusion module and is used to find the maximum value in the fused feature map and output the position corresponding to the maximum value as the target position. Step (3): Train the local tracker to obtain a trained local tracker; Step (4): Divide the region to be identified into multiple local regions and deploy the trained local trackers in different local regions for target tracking. During the tracking process, if the local tracker locates the target, the local tracker will follow the target and continue to track the target in subsequent frames until the target can no longer be tracked.

2. The infrared small target tracking method based on a local tracker according to claim 1, characterized in that: Step (1) specifically includes: Step (1-1): Acquire several infrared images and generate corresponding Gaussian labels; Step (1-2): Input the infrared image into the pre-trained multi-convolutional network. Each convolutional layer of the multi-convolutional network generates a feature map. Steps (1-3): Based on the Gaussian labels of the infrared images and the feature maps of each convolutional layer, the corresponding filters are generated using the following formula: , In the formula, This represents the filter corresponding to the k-th convolutional layer. Represents the discrete Fourier transform of the Gaussian label y. F () denotes the Discrete Fourier Transform. This represents the feature map output by the k-th convolutional layer. Discrete Fourier Transform, Represents the regularization parameter. Denotes the quadratic norm. This represents the matrix formed by all filters.

3. The infrared small target tracking method based on a local tracker according to claim 1, characterized in that: The filter achieves filtration using the following formula: , In the formula, This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer. () denotes the inverse discrete Fourier transform. This represents the feature map output by the k-th convolutional layer. Discrete Fourier Transform, F () represents the Discrete Fourier Transform. This represents the filter corresponding to the k-th convolutional layer.

4. The infrared small target tracking method based on a local tracker according to claim 1, characterized in that: The fusion module achieves fusion using the following formula: Q = , , , In the formula, Q Represents the fused feature map. This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer, where n represents the number of filters and KL represents the Kullback-Leibler divergence. express The element in the i-th row and j-th column, express Q The element in the i-th row and j-th column.

5. The infrared small target tracking method based on a local tracker according to claim 1, characterized in that: The filter is updated using the following formula: , In the formula, This represents the filter corresponding to the k-th convolutional layer at time t and time t-1. It is the learning rate. Indicates the number of filter updates. This indicates element-wise multiplication. Represents the discrete Fourier transform of the Gaussian label y. This represents the infrared image acquired at time t. This represents the regularization parameter.

6. An infrared small target tracking system based on a local tracker, characterized in that: The system includes multiple local trackers deployed in different local regions. These local trackers are trained and include a multi-convolutional layer network, multiple filters, a fusion module, and a maximum value calculation module. The multi-convolutional layer network processes the input image frame and outputs corresponding feature maps at each convolutional layer. Each filter is connected to the corresponding convolutional layer of the multi-convolutional layer network and filters the feature maps output by that layer. The fusion module connects to all filters and merges their outputs to obtain a fused feature map. The maximum value calculation module connects to the fusion module and calculates the maximum value in the fused feature map, outputting the position corresponding to the maximum value as the target position. The filters are generated based on the pre-trained multi-convolutional layer network. During tracking, if a local tracker locates a target, it follows the target and continues tracking it in subsequent frames until the target is no longer tracked.

7. The infrared small target tracking system based on a local tracker according to claim 6, characterized in that: The filter is generated through the following steps: Acquire several infrared images and generate corresponding Gaussian labels; The infrared image is input into a pre-trained multi-convolutional network, and each convolutional layer of the multi-convolutional network generates a feature map. The Gaussian labels based on infrared images and the feature maps of each convolutional layer are used to generate the corresponding filters using the following formula: , In the formula, This represents the filter corresponding to the k-th convolutional layer. Represents the discrete Fourier transform of the Gaussian label y. F () denotes the Discrete Fourier Transform. This represents the feature map output by the k-th convolutional layer. Discrete Fourier Transform, Represents the regularization parameter. Denotes the quadratic norm. This represents the matrix formed by all filters.

8. The infrared small target tracking system based on a local tracker according to claim 6, characterized in that: The filter achieves filtration using the following formula: , In the formula, This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer. () denotes the inverse discrete Fourier transform. This represents the feature map output by the k-th convolutional layer. Discrete Fourier Transform, F () denotes the Discrete Fourier Transform. This represents the filter corresponding to the k-th convolutional layer.

9. The infrared small target tracking system based on a local tracker according to claim 6, characterized in that: The fusion module achieves fusion using the following formula: Q = , , , In the formula, Q Represents the fused feature map. This represents the filtered feature map obtained by applying a filter to the feature map output by the k-th convolutional layer, where n represents the number of filters and KL represents the Kullback-Leibler divergence. express The element in the i-th row and j-th column, express Q The element in the i-th row and j-th column.

10. The infrared small target tracking system based on a local tracker according to claim 6, characterized in that: The filter is updated using the following formula: , In the formula, This represents the filter corresponding to the k-th convolutional layer at time t and time t-1. It is the learning rate. Indicates the number of filter updates. This indicates element-wise multiplication. Represents the discrete Fourier transform of the Gaussian label y. This represents the infrared image acquired at time t. This represents the regularization parameter.

Images