Lightweight infrared single target tracking method and device based on historical information guidance

Abstract

The application discloses a lightweight infrared single-target tracking method and device based on historical information guidance, and the method comprises the following steps: establishing a reference template corresponding to a tracking target; obtaining a search area corresponding to a current frame image based on prediction position information in a previous frame image; extracting corresponding template features and multi-level search features respectively through respective corresponding lightweight feature extraction modules; generating a similarity full-image response map, and obtaining a candidate area corresponding to the tracking target according to the similarity full-image response map; obtaining a target area, and adjusting the position of the target area through regression calculation. The multi-scale feature perception ability is maintained while the parameter quantity is reduced, and the calculation complexity is reduced. The method can be efficiently deployed on edge devices with limited computing resources, and effectively balances the tracking performance and inference efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of infrared tracking, in particular to a lightweight infrared single target tracking method and device based on historical information guidance. BACKGROUND

[0002] In the current field of computer vision, infrared single target tracking technology is widely used in intelligent monitoring, unmanned driving and other scenarios, and faces growing actual deployment needs.

[0003] In traditional schemes, the complex network structure and large number of parameters corresponding to the infrared target tracking algorithm often require high-performance computing platforms to support, making it difficult to meet the real-time and resource constraints of edge devices. With the development of Internet of Things and edge computing technology, more and more application scenarios require deploying target tracking algorithms on edge devices with limited computing resources, such as embedded systems, mobile devices and unmanned aerial vehicles. These edge devices usually have limited processor performance, memory capacity and power consumption budget. Traditional deep learning tracking algorithms often fail to meet the real-time processing requirements on such platforms, or even fail to run normally.

[0004] In the field of infrared single target tracking, due to the characteristics of low contrast, lack of texture information, and serious noise interference of infrared images, target recognition and tracking become difficult.

[0005] In traditional schemes, methods based on correlation filtering and methods based on deep learning are mainly included. Although the method based on correlation filtering has high computational efficiency, it lacks robustness in complex scenes. Although the method based on deep learning has high accuracy, the network structure is complex and the parameter amount is large, making it difficult to realize real-time tracking on resource-constrained edge devices.

[0006] Moreover, in traditional schemes, feature extraction often uses global features or fixed-level features, lacking an adaptive feature selection mechanism for different tracking scenarios. This approach not only increases the computational burden, but also may introduce irrelevant noise information, affecting tracking performance. At the same time, most lightweight methods mainly reduce model complexity through network pruning, quantization and other techniques, but at the cost of sacrificing tracking accuracy, making it difficult to achieve a good balance between accuracy and efficiency.

[0007] Therefore, there is a need for an infrared single target tracking method that can reduce computational complexity and parameter amount while ensuring tracking accuracy, to meet the actual needs of edge deployment. SUMMARY

[0008] To solve the above problems, the present application proposes a lightweight infrared single target tracking method based on historical information guidance, comprising:

[0009] acquire an infrared video, and establish a reference template corresponding to a tracking target according to a first image of the infrared video;

[0010] For a current frame image of the infrared video, a search region corresponding to the current frame image is obtained based on prediction position information in a previous frame image;

[0011] For the reference template and the search region, corresponding template features and multi-level search features are respectively extracted through respective corresponding lightweight feature extraction modules;

[0012] The feature cross-correlation between the template features and the search features is obtained through a hierarchical screening strategy, a similarity full-image response map is generated, and a candidate region corresponding to the tracking target is obtained according to the similarity full-image response map;

[0013] Based on the historical position information of the tracking target in the previous frame image, the candidate region is screened to obtain a target region, and the target region is positionally adjusted through regression calculation.

[0014] In one example, the reference template corresponding to the tracking target is established according to the first image of the infrared video, specifically including:

[0015] The first image of the infrared video is determined, and an initial region selected in the first image is determined;

[0016] The initial region is cropped, scaled to a standard resolution through bilinear interpolation, and subjected to grayscale normalization processing;

[0017] The reference template corresponding to the tracking target is obtained according to the processed initial region, and template position information corresponding to the reference template is recorded.

[0018] In one example, for the current frame image of the infrared video, the search region corresponding to the current frame image is obtained based on the prediction position in the previous frame image, specifically including:

[0019] For the current frame image of the infrared video, prediction position information in a previous frame image is determined; wherein, if the previous frame image is the first frame image, the prediction position information is the template position information;

[0020] Based on the prediction position information, the search region is obtained through expansion cropping processing according to target size information;

[0021] The search region is scaled to a standard resolution through bilinear interpolation, and a region boundary check is performed.

[0022] In one example, for the baseline template and the search region, corresponding template features and multi-level search features are extracted using their respective lightweight feature extraction modules, specifically including:

[0023] A first lightweight feature extraction module is determined for the benchmark template, and a second lightweight feature extraction module is determined for the search region;

[0024] The first lightweight feature extraction module extracts the general template features corresponding to the benchmark template, and the second lightweight feature extraction module extracts the multi-level general search features corresponding to the search region.

[0025] By performing feature processing on the general template features through category branching and regression branching, task-specific template features are obtained. Similarly, feature processing is performed on the multi-level general search features to obtain multi-level task-specific search features.

[0026] In one example, both the first lightweight feature extraction module and the second lightweight feature extraction module include a multi-branch residual convolution module and an adaptive feature enhancement module; the second lightweight feature extraction module includes a multi-level structure, which can output infrared image features of various resolutions in a hierarchical manner.

[0027] In one example, a hierarchical filtering strategy is used to obtain the feature cross-correlation between the template features and the search features, generating a full similarity response map, specifically including:

[0028] Based on the multi-level task-specific search features, the resolutions of each level are sorted from low to high.

[0029] Based on the sorting, the similarity full-image response map corresponding to the search area of ​​the corresponding level is determined sequentially according to the task-specific template features and the task-specific search features corresponding to the level in the sorting. The search area is then refined according to the similarity full-image response map, and the refined search area is used as the search area corresponding to the next level, until the similarity full-image response map corresponding to the highest level is used. The similarity full-image response map includes a category response map and a regression response map.

[0030] In one example, based on the historical location information of the tracked target in the previous frame image, the candidate regions are filtered to obtain the target region, and the position of the target region is adjusted through regression calculation, specifically including:

[0031] A candidate region token is constructed based on the candidate region, a historical query token is constructed based on the historical location information of the tracked target in the previous frame image, a target location prediction token is constructed based on the region where the historical query token is located and the adjacent regions, and a context token is constructed based on the auxiliary features of the previous frame image.

[0032] The candidate region token, the historical query token, the target location prediction token, and the context token are fused to generate a fused sequence, and modeled based on a lightweight Transformer architecture and a self-attention mechanism.

[0033] Based on the established tracker module, the existence result of the tracked target corresponding to the candidate region is returned, the target region corresponding to the current frame image is determined, and the absolute coordinate position of the tracked target is regressed so as to adjust the position of the target region through the absolute coordinate position.

[0034] In one example, the training process of the lightweight feature extraction module includes:

[0035] For the first stage of training, the lightweight feature extraction module is trained until it converges, and the corresponding initial pre-training weights are obtained.

[0036] For the second stage of training, the initial pre-trained weights are frozen as prior knowledge, and the learnable weight increments that are isomorphic to the initial pre-trained weights are dynamically optimized by residual summation. Finally, the initial pre-trained weights and the learnable weight increments are fused to obtain the final pre-trained weights.

[0037] In one example, the training process of the lightweight feature extraction module further includes:

[0038] The final pre-trained weights obtained by training the lightweight feature extraction module are then fused with the convolutional layer and batch normalization layer in the lightweight feature extraction module to obtain the fused convolutional weights and convolutional biases.

[0039] The inference process of the lightweight feature extraction module also includes:

[0040] Based on the fused convolution weights corresponding to each branch structure in the multi-branch residual convolution module, the branch structures in the multi-branch residual convolution module are fused through branch alignment, branch splicing, and dimensionality reduction transformation to obtain a single-branch structure.

[0041] On the other hand, this application also proposes a lightweight infrared single-target tracking device guided by historical information, comprising:

[0042] At least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform a lightweight infrared single-target tracking method based on historical information as described in any of the above examples.

[0045] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions configured to implement a lightweight infrared single-target tracking method based on historical information as described in any of the above examples.

[0046] The lightweight infrared single-target tracking method based on historical information proposed in this application can bring the following benefits:

[0047] By employing a lightweight feature extraction module and a multi-level search feature structure, the system maintains multi-scale feature perception capabilities while reducing the number of parameters, thus lowering computational complexity. A hierarchical filtering strategy is introduced to perform coarse-to-fine feature cross-correlation calculations, effectively focusing on potential target regions, reducing redundant computations, and improving tracking efficiency. By combining historical location information to construct a lightweight tracker, robustness and continuous tracking capabilities are enhanced under challenges such as target occlusion, deformation, and complex backgrounds. This allows for efficient deployment on edge devices with limited computing resources, achieving an effective balance between tracking performance and inference efficiency. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a flowchart illustrating the lightweight infrared single-target tracking method based on historical information in the embodiments of this application.

[0050] Figure 2 This is a flowchart illustrating a lightweight infrared single-target tracking method guided by historical information, as described in one embodiment of this application.

[0051] Figure 3 This is a schematic diagram illustrating the application framework of a lightweight infrared single-target tracking method guided by historical information in one scenario of this application embodiment.

[0052] Figure 4 This is a schematic diagram of a 5×5 multi-branch residual convolution module in one scenario of this application embodiment;

[0053] Figure 5 This is a schematic diagram of a 3×3 multi-branch residual convolution module in one scenario of this application embodiment;

[0054] Figure 6 This is a schematic diagram of the adaptive feature enhancement module in one scenario of an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the tracker module in one scenario of an embodiment of this application;

[0056] Figure 8 This is a schematic diagram of the Transformer layer in the tracker module under one scenario in an embodiment of this application;

[0057] Figure 9 This is a schematic diagram illustrating reparameterization in one scenario of an embodiment of this application;

[0058] Figure 10 This is a schematic diagram of a lightweight infrared single-target tracking device guided by historical information in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0061] Developing lightweight target tracking algorithms has become a current research hotspot and challenge.

[0062] like Figure 1 and Figure 2 As shown, this application provides a lightweight infrared single-target tracking method guided by historical information, including:

[0063] S101: Acquire infrared video and establish a reference template corresponding to the tracking target based on the first frame image of the infrared video.

[0064] Infrared video can be captured by corresponding infrared camera equipment, from which the first frame image can be extracted for the initialization process of the reference template.

[0065] Specifically, the first frame of the infrared video is determined, and an initial region is selected within that first frame. This initial region can be labeled by researchers during model training, or it can be directly inferred during inference.

[0066] The initial region is cropped and scaled to a standard resolution using bilinear interpolation. The standard resolution can be 127×127 pixels. Grayscale normalization is then performed to map the grayscale data range to [0,1].

[0067] Based on the processed initial region, a reference template corresponding to the tracking target is obtained, and the template position information corresponding to the reference template is recorded. At this point, the reference template can be represented as... Where T is the reference template, R is the real number space, and the recorded template position information is the center coordinate of the initial region. and width and height .

[0068] S102: For the current frame image of the infrared video, based on the predicted position information in the previous frame image, obtain the search area corresponding to the current frame image.

[0069] The search area represents the range within which the target is tracked in the current frame image.

[0070] Specifically, such as Figure 3 As shown, for the current frame of an infrared video, the predicted position information in the previous frame is determined; where, if the previous frame is the first frame, the predicted position information is the template position information. In this case, the predicted position information in the previous frame can be represented by the center coordinates.

[0071] Based on the predicted location information, the search area is obtained by expanding and cropping according to the target size information. For example, cropping the target size information by 2 times results in a square search area.

[0072] The search area is scaled to a standard resolution using bilinear interpolation, and region boundary checks are performed. At this point, the standard resolution can be 255×255 pixels, and the search area can be represented as... Where S is the search region and R is the real number space. Boundary checks and out-of-bounds handling are performed simultaneously to ensure the integrity of the search region.

[0073] S103: For the baseline template and the search region, the corresponding template features and multi-level search features are extracted respectively through their respective lightweight feature extraction modules.

[0074] Base template and search area The synchronous input lightweight network processing flow extracts features through their respective lightweight feature extraction modules.

[0075] Specifically, a first lightweight feature extraction module is determined for the benchmark template. And a second lightweight feature extraction module for the search region. .

[0076] Among them, the first lightweight feature extraction module Second lightweight feature extraction module Each module includes a multi-branch residual convolution module and an adaptive feature enhancement module. For example, it consists of a 5×5 and a 3×3 multi-branch residual convolution module and an adaptive feature enhancement module.

[0077] Second lightweight feature extraction module It includes a multi-level structure, capable of outputting infrared image features at various resolutions in a hierarchical manner. For example, it can be configured to output infrared image features at three different resolutions. , and These correspond to low, medium, and high resolution features, respectively.

[0078] The first lightweight feature extraction module extracts the general template features corresponding to the benchmark template. ,in, For general template features, The baseline template is processed using the first lightweight feature extraction module. The second lightweight feature extraction module extracts multi-level general search features corresponding to the search region. ,in, It is a multi-level general search feature. , and These correspond to different levels, and respectively to low, medium, and high resolution features. The search region is processed by the second lightweight feature extraction module.

[0079] like Figure 4 As shown, in the 5×5 multi-branch residual convolution module, the input features are processed by convolutional layers 5×5, 5×1, 5×5, 1×5, and 1×1 respectively, then concatenated by the corresponding batch normalization layer, and then input into the convolutional layer 1×1 for convolutional processing.

[0080] like Figure 5As shown, in the 3×3 multi-branch residual convolution module, the input features are processed by convolutional layers 3×3, 3×1, 1×3 and 1×1 respectively, then concatenated by the corresponding batch normalization layer, and then input into the convolutional layer 1×1 for convolutional processing.

[0081] like Figure 6 As shown, in the adaptive feature enhancement module, the input features are scaled by learnable parameter weight 1 and learnable parameter weight 2 respectively to obtain weighted feature 1 and weighted feature 2. Through feature multiplication, and combined with the bias parameters corresponding to the learnable parameters, the final output features are obtained, which are the corresponding general template features and multi-level general search features respectively.

[0082] By category branch Regression branch For general template features Perform feature processing to obtain task-specific template features. and general search features for multiple levels. Feature processing is performed to obtain multi-level task-specific search features. .

[0083] The category branch consists of several convolutional layers to classify the target and the background, while the regression branch consists of several convolutional layers to perform bounding box regression. For task-specific template features, Specific search features for multi-level tasks; These are task-specific template features obtained by processing general template features through categorical branch and regression branch features, respectively. Chalk is a task-specific search feature obtained by processing general search features through category branching and regression branching features.

[0084] S104: Obtain the feature cross-correlation between the template features and the search features through a hierarchical filtering strategy, generate a similarity full-map response map, and obtain the candidate region corresponding to the tracking target based on the similarity full-map response map.

[0085] Specifically, based on the multi-level task-specific search features, the resolutions of each level are sorted from low to high. For example, a total of [number] features are included. , and These correspond to three levels: low, medium, and high.

[0086] Based on the ranking, the similarity full-image response map corresponding to the search area of ​​the task-specific template features and the task-specific search features corresponding to the level in the ranking is determined in turn. The search area is then refined based on the similarity full-image response map, and the refined search area is used as the search area corresponding to the next level, until the similarity full-image response map corresponding to the highest level is used. The similarity full-image response map includes a category response map and a regression response map.

[0087] Using a hierarchical filtering strategy on the feature dimension to calculate task-specific template features and task-specific search features The cross-correlation of features between them. Taking low, medium, and high resolution levels as an example, the category branches are first calculated at the low resolution level. and regression branch The similarity of the full-image response map is used to obtain the coarse localization region. Among them, and They are respectively In The corresponding feature representation in the low-resolution level, similarly, and They are respectively In The corresponding feature representation at the medium resolution level, and, and They are respectively In The corresponding feature representation at the high-resolution level. The same task-specific template features are used to calculate feature cross-correlation. Therefore, it will not be elaborated further here. The similarity full-map response map reflects a heatmap of the target location, which can be used to refine the search area.

[0088] Next, the search region at the medium resolution level is cropped based on the coarse localization results. The category branch is then calculated within the cropped medium resolution search region. and regression branch The similarity full-map response map is used to further refine the localization region.

[0089] Finally, the search region at the high-resolution level is cropped based on the refined localization results. Category branches are then calculated within the cropped high-resolution search region. and regression branch The final result will be a full-image similarity response map. and ,in, For category response maps and This is a regression response plot.

[0090] S105: Based on the historical position information of the tracked target in the previous frame image, the candidate regions are filtered to obtain the target region, and the position of the target region is adjusted by regression calculation.

[0091] To improve the robustness of target tracking across consecutive frames, a tracker module incorporating historical target information is introduced. This module enables the association modeling of target features in the similarity full-image response map and improves target localization accuracy guided by historical tracking information. As an extension of the aforementioned feature-level filtering framework, this tracker module is embedded after the baseline template and search region features are aligned (i.e., acquiring task-specific template features and task-specific search posts) and before the final prediction of the target object's position. This effectively enhances recovery capabilities under occlusion and stability under scene changes.

[0092] Specifically, such as Figure 7 As shown, a candidate region token is constructed based on the candidate region, a historical query token is constructed based on the historical location information of the tracked target in the previous frame image, a target location prediction token is constructed based on the region where the historical query token is located and the adjacent regions, and a context token is constructed based on the auxiliary features of the previous frame image.

[0093] At different resolutions, through the final category response map The top-k feature locations with the largest localization response values ​​are considered as candidate regions where the target is most likely to exist in the current frame.

[0094] For each candidate region, a candidate region token, also known as a candidate token, is constructed, and its vector is recorded as [B, N1, C]. High-resolution feature blocks within the center and surrounding 3×3 neighborhood are extracted from the multi-level branch features of the search region. These blocks are cropped from the first 64 feature maps of the original channel dimension, and each feature block is flattened into a single-channel feature vector of size 64×9.

[0095] The historical query token, also known as the historical query token, is constructed and recorded as a vector [B, N2, C]. In the historical information, it is obtained by extracting the center position of the tracked target and its features within a 3×3 neighborhood in the previous frame image. It is also expanded into a 64×9 single-channel feature, and together with its absolute position encoding on the feature map, it constitutes the historical query token, which serves as a reference vector for matching the target region in the current frame image.

[0096] A target location prediction token and a context token are constructed, referred to as the target location prediction token and the context token, respectively, and denoted as [B,N3,C] and [B,N4,C]. The target location prediction token is composed of the target location and location information (absolute coordinate encoding) in the previous frame image, as well as neighborhood features, and is mainly used for target relocalization when occlusion occurs. The context token is an auxiliary feature for non-target regions, directly inherited from the context token set of the previous frame, and is used to guide the context modeling of the Transformer in the current frame. The context token does not participate in target regression, but it plays an auxiliary role in maintaining tracking continuity.

[0097] like Figure 7 As shown, candidate region tokens, historical query tokens, target location prediction tokens, and context tokens are fused to generate a fused sequence, which is then modeled based on a lightweight Transformer architecture and a self-attention mechanism.

[0098] Self-attention modeling and token-level fusion based on lightweight Transformer are performed. The candidate token, target history query token, target location prediction token and context token constructed above are fused into a unified fusion sequence, and learnable type encoding is added. The sequence is then input into the lightweight self-attention Transformer module for multi-head attention modeling.

[0099] In the lightweight Transformer architecture, feature processing is performed through learnable type encoding and corresponding layers (including lightweight Transformer Block1 to lightweight Transformer Block3 and the final normalization layer), and token sequence segmentation is performed. At this point, it is already an enhanced fusion sequence after feature processing, resulting in enhanced candidate tokens, enhanced target history query tokens, enhanced target location prediction tokens, and enhanced context tokens.

[0100] At the same time, such as Figure 8As shown, in each of the lightweight Transformer Blocks 1 through 3, the input token sequence is dimensionality reduced to C / r through Q-projection and K-projection respectively, resulting in multi-head reshaping Q and K. Attention calculation, normalized Softmax, and regularized Dropout are then performed. The C-dimensionality is preserved through V-projection to obtain multi-head reshaping V. Here, Q, K, and V refer to the query vector, key vector, and value vector, respectively, r is the dimensionality reduction ratio, and multi-head reshaping Q, multi-head reshaping K, and multi-head reshaping V correspond to the vectors after splitting and reshaping.

[0101] The computation results after normalized Softmax and regularized Dropout are used to perform attention calculations with the multi-head reshaping V. After multi-head merging (concatenating the outputs of multiple heads) and multi-branch residual convolution (convolution is good at processing local spatial information, which complements the global modeling ability of the attention mechanism, thereby introducing local priors and further enhancing feature expression), the output projection is connected with the residual of the input token sequence and directly added to alleviate the gradient vanishing problem in deep networks and output the final vector.

[0102] Based on the established tracker module, the existence result of the tracking target corresponding to the candidate region is returned, the target region corresponding to the current frame image is determined, and the absolute coordinate position of the tracking target is regressed so as to adjust the position of the target region through the absolute coordinate position.

[0103] Through this mechanism, the established tracker module can dynamically capture the matching relationship between candidate regions and historical target locations, and adjust the final prediction of the current frame based on the context. In the Transformer's output tokens for prediction and target regression strategies, candidate tokens are regressed to target existence (1 for target, 0 for non-target); the target location prediction token simultaneously regresses the target's absolute coordinates. Among them, the coordinates of the center of the detection box in the target area are respectively used to adjust the center coordinates of the detection box in the target area. and width and height .

[0104] Specifically, the center coordinate correction is based on the coordinates of the feature map position mapped to the original image, plus the offset from the regression prediction. ; ;in, , These are the center coordinates of the adjusted target area. The width and height are adjusted by multiplying the initial target size by the regression prediction scaling factor, i.e. ; ;in, , The adjusted width and height of the target area. , The dimensions are the width and height of the target area before adjustment.

[0105] Especially in occluded scenarios, this token output is prioritized as a supplement or alternative to the target location information. The context token is then directly passed to the next frame to continue participating in modeling, achieving information inheritance and continuous modeling.

[0106] For occlusion detection and target recovery, occlusion is considered to have occurred when the target response drops significantly for several consecutive frames and the confidence of the predicted target position token is higher than that of the candidate token. In this case, target recovery is performed based on the predicted target position token. For scene switching and target disappearance handling, dynamic updates of the search box in the search area and consistency checks of the context token are used to determine whether a new scene has been entered or the target has disappeared. The search area is kept stable and the target template update is delayed until a new target is confirmed.

[0107] By employing a lightweight feature extraction module and a multi-level search feature structure, the system maintains multi-scale feature perception capabilities while reducing the number of parameters, thus lowering computational complexity. A hierarchical filtering strategy is introduced to perform coarse-to-fine feature cross-correlation calculations, effectively focusing on potential target regions, reducing redundant computations, and improving tracking efficiency. By combining historical location information to construct a lightweight tracker, robustness and continuous tracking capabilities are enhanced under challenges such as target occlusion, deformation, and complex backgrounds. This allows for efficient deployment on edge devices with limited computing resources, achieving an effective balance between tracking performance and inference efficiency.

[0108] In one embodiment, the training process of the lightweight feature extraction module includes:

[0109] For the first stage of training, the lightweight feature extraction module is trained until it converges, yielding the corresponding initial pre-trained weights. At this point, the first lightweight feature extraction module... Second lightweight feature extraction module Category branches Regression branch Perform regular training until convergence to obtain the initial pre-trained weights. .

[0110] For the second phase of training, the initial pre-trained weights are frozen. As prior knowledge, learnable weight increments are introduced. And by summing the residuals, the learnable weight increments that are isomorphic to the initial pre-trained weights are calculated. Dynamic optimization is performed, and the final pre-trained weights are obtained by fusing the initial pre-trained weights and the learnable weight increments. The results obtained from the first phase of training The value is fixed as a constant to avoid being overwritten in the second stage of training, ensuring the stability of prior knowledge. In the second stage of training, only gradient updates are performed on ΔW, reducing the degrees of freedom in optimization, mitigating the risk of overfitting, and preserving the ability to learn high-frequency details through the residual path.

[0111] Furthermore, such as Figure 9 As shown, reparameterization is used to further reduce the computational cost of the model. By leveraging the linear additivity and homogeneity of convolution operations, the complex structure of multi-branch, multi-batch normalization is equivalently fused into a single convolution kernel and bias, achieving "multi-branch training and single-branch inference".

[0112] At this point, the training process of the lightweight feature extraction module also includes:

[0113] The final pre-trained weights obtained by training the lightweight feature extraction module are then fused with the convolutional layers and batch normalization layers in the lightweight feature extraction module to obtain the fused convolutional weights and convolutional biases.

[0114] Extracting statistical information obtained during the training phase from the batch normalization layer. Including running average and operating variance They record the mean and variance statistics of features during training, respectively, and are used for normalization during the inference phase; It is a small constant (for example, set to 1e-5) used for numerical stability; and These are two learnable parameters that control the scaling and translation of the normalized features, respectively.

[0115] Calculate the standardized coefficient and offset coefficient Next, the one-dimensional coefficient k is expanded into a four-dimensional tensor through a tensor dimension expansion operation to match the dimension of the convolution weights. This process is repeated sequentially with the original convolution weights... (Here you can choose the final pre-trained weights from the above text) Multiply by the standardized factor and batch-normalized learnable scaling parameters This yields the fused convolutional weights (referred to as fused convolutional weights). Simultaneously, similar processing is applied to the bias term, first adjusting the original convolution bias... Multiply And add the offset coefficient Then multiply by the batch-normalized learnable scaling parameter And add offset parameters Finally, the fused convolutional bias is obtained (referred to as the fused convolutional bias). .

[0116] Here Examples of multi-branch residual convolution modules of varying sizes:

[0117] During the training phase, parallel The system consists of four branches: convolutional layers with batch normalization, 1×1 convolutional layers with batch normalization, 3×1 convolutional layers with batch normalization, and 1×3 convolutional layers with batch normalization. Finally, a residual method is used to concatenate the features from the convolutional layers alone and those from the convolutional layers with batch normalization, focusing on channel dimension, to obtain multi-branch features. A 1×1 convolution is then used to reduce the dimensionality of these multi-branch features.

[0118] The inference process of the lightweight feature extraction module also includes:

[0119] Based on the fused convolution weights corresponding to each branch structure in the multi-branch residual convolution module, the branch structures in the multi-branch residual convolution module are fused through branch alignment, branch splicing, and dimensionality reduction transformation to obtain a single-branch structure.

[0120] Still with Examples of multi-branch residual convolution modules of varying sizes:

[0121] During the inference phase: The kernels of the 1×1, 3×1, and 1×3 convolutions in each branch are zero-padded and expanded to 3×3 to achieve branch alignment; then, the convolution kernels of each branch are fused with the batch-normalized parameters to obtain the fused convolution weights. and fusion convolution bias The original convolution weights of all branches will be used in a residual manner. With fused convolutional weights and the original convolution bias With fusion convolution bias By concatenating the concatenated convolutional weights along the channel dimensions, the concatenated convolutional weights can be obtained. With splicing convolution bias To achieve branch concatenation; using learnable parameters of a 1×1 convolution kernel. (Weights of the 1×1 convolution kernel) and (Bias of 1×1 convolution kernel) for concatenated convolution weights With splicing convolution bias Dimensionality reduction, i.e. , ;in, These are the equivalent convolution kernel weights after reparameterization. The equivalent convolution kernel bias is set after reparameterization. Finally, during the model inference stage, the 3×3 multi-branch residual convolution module is reparameterized into a 3×3 convolution kernel that is computationally friendly to edge terminals (that is, the equivalent convolution kernel after reparameterization), thus realizing the generation of a single-branch structure.

[0122] like Figure 10 As shown in the illustration, this application also provides a lightweight infrared single-target tracking device guided by historical information, comprising:

[0123] At least one processor; and,

[0124] A memory communicatively connected to the at least one processor; wherein,

[0125] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the lightweight infrared single-target tracking method based on historical information as described in any of the above embodiments.

[0126] This application also provides a non-volatile computer storage medium storing computer-executable instructions configured to implement the lightweight infrared single-target tracking method based on historical information as described in any of the above embodiments.

[0127] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0128] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A lightweight infrared single-target tracking method based on historical information guidance, characterized in that, include: Infrared video is acquired, and a reference template corresponding to the tracking target is established based on the first frame image of the infrared video. For the current frame image of the infrared video, the search area corresponding to the current frame image is obtained based on the predicted position information in the previous frame image; For the baseline template and the search region, corresponding template features and multi-level search features are extracted using their respective lightweight feature extraction modules; specifically including: A first lightweight feature extraction module is defined for the baseline template, and a second lightweight feature extraction module is defined for the search region. The first lightweight feature extraction module extracts general template features corresponding to the baseline template, and the second lightweight feature extraction module extracts multi-level general search features corresponding to the search region. Through category branching and regression branching, feature processing is performed on the general template features to obtain task-specific template features, and feature processing is performed on the multi-level general search features to obtain multi-level task-specific search features. Both the first and second lightweight feature extraction modules include a multi-branch residual convolution module and an adaptive feature enhancement module. The second lightweight feature extraction module includes a multi-level structure, capable of hierarchically outputting infrared image features at various resolutions. The process involves obtaining the cross-correlation between the template features and the search features through a hierarchical filtering strategy, generating a full similarity response map, and obtaining the candidate region corresponding to the tracked target based on the full similarity response map. Specifically, this includes: sorting the resolution of each level of the multi-level task-specific search features from low to high; based on the sorting, determining the full similarity response map corresponding to the search region for each level according to the task-specific template features and the task-specific search features corresponding to that level in the sorting; refining the search region based on the full similarity response map; and using the refined search region as the search region for the next level, until the highest level's full similarity response map is used; wherein, the full similarity response map includes a category response map and a regression response map. Based on the historical position information of the tracked target in the previous frame image, the candidate regions are filtered to obtain the target region, and the position of the target region is adjusted through regression calculation. Specifically, this includes: constructing candidate region tokens based on the candidate regions, constructing historical query tokens based on the historical position information of the tracked target in the previous frame image, constructing target position prediction tokens based on the region where the historical query token is located and adjacent regions, and constructing context tokens based on auxiliary features of the previous frame image; fusing the candidate region tokens, the historical query tokens, the target position prediction tokens, and the context tokens to generate a fusion sequence, and modeling it based on a lightweight Transformer architecture and a self-attention mechanism; based on the established tracker module, returning the existence result of the tracked target corresponding to the candidate region, determining the target region corresponding to the current frame image, and regressing the absolute coordinate position of the tracked target, so as to adjust the position of the target region through the absolute coordinate position.

2. The method according to claim 1, characterized in that, Based on the first frame of the infrared video, a reference template corresponding to the tracking target is established, specifically including: Determine the first frame of the infrared video and determine the initial region selected in the first frame; The initial region is cropped, and then scaled to standard resolution using bilinear interpolation and grayscale normalization is performed. The reference template corresponding to the tracking target is obtained from the processed initial region, and the template position information corresponding to the reference template is recorded.

3. The method according to claim 2, characterized in that, For the current frame of the infrared video, based on the predicted position in the previous frame, the search region corresponding to the current frame is obtained, specifically including: For the current frame image of the infrared video, the predicted position information in the previous frame image is determined; wherein, if the previous frame image is the first frame image, the predicted position information is the template position information; Based on the predicted location information, the target size information is used to perform an enlargement and cropping process to obtain the search area; The search area is scaled to standard resolution using bilinear interpolation, and region boundary checks are performed.

4. The method according to claim 1, characterized in that, The training process of the lightweight feature extraction module includes: For the first stage of training, the lightweight feature extraction module is trained until it converges, and the corresponding initial pre-training weights are obtained. For the second stage of training, the initial pre-trained weights are frozen as prior knowledge, and the learnable weight increments that are isomorphic to the initial pre-trained weights are dynamically optimized by residual summation. Finally, the initial pre-trained weights and the learnable weight increments are fused to obtain the final pre-trained weights.

5. The method according to claim 4, characterized in that, The training process of the lightweight feature extraction module also includes: The final pre-trained weights obtained by training the lightweight feature extraction module are then fused with the convolutional layer and batch normalization layer in the lightweight feature extraction module to obtain the fused convolutional weights and convolutional biases. The inference process of the lightweight feature extraction module also includes: Based on the fused convolution weights corresponding to each branch structure in the multi-branch residual convolution module, the branch structures in the multi-branch residual convolution module are fused through branch alignment, branch splicing, and dimensionality reduction transformation to obtain a single-branch structure.

6. A lightweight infrared single-target tracking device guided by historical information, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the lightweight infrared single-target tracking method based on historical information as described in any one of claims 1 to 5.

Images