Lightweight infrared single target tracking method and device based on historical information guidance

Abstract

The invention discloses a lightweight infrared single target tracking method and device based on historical information guidance. The method comprises the following steps: establishing a reference template corresponding to a tracking target; obtaining a search area corresponding to the current frame image based on the predicted position information in the previous frame image; respectively extracting corresponding template features and multi-level search features through respective corresponding lightweight feature extraction modules; generating a similarity full-graph response graph, and obtaining a candidate region corresponding to the tracking target according to the similarity full-graph response graph; and obtaining a target area, and adjusting the position of the target area through regression calculation. The multi-scale feature perception capability is maintained while the parameter quantity is reduced, and the calculation complexity is reduced. The method can be efficiently deployed in edge equipment with limited computing resources, and effective balance between tracking performance and reasoning efficiency is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of infrared tracking, in particular to a lightweight infrared single target tracking method and device based on historical information guidance. BACKGROUND

[0002] In the current field of computer vision, infrared single target tracking technology is widely used in intelligent monitoring, unmanned driving and other scenarios, and faces growing actual deployment needs.

[0003] In traditional schemes, the complex network structure and large number of parameters corresponding to the infrared target tracking algorithm often require high-performance computing platforms to support, making it difficult to meet the real-time and resource constraints of edge devices. With the development of Internet of Things and edge computing technology, more and more application scenarios require deploying target tracking algorithms on edge devices with limited computing resources, such as embedded systems, mobile devices and unmanned aerial vehicles. These edge devices usually have limited processor performance, memory capacity and power consumption budget. Traditional deep learning tracking algorithms often fail to meet real-time processing requirements on such platforms, or even fail to run normally.

[0004] In the field of infrared single target tracking, due to the characteristics of low contrast, lack of texture information, and serious noise interference of infrared images, target recognition and tracking become difficult.

[0005] In traditional schemes, methods based on correlation filtering and methods based on deep learning are mainly included. Although the method based on correlation filtering has high computational efficiency, it lacks robustness in complex scenes; while the method based on deep learning has high accuracy, but the network structure is complex and the parameter amount is large, making it difficult to realize real-time tracking on resource-constrained edge devices.

[0006] Moreover, in traditional schemes, feature extraction often uses global features or fixed-level features, lacking an adaptive feature selection mechanism for different tracking scenarios. This approach not only increases the computational burden, but also may introduce irrelevant noise information, affecting tracking performance. At the same time, most lightweight methods mainly reduce model complexity through network pruning, quantization and other techniques, but at the cost of sacrificing tracking accuracy, making it difficult to achieve a good balance between accuracy and efficiency.

[0007] Therefore, there is a need for an infrared single target tracking method that can reduce computational complexity and parameter amount while ensuring tracking accuracy, to meet the actual needs of edge deployment. SUMMARY

[0008] To solve the above problems, the present application proposes a lightweight infrared single target tracking method based on historical information guidance, comprising: acquire an infrared video, and establish a reference template corresponding to a tracking target according to a first frame image of the infrared video; For a current frame image of the infrared video, a search region corresponding to the current frame image is obtained based on prediction position information in a previous frame image; For the reference template and the search region, corresponding template features and multi-level search features are respectively extracted through respective corresponding lightweight feature extraction modules; The feature cross-correlation between the template features and the search features is obtained through a hierarchical screening strategy, a similarity full-map response map is generated, and a candidate region corresponding to the tracking target is obtained according to the similarity full-map response map; Based on the historical position information of the tracking target in the previous frame image, the candidate region is screened to obtain a target region, and the target region is positionally adjusted through regression calculation.

[0009] In one example, according to the first frame image of the infrared video, a reference template corresponding to a tracking target is established, specifically including: The first frame image of the infrared video is determined, and an initial region selected in the first frame image is determined; The initial region is cropped, and the initial region is scaled to a standard resolution through bilinear interpolation, and grayscale normalization processing is performed; A reference template corresponding to a tracking target is obtained according to the processed initial region, and template position information corresponding to the reference template is recorded.

[0010] In one example, for a current frame image of the infrared video, a search region corresponding to the current frame image is obtained based on a prediction position in a previous frame image, specifically including: For the current frame image of the infrared video, prediction position information in a previous frame image is determined; wherein, if the previous frame image is the first frame image, the prediction position information is the template position information; Based on the prediction position information, the search region is obtained through expansion cropping processing according to target size information; The search region is scaled to a standard resolution through bilinear interpolation, and a region boundary check is performed.

[0011] In one example, for the reference template and the search region, corresponding template features and multi-level search features are respectively extracted through respective corresponding lightweight feature extraction modules, specifically including: A first lightweight feature extraction module for the reference template and a second lightweight feature extraction module for the search region are determined; extracting general template features corresponding to the reference template through the first lightweight feature extraction module, and extracting multi-level general search features corresponding to the search region through the second lightweight feature extraction module; processing the general template features through the category branch and the regression branch to obtain task-specific template features, and processing the multi-level general search features to obtain multi-level task-specific search features.

[0012] In one example, the first lightweight feature extraction module and the second lightweight feature extraction module each include a multi-branch residual convolution module and an adaptive feature enhancement module; the second lightweight feature extraction module includes a multi-level structure and can output infrared image features of multiple resolution sizes in a hierarchical manner.

[0013] In one example, the feature cross-correlation between the template features and the search features is obtained through a hierarchical screening strategy to generate a similarity full-map response map, specifically including: According to the multi-level task-specific search features, the resolution corresponding to each level is sorted from low to high; Based on the sorting, the task-specific template features and the task-specific search features corresponding to the level in the sorting are sequentially used to determine the similarity full-map response map of the level corresponding to the search region, and the search region is refined according to the similarity full-map response map, and the refined search region is used as the search region corresponding to the next level until the similarity full-map response map corresponding to the highest level is used; wherein the similarity full-map response map includes a category response map and a regression response map.

[0014] In one example, based on the historical position information of the tracking target in the previous frame of image, the candidate region is screened to obtain a target region, and the target region is positionally adjusted through regression calculation, specifically including: According to the candidate region, a candidate region token is constructed, according to the historical position information of the tracking target in the previous frame of image, a historical query token is constructed, according to the region and adjacent region where the historical query token is located, a target position prediction token is constructed, and according to the auxiliary features of the previous frame of image, a context token is constructed; The candidate region token, the historical query token, the target position prediction token, and the context token are fused to generate a fused sequence, and a lightweight Transformer architecture and a self-attention mechanism are modeled based on the fused sequence; Based on the established tracker module, the existence result of the tracking target corresponding to the candidate region is returned, the target region corresponding to the current frame image is determined, and the absolute coordinate position of the tracking target is regressed, so as to adjust the position of the target region through the absolute coordinate position.

[0015] In one example, the training process of the lightweight feature extraction module includes: For the first stage training, the lightweight feature extraction module is trained until the lightweight feature extraction module converges, and the corresponding initial pre-training weight is obtained; For the second stage training, the initial pre-training weight is frozen as prior knowledge, the learnable weight increment isomorphism with the initial pre-training weight is dynamically optimized through the residual sum, and the final pre-training weight is obtained through the initial pre-training weight and the learnable weight increment.

[0016] In one example, the training process of the lightweight feature extraction module further includes: The final pre-training weight obtained by training the lightweight feature extraction module is fused with the convolution layer and the batch normalization layer in the lightweight feature extraction module to obtain the fused convolution weight and the convolution bias; The inference process of the lightweight feature extraction module further includes: According to the fused convolution weight corresponding to each branch structure in the multi-branch residual convolution module, each branch structure in the multi-branch residual convolution module is fused through branch alignment, branch splicing and dimension reduction transformation to obtain a single branch structure.

[0017] On the other hand, the application also proposes a lightweight infrared single target tracking device based on historical information guidance, which includes: At least one processor; and, The memory is in communication connection with the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the lightweight infrared single target tracking method based on historical information guidance as described in any of the above examples.

[0018] On the other hand, the application also proposes a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to implement the lightweight infrared single target tracking method based on historical information guidance as described in any of the above examples.

[0019] The lightweight infrared single target tracking method based on historical information guidance proposed in the application can bring the following beneficial effects: By means of the lightweight feature extraction module and the multi-level search feature structure, the multi-scale feature perception ability is maintained while the parameter quantity is reduced and the calculation complexity is reduced. The hierarchical screening strategy is introduced to perform coarse-to-fine feature cross-correlation calculation, effectively focusing on the potential target area, reducing redundant calculation and improving tracking efficiency. By combining the historical position information to construct a lightweight tracker, the robustness and continuous tracking ability under the challenges of target occlusion, deformation and complex background are enhanced, which can be efficiently deployed on edge devices with limited computing resources, achieving an effective balance between tracking performance and inference efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application. In the drawings: Figure 1 A flowchart of a lightweight infrared single-target tracking method based on historical information guidance in an embodiment of the application; Figure 2 A flowchart of a lightweight infrared single-target tracking method based on historical information guidance in an embodiment of the application under one condition; Figure 3 An application framework diagram of a lightweight infrared single-target tracking method based on historical information guidance in an embodiment of the application under one condition; Figure 4 A diagram of a 5x5 multi-branch residual convolution module in an embodiment of the application under one condition; Figure 5 A diagram of a 3x3 multi-branch residual convolution module in an embodiment of the application under one condition; Figure 6 A diagram of an adaptive feature enhancement module in an embodiment of the application under one condition; Figure 7 A diagram of a tracker module in an embodiment of the application under one condition; Figure 8 A diagram of a Transformer layer in a tracker module in an embodiment of the application under one condition; Figure 9 A diagram of reparameterization in an embodiment of the application under one condition; Figure 10 A diagram of a lightweight infrared single-target tracking device based on historical information guidance in an embodiment of the application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0022] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0023] Developing a lightweight target tracking algorithm has become a hot and difficult point of current research.

[0024] As shown in Figure 1 and Figure 2 , the embodiments of the present application provide a lightweight infrared single target tracking method based on historical information guidance, which includes: S101: Collecting an infrared video, and establishing a reference template corresponding to a tracking target according to a first frame image of the infrared video.

[0025] The infrared video can be collected by a corresponding infrared camera device, and the first frame image is extracted therefrom to perform the initialization process of the reference template.

[0026] Specifically, the first frame image of the infrared video is determined, and an initial region selected in the first frame image is determined. The selection of the initial region can be annotated by a developer in the model training stage, and can be directly inferred in the inference stage.

[0027] The initial region is cropped according to the initial region, and the initial region is scaled to a standard resolution through bilinear interpolation, where the standard resolution can be 127x127 pixels, and a gray scale normalization process is performed to map the data range of the gray scale value to [0, 1].

[0028] A reference template corresponding to the tracking target is obtained according to the processed initial region, and template position information corresponding to the reference template is recorded. At this time, the reference template can be represented as , where T is the reference template, R is the real number space, and the recorded template position information is the center coordinates and the width and height of the initial region.

[0029] S102: For a current frame image of the infrared video, a search region corresponding to the current frame image is obtained based on the predicted position information in the previous frame image.

[0030] The search region is represented in the current frame image, and the tracking target is tracked within the range of the search region.

[0031] Specifically, as shown in Figure 3 , for a current frame image of the infrared video, prediction position information in a previous frame image is determined; wherein, if the previous frame image is a first frame image, the prediction position information is template position information. At this time, the prediction position information in the previous frame image can be represented by a center coordinate.

[0032] Based on the prediction position information, according to the target size information, an enlarged clipping processing is performed to obtain a search region. For example, the target size information is clipped by 2 times to obtain a square search region.

[0033] The search region is scaled to a standard resolution by bilinear interpolation, and a region boundary check is performed. At this time, the standard resolution can be 255x255 pixels, and the search region can be represented as , wherein S is the search region, and R is a real number space. At the same time, the boundary check and the out-of-bound processing are implemented to ensure the integrity of the search region.

[0034] S103: For the reference template and the search region, a corresponding light-weight feature extraction module is used to extract a corresponding template feature and a multi-level search feature, respectively.

[0035] The reference template and the search region are synchronously input into a light network processing flow, and feature extraction is performed by a corresponding light-weight feature extraction module.

[0036] Specifically, a first light-weight feature extraction module for the reference template is determined, and a second light-weight feature extraction module for the search region is determined.

[0037] The first light-weight feature extraction module and the second light-weight feature extraction module both include a multi-branch residual convolution module and an adaptive feature enhancement module. For example, they are composed of a 5x5 size, 3x3 size multi-branch residual convolution module and an adaptive feature enhancement module.

[0038] The second light-weight feature extraction module includes a multi-level structure and can output infrared image features of multiple resolution sizes in a hierarchical manner. For example, three resolution sizes of infrared image features , and are set to correspond to low, medium and high resolution features, respectively.

[0039] The general template feature corresponding to the reference template is extracted by the first light-weight feature extraction module , wherein for the general template feature, for processing the reference template through the first lightweight feature extraction module. The second lightweight feature extraction module extracts the multi-level general search features corresponding to the search region wherein, for the multi-level general search features, , and correspond to different levels, respectively, corresponding to low, medium and high resolution features, for processing the search region through the second lightweight feature extraction module.

[0040] As shown in Figure 4 , in the 5x5 multi-branch residual convolution module, the input features are respectively convolved through the convolution 5x5 layer, the convolution 5x1 layer, the convolution 5x5 layer, the convolution 1x5 layer and the convolution 1x1 layer, then spliced after the corresponding batch normalization layer, and input to the convolution 1x1 layer for convolution processing.

[0041] As shown in Figure 5 , in the 3x3 multi-branch residual convolution module, the input features are respectively convolved through the convolution 3x3 layer, the convolution 3x1 layer, the convolution 1x3 layer and the convolution 1x1 layer, then spliced after the corresponding batch normalization layer, and input to the convolution 1x1 layer for convolution processing.

[0042] As shown in Figure 6 , in the adaptive feature enhancement module, the input features are respectively scaled by the learnable parameter weight 1 and the learnable parameter weight 2 to obtain the weighted feature 1 and the weighted feature 2, and then combined with the corresponding bias parameters of the learnable parameters through feature multiplication operation to obtain the final output features, that is, the general template features and the multi-level general search features, respectively.

[0043] The general template features are processed through the category branch , the regression branch , to obtain the task-specific template features , and the multi-level general search features are processed through the category branch , the regression branch .

[0044] The category branch is composed of a plurality of convolution layers for classifying the target and the background, and the regression branch is composed of a plurality of convolution layers for bounding box regression. for the task-specific template features, for the multi-level task-specific search features; They are task-specific template features obtained by processing the general template features through the category branch and regression branch features; Chalk is a task-specific search feature obtained by processing the general search feature through the category branch and regression branch features.

[0045] S104: Obtaining feature mutual correlation between the template feature and the search feature through a hierarchical screening strategy, generating a similarity full-image response map, and obtaining a candidate region corresponding to the tracking target according to the similarity full-image response map.

[0046] Specifically, according to the multi-level task-specific search features, the resolutions corresponding to each level are sorted from low to high, for example, 、 and Corresponding to the low, medium and high levels respectively.

[0047] Based on the sorting, the similarity full-image response map corresponding to the search area of ​​the level is determined in turn according to the task-specific template features and the task-specific search features corresponding to the level in the sorting, and the search area is refined according to the similarity full-image response map, and the refined search area is used as the search area corresponding to the next level, until the similarity full-image response map corresponding to the highest level is determined; wherein the similarity full-image response map includes a category response map and a regression response map.

[0048] Use hierarchical filtering strategies on feature dimensions to compute task-specific template features and task-specific search features Taking the low, medium and high levels as an example, we first calculate the category branches at the low resolution level. and regression branch The similarity full-image response map is obtained to obtain the rough positioning area. Among them, and They are in The corresponding feature representation in the low-resolution layer is similar. and They are in The corresponding feature representation at the medium-resolution level, and and They are in The corresponding feature representations in the high-resolution layer. The same task-specific template features are used to calculate feature cross-correlation. The similarity full-image response graph reflects the heat map of the tracking target location, which can be used to refine the search area.

[0049] Next, the search region of the mid-resolution level is cropped according to the coarse positioning result. The similarity full-map response map of the category branch and the regression branch is calculated in the cropped mid-resolution search region, and a refined positioning region is further obtained.

[0050] Finally, the search region of the high-resolution level is cropped according to the refined positioning result. The similarity full-map response map of the category branch and the regression branch is calculated in the cropped high-resolution search region. Finally, the similarity full-map response map and is obtained, wherein, is the category response map, and is the regression response map.

[0051] S105: Based on the historical position information of the tracking target in the previous frame image, the candidate region is screened to obtain a target region, and the target region is positionally adjusted through regression calculation.

[0052] To improve the robust tracking effect of the target in the continuous frames, a tracker module combining target historical information is introduced to realize the correlation modeling of the target features in the similarity full-map response map and the improvement of the target positioning accuracy guided by the historical tracking information. As an extension of the above feature level screening framework, the tracker module is embedded after the alignment of the features of the benchmark template and the search region (that is, the task-specific template features and the task-specific search post are obtained), and before the final prediction of the target object position, effectively improving the recovery ability in the occlusion case and the stability under the scene change.

[0053] Specifically, as shown in Figure 7 , a candidate region token is constructed according to the candidate region, a historical query token is constructed according to the historical position information of the tracking target in the previous frame image, a target position prediction token is constructed according to the region and adjacent region where the historical query token is located, and a context token is constructed according to the auxiliary features of the previous frame image.

[0054] In different resolutions, the Top-k feature positions with the largest response values in the final category response map are located, and these positions are considered as the candidate regions where the target is most likely to exist in the current frame.

[0055] For each candidate region, a candidate region token, also called candidate Token, is constructed, whose vector is recorded as [B, N1, C]. The high-resolution center and its surrounding 3x3 neighborhood features in the search region multi-level branch features are extracted, cropped from the first 64 feature maps of the original channel dimension, and each feature block is flattened into a single-channel feature vector with a size of 64x9.

[0056] A history query token, also called history Query Token, is constructed, whose vector is recorded as [B, N2, C]. In the history information, the target center position and the features in its 3x3 neighborhood range are tracked in the previous frame image, and are also flattened into a single-channel feature with a size of 64x9. The history Query Token is jointly constructed with the absolute position encoding of the feature map as a reference vector for matching the target region in the current frame image.

[0057] A target position prediction token and a context token are constructed, respectively called target position prediction Token and context Token, whose vectors are recorded as [B, N3, C] and [B, N4, C]. The target prediction position Token is jointly constructed by the target position and position information (absolute coordinate encoding) in the previous frame image and the neighborhood features, and is mainly used for target relocation in occlusion. The context Token is an auxiliary feature of the non-target region, which is directly inherited from the context Token set of the previous frame and is used to guide the context modeling of the current frame Transformer. The context Token does not participate in target regression, but has an auxiliary role in maintaining tracking continuity.

[0058] As shown in Figure 7 The candidate region token, history query token, target position prediction token, and context token are fused to generate a fusion sequence, and are modeled based on a lightweight Transformer architecture and self-attention mechanism.

[0059] The self-attention modeling based on the lightweight Transformer and the Token-level fusion are performed, the candidate Token, the target history Query Token, the target position prediction Token, and the context Token constructed above are fused into a unified fusion sequence, and a learnable type encoding is added, which is input into the lightweight self-attention Transformer module for multi-head attention modeling.

[0060] In the lightweight Transformer architecture, the features are processed by the learnable type encoding and the corresponding layers (including lightweight Transformer Block1~lightweight Transformer Block3 and the final normalization layer), and the Token sequence is cut. At this time, it is the enhanced fusion sequence after feature processing, and the enhanced candidate Token, the enhanced target history Query Token, the enhanced target position prediction Token and the enhanced context Token are obtained.

[0061] Meanwhile, as shown in Figure 8 In each lightweight Transformer Block of the lightweight Transformer Block1~lightweight Transformer Block3, the Token sequence is input, and the Q projection and the K projection are used to reduce the dimension to C / r, to obtain the multi-head reshaped Q and the multi-head reshaped K, and the attention calculation, the normalization Softmax and the regularization Dropout are performed; and the V projection is used to keep the C dimension, to obtain the multi-head reshaped V; wherein Q, K and V respectively represent the query vector Query, the key vector Key and the value vector Value, r is the dimension reduction ratio, and the multi-head reshaped Q, the multi-head reshaped K and the multi-head reshaped V respectively correspond to the split and reshaped vectors.

[0062] The calculation results after the normalization Softmax and the regularization Dropout are subjected to attention calculation with the multi-head reshaped V, and the projection is output after the multi-head merging (the output results of multiple heads are spliced), the multi-branch residual convolution (convolution is good at processing local spatial information, which is complementary to the global modeling ability of the attention mechanism, thereby introducing the locality prior, and further enhancing the feature expression), and the residual connection with the input Token sequence is directly added, thereby relieving the gradient vanishing problem in the deep network, and outputting the final vector.

[0063] Based on the established tracker module, the existence result of the tracking target corresponding to the candidate region is returned, the target region corresponding to the current frame image is determined, and the absolute coordinate position of the tracking target is regressed, so as to adjust the position of the target region through the absolute coordinate position.

[0064] Through the mechanism, the established tracker module can dynamically capture the matching relationship between the candidate region and the historical target position, and adjust the final prediction of the current frame based on the context. In the Token output by the prediction and target regression strategy Transformer, the candidate Token is regressed as the target existence (regression 1 as the target, and 0 as the non-target); and the target position prediction Token is regressed as the absolute coordinate position of the target wherein the center coordinates of the detection frame of the target region are respectively adjusted and height .

[0065] Specifically, the correction of the center coordinates is based on the coordinates of the feature map position mapped to the original image, plus the offset predicted by regression, that is, ; ; wherein, , is the center coordinate of the adjusted target region. The adjustment of the width and height is achieved by multiplying the initial target size by the scale factor predicted by regression, that is, ; ; wherein, , is the width and height of the adjusted target region, , is the width and height of the target region before adjustment.

[0066] In particular, in the occlusion scene, the Token output is preferentially used as a supplement or alternative basis for target position information. The context Token is directly passed to the next frame to continue modeling, achieving information transmission and continuous modeling.

[0067] For occlusion detection and target recovery, when the target response of consecutive frames significantly decreases and the target prediction position Token confidence is higher than the candidate Token, it is considered that occlusion occurs, and target recovery is performed by relying on the target prediction position Token. Scene switching and target disappearance processing are performed by searching the search box of the search region and detecting the consistency of the context Token to determine whether to enter a new scene or the target disappears, maintain the stability of the search region and delay updating the target template until a new target is confirmed.

[0068] By using a lightweight feature extraction module and a multi-level search feature structure, the multi-scale feature perception ability is maintained while the parameter amount is reduced, and the computational complexity is reduced. A hierarchical screening strategy is introduced to perform coarse-to-fine feature cross-correlation calculation, effectively focusing on the potential target region, reducing redundant calculation, and improving tracking efficiency. By combining historical position information to construct a lightweight tracker, the robustness and continuous tracking ability in the face of challenges such as target occlusion, deformation, and complex background are enhanced, which can be efficiently deployed on edge devices with limited computing resources, achieving an effective balance between tracking performance and inference efficiency.

[0069] In one embodiment, the training process of the lightweight feature extraction module includes: For the first stage training, the lightweight feature extraction module is trained until the lightweight feature extraction module converges, and the corresponding initial pre-training weight is obtained. At this time, the first lightweight feature extraction module , the second lightweight feature extraction module , the category branch , the regression branch Conduct regular training until convergence to obtain initial pre-training weights .

[0070] For the second stage training, the initial pre-training weights are frozen As prior knowledge, introduce the learnable weight increment , and through the residual summation method, the learnable weight increment which is isomorphic to the initial pre-training weight is dynamically optimized, and the final pre-training weight is obtained by fusing the initial pre-training weight and the learnable weight increment . The initial pre-training weight obtained in the first stage training is fixed as a constant to avoid being covered in the second stage training, ensuring the stability of the prior knowledge. Only ΔW is updated in the second stage training, reducing the optimization freedom and alleviating the risk of overfitting, while retaining the high-frequency detail learning ability through the residual path.

[0071] Further, as shown in Figure 9 , the model calculation amount is further reduced using reparameterization. Using the linearity and homogeneity of convolution operation, the complex structure of multiple branches and batch normalization is equivalent to fusion into a single convolution kernel and bias, realizing “training multiple branches, reasoning single branch”.

[0072] At this time, the training process of the lightweight feature extraction module further includes: The final pre-training weight obtained by training the lightweight feature extraction module is fused with the convolution layer and the batch normalization layer in the lightweight feature extraction module to obtain the fused convolution weight and the convolution bias.

[0073] The statistical information obtained during the training stage is extracted from the batch normalization layer , including the running mean and the running variance , which respectively record the mean and variance statistics of the features during the training process, for normalization in the inference stage; is a small constant (for example, set to 1e-5) for numerical stability; and are two learnable parameters that control the scaling and translation of the normalized features, respectively.

[0074] Calculate the standardization coefficient and the offset coefficient ; then expand the one-dimensional coefficient k to a four-dimensional tensor through a tensor dimension expansion operation to match the dimension of the convolution weight, and multiply the original convolution weight (in this case, the final pre-training weight in the above can be selected) by the standardization coefficient and learnable scaling parameters for batch normalization , get the fused convolution weight (referred to as fused convolution weight) ; At the same time, similar processing is performed on the bias term, first the original convolution bias Multiply And add the offset factor , multiplied by the learnable scaling parameter of batch normalization And add the offset parameter , and finally get the fused convolution bias (referred to as fused convolution bias) .

[0075] Here Examples of multi-branch residual convolution modules of different sizes: During the training phase, parallel The network consists of four branches: convolution + batch normalization layer, 1×1 convolution + batch normalization layer, 3×1 convolution + batch normalization layer, and 1×3 convolution + batch normalization layer. Finally, a residual approach is used to concatenate the features that have passed through only the convolution layer and those that have passed through the convolution + batch normalization layer in the channel dimension to generate multi-branch features. Finally, a 1×1 convolution is used to reduce the dimensionality of these multi-branch features.

[0076] The reasoning process of the lightweight feature extraction module also includes: According to the fused convolution weights corresponding to each branch structure in the multi-branch residual convolution module, the branch structures in the multi-branch residual convolution module are fused through branch alignment, branch splicing and dimensionality reduction transformation to obtain a single branch structure.

[0077] Still Examples of multi-branch residual convolution modules of different sizes: In the inference stage: the convolution kernels of 1×1 convolution, 3×1 convolution and 1×3 convolution in the branch are padded with zeros and expanded to 3×3 to achieve branch alignment; then the convolution kernel of each branch is fused with the batch normalization parameters to obtain the fused convolution weight and fused convolution bias ; All branch original convolution weights will be used in a residual manner and fused convolution weights And the original convolution bias and fused convolution bias Perform splicing of channel dimensions separately to obtain splicing convolution weights With concatenated convolution bias , realize branch splicing; use the learnable parameters of 1×1 convolution kernel (weight of 1×1 convolution kernel) and (1×1 convolution kernel bias) concatenated convolution weight With concatenated convolution bias dimensionality reduction, i.e. , ; wherein, is the re-parameterized equivalent convolution kernel weight, is the re-parameterized equivalent convolution kernel bias. Finally, the 3x3 size multi-branch residual convolution module is re-parameterized into edge terminal computation friendly 3x3 convolution kernel (i.e. the re-parameterized equivalent convolution kernel) in the model inference stage, to realize the generation of single-branch structure.

[0078] As shown in Figure 10 , the embodiment of the present application also provides a lightweight infrared single-target tracking device based on historical information guidance, comprising: at least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the lightweight infrared single-target tracking method based on historical information guidance according to any one of the above embodiments.

[0079] The embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to implement the lightweight infrared single-target tracking method based on historical information guidance according to any one of the above embodiments.

[0080] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0081] The device and medium provided by the embodiment of the present application are one-to-one corresponding to the method, so the device and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here.

[0082] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0083] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0084] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0085] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0086] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0087] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), optical or optical disk storage, flash memory, etc. The memory can be included as part of the computer-readable media described above.

[0088] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0089] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0090] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A lightweight infrared single target tracking method based on historical information guidance, characterized in that: include: Collecting infrared video and establishing a reference template corresponding to the tracking target based on the first frame image of the infrared video; For the current frame image of the infrared video, based on the predicted position information in the previous frame image, a search area corresponding to the current frame image is obtained; For the reference template and the search area, respectively extract corresponding template features and multi-level search features through their corresponding lightweight feature extraction modules; Obtaining feature mutual correlation between the template feature and the search feature through a hierarchical screening strategy, generating a similarity full-image response map, and obtaining a candidate region corresponding to the tracking target based on the similarity full-image response map; Based on the historical position information of the tracking target in the previous frame image, the candidate areas are screened to obtain the target area, and the position of the target area is adjusted through regression calculation.

2. The method according to claim 1, characterized in that According to the first frame image of the infrared video, a reference template corresponding to the tracking target is established, specifically including: Determining a first frame image of the infrared video and determining an initial area selected from the first frame image; Cropping is performed according to the initial area, scaling the initial area to a standard resolution by bilinear interpolation, and performing grayscale normalization processing; A reference template corresponding to the tracking target is obtained according to the processed initial area, and template position information corresponding to the reference template is recorded.

3. The method according to claim 2, characterized in that For the current frame image of the infrared video, based on the predicted position in the previous frame image, a search area corresponding to the current frame image is obtained, specifically including: For the current frame image of the infrared video, determining the predicted position information in the previous frame image; wherein, if the previous frame image is the first frame image, the predicted position information is the template position information; Based on the predicted position information, enlarging and cropping are performed according to the target size information to obtain a search area; The search region is scaled to a standard resolution by bilinear interpolation and a region boundary check is performed.

4. The method according to claim 1, wherein For the reference template and the search area, the corresponding template features and multi-level search features are extracted respectively through their corresponding lightweight feature extraction modules, specifically including: Determining a first lightweight feature extraction module for the reference template and a second lightweight feature extraction module for the search area; Extracting universal template features corresponding to the reference template by the first lightweight feature extraction module, and extracting multi-level universal search features corresponding to the search area by the second lightweight feature extraction module; Through the category branch and the regression branch, the general template features are feature processed to obtain task-specific template features, and the multi-level general search features are feature processed to obtain multi-level task-specific search features.

5. The method according to claim 4, characterized in that The first lightweight feature extraction module and the second lightweight feature extraction module both include a multi-branch residual convolution module and an adaptive feature enhancement module; the second lightweight feature extraction module includes a multi-layer structure, which can hierarchically output infrared image features of various resolution sizes.

6. The method according to claim 4, characterized in that The feature mutual correlation between the template feature and the search feature is obtained through a hierarchical screening strategy to generate a similarity full-image response graph, specifically including: According to the multi-level task-specific search features, the resolution corresponding to each level is sorted from low to high; Based on the sorting, the similarity full-image response map corresponding to the level in the sorting is determined in turn according to the task-specific template features and the task-specific search features corresponding to the level in the sorting, and the search area is refined according to the similarity full-image response map, and the refined search area is used as the search area corresponding to the next level, until the similarity full-image response map corresponding to the highest level is determined; wherein the similarity full-image response map includes a category response map and a regression response map.

7. The method according to claim 1, characterized in that Based on the historical position information of the tracking target in the previous frame image, the candidate areas are screened to obtain the target area, and the position of the target area is adjusted through regression calculation, specifically including: Constructing a candidate region token based on the candidate region, constructing a historical query token based on the historical position information of the tracking target in the previous frame image, constructing a target position prediction token based on the region and adjacent regions of the historical query token, and constructing a context token based on the auxiliary features of the previous frame image; Fusing the candidate region token, the historical query token, the target position prediction token, and the context token to generate a fused sequence, and modeling it based on a lightweight Transformer architecture and a self-attention mechanism; Based on the established tracker module, the existence result of the tracking target corresponding to the candidate area is returned, the target area corresponding to the current frame image is determined, and the absolute coordinate position of the tracking target is regressed to adjust the position of the target area through the absolute coordinate position.

8. The method according to claim 4, characterized in that The training process of the lightweight feature extraction module includes: For the first stage of training, the lightweight feature extraction module is trained until the lightweight feature extraction module converges and the corresponding initial pre-training weights are obtained; For the second stage of training, the initial pre-trained weights are frozen as prior knowledge, and the learnable weight increments that are isomorphic to the initial pre-trained weights are dynamically optimized by summing the residuals, and the final pre-trained weights are obtained by fusing the initial pre-trained weights and the learnable weight increments.

9. The method according to claim 8, characterized in that The training process of the lightweight feature extraction module also includes: The convolution layer and the batch normalization layer in the lightweight feature extraction module are fused using the final pre-trained weights obtained by training the lightweight feature extraction module to obtain fused convolution weights and convolution biases; The reasoning process of the lightweight feature extraction module also includes: According to the fused convolution weights corresponding to each branch structure in the multi-branch residual convolution module, the branch structures in the multi-branch residual convolution module are fused through branch alignment, branch splicing and dimensionality reduction transformation to obtain a single branch structure.

10. A lightweight infrared single target tracking device based on historical information guidance, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the lightweight infrared single target tracking method based on historical information guidance as described in any one of claims 1 to 8.

Images