Foreground sensing Transform target tracking method and system based on re-tracking strategy

Abstract

The invention discloses a foreground perception Transform target tracking method and system based on a re-tracking strategy, and relates to the technical field of image recognition and target tracking, and the method comprises the steps: inputting a nearest frame template image and a search region image into a pre-constructed preprocessing layer, then respectively generating a template token sequence and a search token sequence, and caching the template token sequence and the search token sequence to a re-tracking cache region; splicing the two sequences to obtain a mixed sequence, inputting the mixed sequence into a pre-established foreground perception encoder to obtain an output sequence, filling a search region feature sequence in the output sequence, and inputting the search region feature sequence into a prediction head to obtain a prediction result; and if the prediction results of the continuous t frames are all lower than a preset expected value, starting a re-tracking strategy, obtaining a re-tracking prediction result through a general encoder, and carrying out comprehensive evaluation in a re-tracking evaluation area to obtain an optimal tracking result. According to the method, background interference is effectively reduced, and tracking deviation can be corrected in time.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and target tracking, and in particular to a foreground-aware Transformer target tracking method and system based on a re-tracking strategy. Background Art

[0002] In the field of computer vision, visual target tracking has become a hot research topic that has attracted much attention. It has shown broad and promising application prospects in many key fields such as military, unmanned driving, and medical treatment. Specifically, visual target tracking aims to automatically and accurately define the position and shape outline of the target in subsequent consecutive frames of the video after the target of interest is accurately marked with a bounding box in the first frame. However, in real-world complex scene applications, visual target tracking encounters many difficulties, such as background clutter interference, frequent target occlusion, and target deformation. These unfavorable factors will significantly weaken the accuracy of tracking.

[0003] At present, mainstream visual target tracking methods focus on the Transformer architecture to carry out innovative exploration, give full play to the Transformer's outstanding advantages in processing local and global information, greatly enhance the algorithm's modeling potential, and thus significantly improve the overall performance of the tracking algorithm. According to the differences in working modes, the current Transformer-based target tracking strategies are mainly divided into two categories: dual-stream dual-stage tracking method and single-stream single-stage tracking method. Among them, the dual-stream dual-stage tracking method uses two completely consistent network branches, focusing on extracting template features and searching for area features respectively, and in process design, the feature extraction and feature fusion are finely divided into two independent stages that follow one after another and are implemented step by step; in contrast, the single-stream single-stage tracking rule only relies on a single network branch, and achieves feature extraction and fusion simultaneously with a single-stage process, which greatly simplifies the overall architecture of the algorithm and effectively improves the computational efficiency.

[0004] Despite this, most existing Transformer tracking methods have certain limitations. The self-attention mechanism lacks attention to the most relevant foreground information in the search area, and is prone to paying too much attention to background information and being disturbed. Moreover, the self-attention calculation of the entire image is too large and the reasoning speed is slow. Secondly, in real dynamic scenes, the tracker not only needs to deal with complex and changeable background interference and frequent occlusion problems, but also as the tracking time span continues to extend, such challenges are exponentially magnified. Summary of the invention

[0005] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a foreground-aware Transformer target tracking method and system based on a re-tracking strategy.

[0006] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a foreground-aware Transformer target tracking method based on a re-tracking strategy, the method comprising the following steps: Receiving the most recent frame template image and the search area image, inputting the most recent frame template image and the search area image into a pre-built preprocessing layer to generate a template token sequence and a search token sequence respectively, and caching the template token sequence and the search token sequence into a re-tracking buffer area; The template token sequence and the search token sequence are concatenated to obtain a mixed sequence, and the mixed sequence is input into the pre-established foreground-aware Transformer encoder to obtain an output sequence, and the search area feature sequence in the output sequence is padded, and the padded output sequence is input into the prediction head to obtain a prediction result; The prediction result is compared with the preset expected value. If the prediction results of t consecutive frames are lower than the preset expected value, the prediction result is stored in the re-tracking evaluation area, and the template token sequence and search token sequence in the re-tracking buffer are input into the universal Transformer encoder for tracking prediction to obtain the re-tracking prediction result. The prediction result and the re-tracking prediction result are comprehensively evaluated in the re-tracking evaluation area to obtain the best tracking result, and the template image is updated online.

[0007] In combination with the first aspect, in some implementations of the first aspect, the method further includes: , where the template image height is H z , width is W z , the number of channels is 3, represents a set of real numbers. The image is divided into multiple image blocks with a side length of P and flattened into a template block sequence , the number of template blocks is . Search area image , where the template image height is H x , width is W x , the number of channels is 3, the image is divided into multiple image blocks with a side length of P, flattened into a search block sequence , the number of search blocks is .

[0008] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of generating the template token sequence and the search token sequence: Using a linear projection layer with an input parameter of L, the template block sequence and the search block sequence are mapped to a D-dimensional space, and a learnable position embedding is added to each block embedding to obtain the template token embedding Z and the search token embedding X: In the formula, z p i is the i-th block in the template block sequence, x p i is the i-th block in the search block sequence, P z is the position embedding of the template token, P x To search for the position embedding of the token, L represents a linear projection layer with parameter L.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the mixed sequence is obtained by splicing Z and X according to the channel to obtain a mixed sequence M 0 ; The mixed sequence is input into the pre-established foreground-aware Transformer encoder and the query q of the template is obtained after the normalization layer. z , key k z 、value v z and the query q for the search area x , key k x 、value v x .

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: after inputting the mixed sequence into a pre-established foreground-aware Transformer encoder, using a foreground-enhanced multi-head self-attention module to extract features of the fused mixed sequence, and using a background screening module to filter and discard background tokens.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the calculation process of extracting the features of the fused mixed sequence using the foreground enhanced multi-head self-attention module is as follows: In the formula, FEA Represents the calculation of a single foreground enhancement self-attention head, q, k, v represent the input query vector, key vector, value vector, Softmax is the activation function, Maxj represents the K largest values ​​in each row of the attention weight matrix, k T represents the transpose of the mixed sequence key k, is the dimension of the mixed token sequence, represents a zero-value matrix, k z T and k x T They are the transposition of the template sequence key and the transposition of the search sequence key respectively. The foreground enhancement multi-head self-attention calculation formula is as follows: Where FEMSA represents the foreground enhanced multi-head attention result, Concat Indicates the splicing operation, head i It represents the calculation result of the i-th foreground enhancement self-attention head. The output of the foreground enhancement multi-head attention module is added to the residual of the mixed token sequence M1 and input into the background removal module.

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of using the background screening module to screen and discard background tokens: The similarity between the center position of the template and all positions in the search area is selected as the basis for screening the background. The expression is: In the formula, w i It represents the similarity calculated by the i-th head. Since there are h attention heads, the center position of the template can calculate h similarity results. The background removal threshold is the average value of h similarity results, expressed as According to the background screening threshold, the search tokens with a lower similarity are deleted, and the retained search tokens need to record their original positions and then reassembled with the template token to obtain the mixed token sequence M2; After normalization and feedforward network, M2 is added to its own residual to obtain the output M of the Transformer block. 1 ; After N Transformer blocks, the final output M of the encoder is obtained N .

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: N The search token sequence is extracted and restored to the initial order according to the initial position embedding of the token. For the missing positions, a 0-filling strategy is adopted to obtain the complete search token sequence X'; Reshape X' into a two-dimensional feature map and input it into the three-branch prediction head to obtain the classification score, position deviation, and target scale respectively, and generate the predicted bounding box; Compare the prediction result with the preset expected value τ. If the value of consecutive t frames is lower than τ, it indicates that there is a deviation in the current prediction result. At this time, the re-tracking strategy is started, and the classification result and the bounding box result are temporarily stored in the re-tracking evaluation area. The template token sequence and search token sequence in the re-tracking buffer are taken out and input into the universal Transformer encoder. Different from the foreground-aware Transformer encoder, the universal Transformer encoder replaces the foreground-enhanced multi-head self-attention module with a multi-head self-attention module and reduces the number of Transformer blocks. The calculation process of the multi-head self-attention module is as follows: In the formula, Attention(q,k,v) represents the calculation result of a single self-attention head for mixed sequence input, Softmax is the activation function, is the dimension of the mixed token sequence, head i It represents the calculation result of the i-th self-attention head, and MHSA represents the multi-head self-attention result.

[0014] The output result of the general Transformer encoder is input into the prediction head to obtain the prediction result of the re-tracking and store it in the re-tracking evaluation area; Take the bounding box with the highest classification score in the re-tracking evaluation area as the final prediction result; If the classification score of the final prediction result is higher than the update threshold θ, where θ> the preset expected value τ, a new template image is cropped from the current frame image, specifically centered on the predicted bounding box and centered on the size H z ×W z The template image is cropped according to the specifications to achieve online update of the template image.

[0015] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a foreground-aware Transformer target tracking system based on a re-tracking strategy, comprising: An image processing module is used to receive a most recent frame template image and a search area image, input the most recent frame template image and the search area image into a pre-built pre-processing layer to generate a template token sequence and a search token sequence respectively, and cache the template token sequence and the search token sequence in a re-tracking buffer area; The sequence prediction module is used to concatenate the template token sequence and the search token sequence to obtain a mixed sequence, input the mixed sequence into the pre-established foreground-aware Transformer encoder to obtain an output sequence, fill the search area feature sequence in the output sequence, and input the filled output sequence into the prediction head to obtain a prediction result; The prediction evaluation module is used to compare the prediction result with the preset expected value. If the prediction result of t consecutive frames is lower than the preset expected value, the template token sequence and the search token sequence in the re-tracking buffer area are predicted again through the universal Transformer encoder to obtain the prediction result of the re-tracking evaluation area. The prediction result is comprehensively evaluated with the prediction result of the re-tracking evaluation area to obtain the best tracking result.

[0016] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the foreground-aware Transformer target tracking method based on the re-tracking strategy as described above is adopted.

[0017] Beneficial effects of the present invention: The present invention adopts a single-stream single-stage tracking method, which greatly simplifies the tracking architecture; uses a foreground-aware Transformer encoder to extract fusion templates and search area features, which greatly reduces the computational complexity while enhancing foreground target perception and suppressing background interference; uses an adaptive re-tracking strategy to correct tracking deviations in a timely manner, and is more adaptable to complex dynamic scenes; and the online template update strategy avoids tracking failures caused by template aging. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 is a flow chart of the target tracking method of the present invention; Figure 2 This is a workflow diagram of the foreground perception Transformer module of the present invention; Figure 3 It is a schematic diagram of the workflow of the present invention; Figure 4 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Embodiment 1: like Figure 1 As shown, the foreground-aware Transformer target tracking method based on the re-tracking strategy includes the following steps: S101: receiving a most recent frame template image and a search area image, inputting the most recent frame template image and the search area image into a pre-built preprocessing layer to generate a template token sequence and a search token sequence respectively, if the most recent frame template image is an initial frame template image, then caching the template token sequence and the search token sequence into a re-tracking buffer, otherwise only updating the search token sequence in the re-tracking buffer; During the first tracking, the nearest frame template image is used as the initial frame template image, which is input into the preprocessing layer together with the search area image to generate the nearest frame template token sequence (referred to as template token sequence) and the search token sequence, and then the template token sequence and the search token sequence are cached in the re-tracking buffer. In subsequent tracking, after the image passes through the preprocessing layer, the template token sequence and the search token sequence in the re-tracking buffer are updated. The template image of the nearest frame , where the template image height is H z , width is W z , the number of channels is 3, represents a set of real numbers. The image is divided into multiple image blocks with a side length of P and flattened into a template block sequence , the number of template blocks is . Search area image , where the template image height is H x , width is W x , the number of channels is 3, the image is divided into multiple image blocks with a side length of P, flattened into a search block sequence , the number of search blocks is .

[0021] Using a linear projection layer with a parameter L as input, the template block sequence and the search block sequence are mapped to a D-dimensional space, and a learnable position embedding is added to each block embedding to obtain the template token embedding Z and the search token embedding X: In the formula, z p i is the i-th block in the template block sequence, x p i is the i-th block in the search block sequence, P z is the position embedding of the template token, P xTo search for the position embedding of the token, L represents a linear projection layer with parameter L.

[0022] If the template has been updated, the template token sequence and the search token sequence of the current frame are stored in the re-tracking buffer; otherwise, only the search token sequence in the re-tracking buffer is replaced with the search token sequence of the current frame.

[0023] S102: concatenate the template token sequence and the search token sequence to obtain a mixed sequence, input the mixed sequence into a pre-established foreground-aware Transformer encoder to obtain an output sequence, fill the search region feature sequence in the output sequence, and input the filled output sequence into a prediction head to obtain a prediction result; Concatenate Z and X by channel to get the mixed token sequence M 0 ; The mixed token sequence is input into the encoder’s foreground-aware Transformer block and the query q of the template is obtained after the normalization layer. z , key k z 、value v z and the query q for the search area x , key k x 、value v x ; The foreground enhancement multi-head self-attention is used to extract and fuse features of the mixed token sequence to enhance the foreground area and reduce the attention to the background. Different from the previous multi-head attention, the foreground enhancement self-attention will perform Softmax calculation on the first j maximum values ​​of each row in the attention weight matrix, so as to ensure that the attention is focused on the foreground area and reduce the influence of the background on the attention weight. Then, the remaining positions in each row are filled with 0 to discard the weights with low attention and highlight the foreground. The formula is as follows: In the formula, FEA Represents the calculation of a single foreground enhancement self-attention head, q, k, v represent the input query vector, key vector, value vector, Softmax is the activation function, Maxj represents the K largest values ​​in each row of the attention weight matrix, k T represents the transpose of the mixed sequence key k, is the dimension of the mixed token sequence, represents a zero-value matrix, k z T and k x TThey are the transposition of the template sequence key and the transposition of the search sequence key respectively. The foreground enhancement multi-head self-attention calculation formula is as follows: Where FEMSA represents the foreground enhanced multi-head attention result, Concat Indicates the splicing operation, head i Represents the calculation result of the i-th foreground enhancement self-attention head. The output of the foreground enhancement multi-head attention module is added to the residual of the mixed token sequence M1 and input into the background removal module.

[0024] Since there is also background in the template, and the center of the template is mostly the target area, the similarity result between the center position of the template and all positions in the search area is selected as the basis for screening the background. The expression is In the formula, w i Indicates the similarity calculated by the i-th head. Since there are h attention heads, the center position of the template can calculate h similarity results. In order to obtain a fairer result, the background screening threshold is the average value of h similarity results, expressed as According to the background screening threshold, all search tokens with a similarity lower than this threshold are deleted, and the retained search tokens need to record their original positions and then reconnect them with the template token to obtain a mixed token sequence M2.

[0025] After normalization and feedforward network, M2 is added to its own residual to obtain the output M of the Transformer block. 1 ; After N Transformer blocks, the final output M of the encoder is obtained N .

[0026] M N The search token sequence is extracted and restored to its original order according to the initial position embedding of the token. For the missing positions, a 0-filling strategy is adopted to obtain the complete search token sequence X'.

[0027] Reshape X' into a two-dimensional feature map and input it into the three-branch prediction head to obtain the classification score, position deviation, target scale, and generate a predicted bounding box.

[0028] S103: Compare the prediction result with the preset expected value. If the prediction results of t consecutive frames are all lower than the preset expected value, the prediction result is stored in the re-tracking evaluation area, and the template token sequence and the search token sequence in the re-tracking cache are input into the universal Transformer encoder for tracking prediction to obtain the re-tracking prediction result. The prediction result and the re-tracking prediction result are comprehensively evaluated in the re-tracking evaluation area to obtain the best tracking result, and the template image is updated online.

[0029] When the classification scores of consecutive t frames are all lower than the credible threshold τ, it indicates that the current prediction result may be biased. At this time, the re-tracking strategy is decisively enabled to obtain more accurate tracking results.

[0030] Temporarily store the classification results and bounding box results in the re-tracking evaluation area to provide key data reserves for subsequent result evaluation and optimization; The template token sequence and search token sequence in the re-tracking buffer are taken out and input into the general Transformer encoder. Unlike the foreground-aware Transformer encoder, the general Transformer encoder replaces the foreground-enhanced multi-head self-attention module with a multi-head self-attention module and reduces the number of Transformer blocks to prevent discarding too many important clues that may be related to the target, improve tracking accuracy, and reduce the computational burden of re-tracking. The calculation process of the multi-head self-attention module is as follows: In the formula, Attention(q,k,v) represents the calculation result of a single self-attention head for mixed sequence input, Softmax is the activation function, is the dimension of the mixed token sequence, head i It represents the calculation result of the i-th self-attention head, and MHSA represents the multi-head self-attention result.

[0031] The output result of the general Transformer encoder is input into the prediction head to obtain the prediction result of the re-tracking and store it in the re-tracking evaluation area; Take the bounding box with the highest classification score in the re-tracking evaluation area as the final prediction result; If the classification score of the final prediction result is higher than the update threshold θ (θ>credible threshold τ), a new template image is cropped from the current frame image, specifically centered on the predicted bounding box and centered on the size H z ×W z The template image is cropped according to the specifications to achieve online update of the template image.

[0032] Specifically, the present invention is further described below by way of embodiments: like Figure 2 As shown in the figure, first, the template image of the most recent frame and the search area image are input into the preprocessing layer to generate the corresponding template token sequence and search token sequence, and the cache is updated, that is, the cached template token sequence and search token sequence in the re-tracking cache are updated. Then, the two sequences are spliced ​​and input into the foreground-aware Transformer encoder for feature extraction and fusion, and the obtained search sequence is input into the prediction head. The prediction head outputs the classification score and the predicted bounding box. If the classification scores of consecutive t frames are less than the credible threshold τ, the re-tracking strategy is enabled. At this time, the classification score and the bounding box are cached during re-tracking, that is, after the re-tracking strategy is enabled, the classification score and the bounding box are cached in the re-tracking evaluation area; otherwise, the prediction is directly performed and the prediction result is output. After the re-tracking strategy is enabled, the template token and the search area token in the re-tracking cache are taken out, and the general Transformer encoder is used to reprocess them during re-tracking. The obtained search sequence is input into the prediction head to obtain the classification score and the bounding box. The classification score and the bounding box are input into the re-tracking evaluation area, and the classification score and the bounding box are comprehensively evaluated with the previous classification score and bounding box to output the best prediction result. Finally, if the classification score of the predicted result is greater than the update threshold θ, the current search image is cropped to update the template image.

[0033] like Figure 3 As shown in the figure, firstly, the mixed token sequence is input and layer normalization is performed on it, which helps to stabilize the model training process and accelerate the convergence speed. Then, the mixed token sequence uses the foreground enhancement self-attention module to enhance the attention to the foreground information, and uses the foreground enhancement self-attention mechanism to capture the dependency between tokens and highlight the foreground part. The output of the foreground enhancement self-attention module is added to the original input residual to retain the original information and prevent the gradient from disappearing. Then, the background screening module is used to remove the tokens belonging to the background in the search token sequence, so that the model focuses more on the foreground. The output of the background screening module is layer normalized again to standardize the data distribution, and then the high-level features are extracted through the feedforward network. The output result is added to the residual output of the background screening module to fuse the information, and finally the mixed token sequence after the background is screened out is output to improve the model's perception and processing ability of the foreground target, and improve performance and accuracy.

[0034] Embodiment 2: In the second aspect, as Figure 4 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a foreground-aware Transformer target tracking system based on a re-tracking strategy, comprising: The image processing module 11 is used to receive the latest frame template image and the search area image, input the latest frame template image and the search area image into a pre-built pre-processing layer to generate a template token sequence and a search token sequence respectively, and cache the template token sequence and the search token sequence in a re-tracking buffer area; The sequence prediction module 12 is used to concatenate the template token sequence and the search token sequence to obtain a mixed sequence, input the mixed sequence into a pre-established foreground-aware Transformer encoder to obtain an output sequence, fill the search area feature sequence in the output sequence, and input the filled output sequence into the prediction head to obtain a prediction result; The prediction evaluation module 13 is used to compare the prediction result with the preset expected value. If the prediction result of t consecutive frames is lower than the preset expected value, the template token sequence and the search token sequence in the re-tracking buffer area are predicted again through the universal Transformer encoder to obtain the prediction result of the re-tracking evaluation area. The prediction result is comprehensively evaluated with the prediction result of the re-tracking evaluation area to obtain the best tracking result.

[0035] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0036] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0037] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0038] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A foreground-aware Transformer target tracking method based on a re-tracking strategy, characterized in that: The method comprises the following steps: Receiving the most recent frame template image and the search area image, inputting the most recent frame template image and the search area image into a pre-built preprocessing layer to generate a template token sequence and a search token sequence respectively, and caching the template token sequence and the search token sequence into a re-tracking buffer area; The template token sequence and the search token sequence are concatenated to obtain a mixed sequence, and the mixed sequence is input into the pre-established foreground-aware Transformer encoder to obtain an output sequence, and the search area feature sequence in the output sequence is padded, and the padded output sequence is input into the prediction head to obtain a prediction result; The prediction result is compared with the preset expected value. If the prediction results of t consecutive frames are lower than the preset expected value, the prediction result is stored in the re-tracking evaluation area, and the template token sequence and search token sequence in the re-tracking buffer are input into the universal Transformer encoder for tracking prediction to obtain the re-tracking prediction result. The prediction result and the re-tracking prediction result are comprehensively evaluated in the re-tracking evaluation area to obtain the best tracking result, and the template image is updated online.

2. The foreground-aware Transformer target tracking method based on re-tracking strategy according to claim 1 is characterized in that: The most recent frame template image , where the template image height is H z , width is W z , Represents a real number set. The template image of the latest frame is divided into multiple image blocks with a side length of P and flattened into a template block sequence , the number of template blocks is , search area image , where the template image height is H x , width is W x , the search area image is divided into multiple image blocks with a side length of P and flattened into a search block sequence , the number of search blocks is .

3. The foreground-aware Transformer target tracking method based on re-tracking strategy according to claim 2 is characterized in that: The generation process of the template token sequence and the search token sequence: Using a linear projection layer with an input parameter of L, the template block sequence and the search block sequence are mapped to a D-dimensional space, and a learnable position embedding is added to each block embedding to obtain the template token embedding Z and the search token embedding X: In the formula, z p i is the i-th block in the template block sequence, x p i is the i-th block in the search block sequence, P z is the position embedding of the template token, P x To search for the position embedding of the token, L represents a linear projection layer with parameter L.

4. The foreground-aware Transformer target tracking method based on re-tracking strategy according to claim 3 is characterized in that: The mixed sequence is obtained by splicing Z and X according to the channel. 0 ; The mixed sequence is input into the pre-established foreground-aware Transformer encoder and the query q of the template is obtained after the normalization layer. z , key k z , value v z and the query q for the search area x , key k x , value v x .

5. The foreground-aware Transformer target tracking method based on re-tracking strategy according to claim 4 is characterized in that: After the mixed sequence is input into the pre-established foreground-aware Transformer encoder, the foreground-enhanced multi-head self-attention module is used to extract the features of the fused mixed sequence, and the background screening module is used to filter and discard the background tokens.

6. The foreground-aware Transformer target tracking method based on re-tracking strategy according to claim 5 is characterized in that: The calculation process of extracting the features of the fused mixed sequence using the foreground enhanced multi-head self-attention module is as follows: In the formula, FEA Represents the calculation of a single foreground enhancement self-attention head, q, k, v represent the input query vector, key vector, value vector, Softmax is the activation function, Maxj represents the K largest values ​​in each row of the attention weight matrix, k T represents the transpose of the mixed sequence key k, is the dimension of the mixed token sequence, represents a zero-value matrix, k z T and k x T They are the transposition of the template sequence key and the transposition of the search sequence key respectively. The foreground enhanced multi-head self-attention calculation formula is as follows: Where FEMSA represents the foreground enhanced multi-head attention result, Concat Indicates the splicing operation, head i It represents the calculation result of the i-th foreground enhancement self-attention head. The output of the foreground enhancement multi-head attention module is added to the residual of the mixed token sequence M1 and input into the background removal module.

7. The foreground-aware Transformer target tracking method based on re-tracking strategy according to claim 6 is characterized in that: The process of using the background screening module to filter and discard background tokens: The similarity between the center position of the template and all positions in the search area is selected as the basis for screening the background. The expression is: In the formula, w i It represents the similarity calculated by the i-th head. Since there are h attention heads, the center position of the template can calculate h similarity results. The background removal threshold is the average value of h similarity results, expressed as: According to the background screening threshold, the search tokens with a lower similarity are deleted, and the retained search tokens need to record their original positions and then reassembled with the template token to obtain the mixed token sequence M2; After normalization and feedforward network, M2 is added to its own residual to obtain the output M of the Transformer block. 1 ; After N Transformer blocks, the final output M of the encoder is obtained N .

8. The foreground-aware Transformer target tracking method based on re-tracking strategy according to claim 7 is characterized in that: M N The search token sequence is extracted and restored to the initial order according to the initial position embedding of the token. For the missing positions, a 0-filling strategy is adopted to obtain the complete search token sequence X'; Reshape X' into a two-dimensional feature map and input it into the three-branch prediction head to obtain the classification score, position deviation, and target scale respectively, and generate the predicted bounding box; Compare the prediction result with the preset expected value τ. If the value of consecutive t frames is lower than τ, it indicates that there is a deviation in the current prediction result. At this time, the re-tracking strategy is started, and the classification result and the bounding box result are temporarily stored in the re-tracking evaluation area. The template token sequence and search token sequence in the re-tracking buffer are taken out and input into the universal Transformer encoder. The universal Transformer encoder replaces the foreground enhancement multi-head self-attention module with a multi-head self-attention module. The calculation process of the multi-head self-attention module is as follows: In the formula, Attention(q,k,v) represents the calculation result of a single self-attention head for mixed sequence input, Softmax is the activation function, is the dimension of the mixed token sequence, head i represents the calculation result of the i-th self-attention head, and MHSA represents the multi-head self-attention result; The output result of the general Transformer encoder is input into the prediction head to obtain the prediction result of the re-tracking and store it in the re-tracking evaluation area; Take the bounding box with the highest classification score in the re-tracking evaluation area as the final prediction result; If the classification score of the final prediction result is higher than the update threshold θ, where θ> the preset expected value τ, a new template image is cropped from the current frame image: centered on the predicted bounding box, with a size of H z ×W z Cut to specifications.

9. A foreground-aware Transformer target tracking system based on a re-tracking strategy, which adopts the foreground-aware Transformer target tracking method based on a re-tracking strategy as claimed in any one of claims 1 to 8, characterized in that: include: An image processing module is used to receive a most recent frame template image and a search area image, input the most recent frame template image and the search area image into a pre-built pre-processing layer to generate a template token sequence and a search token sequence respectively, and cache the template token sequence and the search token sequence in a re-tracking buffer area; The sequence prediction module is used to concatenate the template token sequence and the search token sequence to obtain a mixed sequence, input the mixed sequence into the pre-established foreground-aware Transformer encoder to obtain an output sequence, fill the search area feature sequence in the output sequence, and input the filled output sequence into the prediction head to obtain a prediction result; The prediction evaluation module is used to compare the prediction result with the preset expected value. If the prediction result of t consecutive frames is lower than the preset expected value, the template token sequence and the search token sequence in the re-tracking buffer area are predicted again through the universal Transformer encoder to obtain the prediction result of the re-tracking evaluation area. The prediction result is comprehensively evaluated with the prediction result of the re-tracking evaluation area to obtain the best tracking result.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the foreground-aware Transformer target tracking method based on the re-tracking strategy described in any one of claims 1 to 8 is adopted.

Images