Visual target tracking method and device based on efficient sequence generation and electronic equipment

Abstract

The invention relates to a visual target tracking method and device based on efficient sequence generation and electronic equipment, and the method comprises the steps: carrying out the processing of an obtained search image, a template image and a randomly initialized tracking mark, and obtaining search visual embedding and template visual embedding; the method comprises the following steps: performing position coding after search, template visual embedding and tracking mark connection, then coding a position coding result by adopting an encoder based on a visual transformer framework, and processing a tracking mark coding feature by adopting a tracking prediction head to obtain a target tracking prediction result; if the target tracking prediction result meets a preset condition, taking the target tracking prediction result as a target tracking result; otherwise, decoding the coding structure by adopting a transformer decoder; and processing by adopting a tracking prediction head according to a decoding result to obtain a target tracking result. By adopting the method, the forward reasoning speed is increased, and the tracking speed is increased.

Description

Technical Field

[0001] The present application relates to the technical field of visual target tracking, and in particular to a visual target tracking method, device and electronic device based on efficient sequence generation. Background Art

[0002] Object tracking has always been a hot and difficult problem in the field of computer vision. It is defined as estimating the state information (position and size, etc.) of the object in the entire video given the position and size of the object in the first frame of the video (usually a rectangular box that tightly surrounds the object). Existing SOTA tracking algorithms usually adopt powerful feature extraction backbones, complex feature fusion modules, and prediction head networks to achieve high accuracy in public benchmarks. However, these trackers have high computational burden and memory usage, which leads to tracking delays and deployment challenges, especially on resource-constrained edge devices. Therefore, how to achieve a good balance between tracking accuracy and efficiency remains a key issue facing the tracking community.

[0003] In recent years, tracking methods represented by SeqTrack model tracking tasks as sequence generation tasks and are popular for their simple network architecture (transformer encoder-decoder network architecture) and loss function (cross entropy loss function). The SeqTrack tracking process is mainly divided into two steps: 1) The transformer encoder is used to perform joint feature extraction and feature fusion from the template and the search area; 2) The transformer decoder is used to generate a bounding box value sequence (start XYWH) from the randomly initialized start token (starttoken) and the search image features in an autoregressive manner (where XY is the upper left corner coordinate of the target; WH is the width and height of the target). Although it has achieved success in both single-modal and multi-modal tracking tasks, SeqTrack has the following two problems: 1) The four values ​​(XYWH) of the target bounding box sequence are generated one by one, which means that the decoder must run four times per frame during forward inference, which inevitably leads to high computational latency; 2) Tracking tokens are randomly initialized, which makes it difficult for the decoder to accurately predict the bounding box sequence. Summary of the invention

[0004] Based on this, it is necessary to provide a visual target tracking method, device and electronic device based on efficient sequence generation to address the above technical problems.

[0005] A visual target tracking method based on efficient sequence generation, the method comprising: Get the search image, template image, and randomly initialized tracking markers.

[0006] The search image and template image are divided into blocks and linearly projected respectively to obtain the search visual embedding and template visual embedding.

[0007] The search visual embedding, template visual embedding, and randomly initialized tracking markers are concatenated and positionally encoded to obtain the positionally encoded input markers.

[0008] An encoder based on the visual transformer framework is used to perform global self-attention operations on all input tags after position encoding to obtain tracking tag encoding features and search image encoding features.

[0009] The tracking marker encoding features are processed using the tracking prediction head to obtain the target tracking prediction result.

[0010] If the target tracking prediction result meets the preset conditions, the target tracking prediction result is used as the target tracking result.

[0011] If the target tracking prediction result does not meet the preset conditions, the transformer decoder is used to decode the tracking mark encoding features and the search image encoding features; the tracking prediction head is used to process the decoding results to obtain the target tracking result; the target tracking result is a bounding box sequence.

[0012] In one embodiment, the process of randomly initializing the tracking mark includes: using the VIT The token initialization method randomly initializes the tracking token.

[0013] In one embodiment, the search visual embedding, the template visual embedding, and the randomly initialized tracking mark are connected and position encoded, and the position encoded input mark is obtained as follows:

[0014] in, is the position-encoded input token, are randomly initialized tracking markers, To search for visual embeddings of images, is the visual embedding of the template image; is the positional encoding, is the set of real numbers, For the dimension, is the number of blocks.

[0015] In one embodiment, an encoder based on a visual transformer framework is used to perform a global self-attention operation on all input tags after position encoding to obtain tracking tag encoding features and search image encoding features, including: The position-encoded input mark is input into the encoder based on the visual transformer framework to realize the interaction between template features and search features, the interaction between tracking marks and search features, and the interaction between tracking marks and template features. The encoding results corresponding to the tracking marks and search images in the last layer output of the encoder are output to the decoder.

[0016] In one embodiment, the transformer decoder includes layer decoder layers; each decoder layer contains an unmasked multi-head self-attention block, a multi-head attention block and a feedforward network block; in the first decoder layer: The tracking marker encoded features interact with each other through an unmasked multi-head attention mechanism as a query, and the interaction between the query and search image features is performed based on the multi-head attention block, and then processed by the feedforward network block and output to the tracking prediction head to obtain the target's bounding box.

[0017] In one embodiment, a transformer decoder is used to decode the tracking mark encoding feature and the search image encoding feature; a tracking prediction head is used to process the decoding result to obtain a target tracking result; the target tracking result is a bounding box sequence, including: The tracking marker encoded features and the search image encoded features are input into the transformer decoder for decoding.

[0018] The obtained decoding results are input into the tracking prediction head to obtain a bounding box sequence; the tracking prediction head is used to map the output of the decoder layer to an integer in the vocabulary V using an embedding-to-word network, and then calculate the SOFTMAX probability through the softmax function, and finally sample words from the vocabulary V according to the SOFTMAX probability to predict the bounding box sequence.

[0019] In one embodiment, a transformer decoder is used to decode the tracking mark encoding feature and the search image encoding feature; a tracking prediction head is used to process the decoding result to obtain a target tracking result; the target tracking result is a bounding box sequence, including: The tracking marker encoding features and the search image encoding features are input into the first encoder layer of the transformer decoder. The output of the first encoder layer is processed by the tracking prediction head to obtain the SOFTMAX probability of the four bounding box values ​​of the target.

[0020] If the SOFTMAX probability of the four bounding box values ​​of the target is not greater than the preset threshold, the output of the first decoder layer is used as the input of the second decoder layer to continue decoding and tracking result judgment.

[0021] If the SOFTMAX probability of the four bounding box values ​​of the target is greater than the preset threshold, the forward propagation process of the transformer decoder is terminated and the tracking result is output.

[0022] A visual target tracking device based on efficient sequence generation, the device comprising: The input module is used to obtain the search image, template image and randomly initialized tracking markers.

[0023] The position encoding module is used to block and linearly project the search image and the template image respectively to obtain the search visual embedding and the template visual embedding; the search visual embedding, the template visual embedding and the randomly initialized tracking mark are connected and position encoded to obtain the input mark after position encoding.

[0024] The encoding module is used to use an encoder based on the visual transformer framework to perform global self-attention operations on all input tags after position encoding to obtain tracking tag encoding features and search image encoding features.

[0025] The decoding module is used to process the tracking mark coding features using the tracking prediction head to obtain the target tracking prediction result; if the target tracking prediction result meets the preset conditions, the target tracking prediction result is used as the target tracking result; if the target tracking prediction result does not meet the preset conditions, the transformer decoder is used to decode the tracking mark coding features and the search image coding features; the tracking prediction head is used to process the decoding results to obtain the target tracking result; the target tracking result is a bounding box sequence.

[0026] The above-mentioned visual target tracking method, device and electronic device based on efficient sequence generation, the method includes: obtaining a search image, a template image and a randomly initialized tracking mark; dividing the search image and the template image into blocks and linearly projecting them respectively to obtain a search visual embedding and a template visual embedding; connecting the search and template visual embeddings and the tracking mark and performing position encoding; encoding the obtained position encoding results using an encoder based on a visual transformer framework, and processing the obtained tracking mark encoding features using a tracking prediction head to obtain a target tracking prediction result; if the target tracking prediction result meets the preset conditions, the target tracking prediction result is used as the target tracking result; if the target tracking prediction result does not meet the preset conditions, the tracking mark encoding features and the search image encoding features are decoded using a transformer decoder; processing the decoding results using a tracking prediction head to obtain a target tracking result; the target tracking result is a bounding box sequence. The use of this method improves the speed of forward reasoning and the tracking speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a flow chart of a visual target tracking method based on efficient sequence generation in one embodiment; Figure 2 A flow chart of a visual target tracking method based on efficient sequence generation in another embodiment; Figure 3 is a schematic diagram of the structure of a decoder layer in another embodiment; Figure 4 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] In one embodiment, Figure 1 As shown, a visual target tracking method based on efficient sequence generation is provided, and the method comprises the following steps: Step 100: Obtain a search image, a template image, and randomly initialized tracking markers.

[0030] Specifically, get a search image , a template image and four randomly initialized tracking markers .

[0031] Step 102: The search image and the template image are divided into blocks and linearly projected to obtain the search visual embedding and the template visual embedding.

[0032] Specifically, the search image and the target image are divided into blocks respectively to obtain the search image blocks and template image block , in Indicates the block size, Indicates the number of blocks.

[0033] After the linear projection matrix Function, search image block and template image block Project to and . Using VIT The token initialization method randomly initializes the tracking tokens to .

[0034] Step 104: Connect the search visual embedding, the template visual embedding, and the randomly initialized tracking marker and perform position encoding to obtain a position-encoded input marker.

[0035] Specifically, the tracking markers, templates, and search image features are concatenated along the spatial dimension and input into the encoder after adding the position encoding. The input of the encoder is expressed as: ,in is a positional encoding.

[0036] Step 106: Use an encoder based on the visual transformer framework to perform a global self-attention operation on all input tags after position encoding to obtain tracking tag encoding features and search image encoding features.

[0037] Specifically, during the forward propagation process, the encoder performs a global self-attention operation on all input tokens, which can be decomposed into three different cross-correlation operations: , and .in Implement the interaction between template features and search features, and Realize the interaction between tracking markers, templates, and search features. and search image features Output to the decoder.

[0038] The four learnable tracking tokens corresponding to the four bounding box values ​​(X, Y, W, H) of the target, the template token and the search token embedded in the template and search area are input into the encoder together. The tracking markers are refined while feature extraction is in progress. These tracking markers are similar to the concept of anchor points in the target detection task and can preliminarily indicate the target location.

[0039] The encoder based on the visual transformer framework can adopt but is not limited to the ViT-B structure (An image isworth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.).

[0040] Step 108: Process the tracking mark coding features with a tracking prediction head to obtain a target tracking prediction result; if the target tracking prediction result meets the preset conditions, the target tracking prediction result is used as the target tracking result; if the target tracking prediction result does not meet the preset conditions, the transformer decoder is used to decode the tracking mark coding features and the search image coding features; the tracking prediction head is used to process the decoding results to obtain the target tracking result; the target tracking result is a bounding box sequence.

[0041] Specifically, the encoder will directly input the tracking tag encoding features into the embedding-to-word network to predict the target sequence. When the tracking result meets the preset conditions, the entire decoder structure will be omitted. When the tracking effect does not meet the requirements, it will be input into the decoder for layer-by-layer operation.

[0042] The transformer decoder is used to regress and predict four bounding box values ​​from four tracking markers and search image features at one time. The decoder only needs to perform a forward propagation operation once to obtain the four bounding box values ​​of the target, which greatly improves the speed of forward reasoning.

[0043] This method is used in an efficient sequence generation tracking framework for single object tracking, which better initializes tracking markers and generates bounding box sequences in one go.

[0044] The process of online tracking of the target using the trained network model (the network model proposed in the method of this application) includes: during online tracking, the encoder receives the search image, the initial template image, the dynamic template image and four randomly initialized tracking markers, inputs the calculated tracking markers and search image features into the decoder, and predicts the target sequence in parallel. At the same time, following the SeqTrack operation, the online template update interval is set to 1, and the judgment threshold in Algorithm 1 Set to 1.6; for the target sequence predicted by the encoder, the final tracking result is obtained based on the Hamming window combined position prior.

[0045] This method is mainly an improvement on the classic sequence generation method SeqTrack. After the search features, template features and randomly initialized tracking markers are connected and positionally encoded, they are input into the encoder to distinguish them from SeqTrack, and can better initialize tracking markers and improve tracking accuracy. Compared with the sequence generation method that generates the target sequence one by one each time (generating x first, then y, then w, then h), this method can predict the output at one time.

[0046] In the above-mentioned visual target tracking method based on efficient sequence generation, the method includes: obtaining a search image, a template image and a randomly initialized tracking mark; dividing the search image and the template image into blocks and linearly projecting them respectively to obtain a search visual embedding and a template visual embedding; connecting the search and template visual embeddings and the tracking mark and performing position encoding; encoding the obtained position encoding results using an encoder based on a visual transformer framework, and processing the obtained tracking mark encoding features using a tracking prediction head to obtain a target tracking prediction result; if the target tracking prediction result meets the preset conditions, the target tracking prediction result is used as the target tracking result; if the target tracking prediction result does not meet the preset conditions, the tracking mark encoding features and the search image encoding features are decoded using a transformer decoder; processing the decoding results using a tracking prediction head to obtain a target tracking result; the target tracking result is a bounding box sequence. The use of this method improves the speed of forward reasoning and the tracking speed.

[0047] In one embodiment, the process of randomly initializing the tracking mark in step 100 includes: using the VIT The token initialization method randomly initializes the tracking token.

[0048] In one embodiment, step 104 includes: concatenating the search visual embedding, the template visual embedding, and the randomly initialized tracking mark and performing position encoding to obtain the position encoded input mark:

[0049] in, is the position-encoded input token, are randomly initialized tracking markers, To search for visual embeddings of images, is the visual embedding of the template image; is the positional encoding, is the set of real numbers, For the dimension, is the number of blocks.

[0050] In one embodiment, step 106 includes: inputting the position-encoded input mark into an encoder based on a visual transformer framework, realizing the interaction between template features and search features, the interaction between tracking marks and search features, and the interaction between tracking marks and template features, and outputting the encoding results corresponding to the tracking marks and search images in the last layer output of the encoder to the decoder.

[0051] In one embodiment, the transformer decoder in step 108 includes layer decoder layer; M is an integer greater than 0 and less than or equal to 6; Figure 3 As shown, each decoder layer contains an unmasked self-attention block, a multi-head attention block and a feed-forward network (FFN) block; in the first decoder layer: the tracking marker encoded features interact with each other through an unmasked multi-head attention mechanism as a query, and the interaction between the query and search image features is performed based on the multi-head attention block, and then processed by the feed-forward network block and output to the tracking prediction head to obtain the bounding box of the target.

[0052] Specifically, the transformer decoder consists of The encoder output is composed of a decoder layer, each of which contains an unmasked self-attention block, a multi-head attention block, and a feed-forward network (FFN) block. As a query, it is executed based on the multi-head attention block Interactive operation, one-time learning to predict the bounding box of the target. Unlike SeqTrack, which randomly initializes the tracking markers, the tracking markers of this method are initialized in advance in the encoder, which helps the decoder to predict the bounding box of the target with high accuracy. Preferably, the number of decoder layers of the transformer decoder is set to 2.

[0053] Starting from the second decoder layer, the output of the previous decoder layer is the input of the next decoder layer.

[0054] In one of the embodiments, a transformer decoder is used to decode the tracking mark coding features and the search image coding features; a tracking prediction head is used to process the decoding results to obtain a target tracking result; the target tracking result is a bounding box sequence, including: inputting the tracking mark coding features and the search image coding features into the transformer decoder for decoding; inputting the obtained decoding results into the tracking prediction head to obtain a bounding box sequence; the tracking prediction head is used to map the output of the decoder layer to an integer in the vocabulary V using an embedding-to-word network, and then calculating the SOFTMAX probability through a softmax function, and finally sampling words from the vocabulary V according to the SOFTMAX probability to predict the bounding box sequence.

[0055] Specifically, the tracking prediction head includes an embedding-to-word network. The embedding-to-word network is added to the last layer of the decoder as the tracking prediction head. The embedding-to-word network contains a multilayer perceptron (FCN) and a softmax function, which can predict the probability of the four bounding box values ​​of the target based on the tracking mark output by the decoder, and then output the target bounding box.

[0056] Each coordinate of the object bounding box [x, y, w, h] is uniformly discretized into the vocabulary The embedding-to-word network maps the high-dimensional tracking token output by the decoder to an integer between [1, nbins], corresponding to the word in V. The embedding-to-word network is implemented by a multilayer perceptron (FCN) and a softmax function (softmax). FCN maps the tracking token Mapping to vocabulary In between The probability is calculated by the softmax function. Finally, we sample words from the vocabulary according to the SOFTMAX probability to predict the target box b=[x, y, w, h]. The softmax probability here is also the basis for judging whether the tracking effect is reliable.

[0057] In one of the embodiments, a transformer decoder is used to decode the tracking mark coding features and the search image coding features; a tracking prediction head is used to process the decoding results to obtain a target tracking result; the target tracking result is a bounding box sequence, including: inputting the tracking mark coding features and the search image coding features into the first encoder layer of the transformer decoder, and processing the output of the first encoder layer through the tracking prediction head to obtain the SOFTMAX probability of the four target bounding box values; if the SOFTMAX probability of the four target bounding box values ​​is not greater than a preset threshold, the output of the first decoder layer is used as the input of the second decoder layer, and the decoding and tracking result judgment are continued; if the SOFTMAX probability of the four target bounding box values ​​is greater than the preset threshold, the forward propagation process of the transformer decoder is terminated and the tracking result is output.

[0058] Specifically, in addition, in order to further improve the tracking efficiency, an early exit mechanism is introduced in the decoder part of the network. The tracking effect is judged layer by layer after each layer of the decoder. When the tracking result is reliable enough, the forward propagation process of the decoder is terminated in advance and the tracking result is output.

[0059] The early exit mechanism is seamlessly integrated into the transformer decoder to terminate the forward propagation process in time according to the prediction accuracy, which further improves the tracking speed.

[0060] The process of visual target tracking method based on efficient sequence generation is as follows Figure 2As shown. After each decoder layer, an embedding-to-word network is added to calculate the average softmax probability of the four bounding box values. If the average probability exceeds a certain threshold , it is determined that the predicted bounding box is reliable enough, then the forward propagation process of the decoder is terminated and the tracking result is output. The pseudo code of the early exit mechanism is shown in Algorithm 1. It should be noted that the parameters of the embedding-to-word network (embedded into the word network) used after each decoder layer are shared, avoiding the increase of tracking model parameters. The embedding-to-word network includes multiple layers of perceptrons and a softmax layer.

[0061] Algorithm 1: Early exit mechanism in decoder Input: Search image encoding features , tracking marker encoding features ; Parameters: Layer l decoder ; Embedding-to-word networks; Output: target box ; Step 1: Input ; Early exit flag = False Step 2: For each decoder layer from 1 to N , loop the following operations; Step 3: Calculate the Layer Output ; Step 4: Calculate the target box and the confidence of the target box based on embedding-to-word ; Step 5: If the confidence is greater than the threshold, ,but: Step 6: Early exit flag = True; Step 7: Return to the target frame ; Step 8: End; Step 9: End; Step 10: If all decoder layers are calculated and the early exit flag = False, then: Step 11: Return to the target frame ; Step 12: End.

[0062] The tracking network architecture (referred to as FastSeqTrack model) used in the method proposed in this application consists of four parts: a) an encoder for joint feature extraction, feature fusion and refinement of tracking markers; b) a decoder for searching image features and tracking marker interactions; c) a tracking prediction head based on an embedding-to-word network; d) an early exit mechanism.

[0063] 1) The loss function during FastSeqTrack model training combines cross entropy loss and generalized IoU loss:

[0064] in, represents the target ground-truth bounding box, express The bounding boxes predicted by the decoder layer, are the hyperparameters of the cross entropy and generalized IOU loss functions, is the set of real numbers, Indicates the number of training samples.

[0065] 2) Training settings. The training data includes training data from COCO, LaSOT, GOT-10k, TrackingNet, and VastTrack datasets. The training parameters follow the parameter settings in the SeqTrack tracking method. The FastSeqTrack model is trained with 60k image pairs for a total of 500 epochs. After 400 epochs, the learning rate is reduced by 10 times. As a preference, the vocabulary In Set to 4000, Set to 1 and 5 respectively.

[0066] Our method runs 3x faster than SeqTrack, exceeding 100 fps, and demonstrates state-of-the-art performance on multiple tracking benchmarks.

[0067] The early exit mechanism is an optimization measure proposed in this application for improving the tracking speed; the early exit mechanism is seamlessly integrated into the transformer decoder, and the forward propagation process is terminated in time according to the prediction accuracy, thereby further improving the tracking speed.

[0068] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0069] In one embodiment, a visual target tracking device based on efficient sequence generation is provided, comprising: an input module, a position encoding module, an encoding module and a decoding module, wherein: The input module is used to obtain the search image, template image and randomly initialized tracking markers.

[0070] The position encoding module is used to block and linearly project the search image and the template image respectively to obtain the search visual embedding and the template visual embedding; the search visual embedding, the template visual embedding and the randomly initialized tracking mark are connected and position encoded to obtain the input mark after position encoding.

[0071] The encoding module is used to use an encoder based on the visual transformer framework to perform global self-attention operations on all input tags after position encoding to obtain tracking tag encoding features and search image encoding features.

[0072] The decoding module is used to process the tracking mark coding features using the tracking prediction head to obtain the target tracking prediction result; if the target tracking prediction result meets the preset conditions, the target tracking prediction result is used as the target tracking result; if the target tracking prediction result does not meet the preset conditions, the transformer decoder is used to decode the tracking mark coding features and the search image coding features; the tracking prediction head is used to process the decoding results to obtain the target tracking result; the target tracking result is a bounding box sequence.

[0073] In one embodiment, the process of randomly initializing the tracking flags in the input module includes: using the VIT The token initialization method randomly initializes the tracking token.

[0074] In one of the embodiments, the position encoding module is also used to connect the search visual embedding, the template visual embedding and the randomly initialized tracking mark and perform position encoding to obtain the input mark after position encoding; the input mark is shown in the expression of the input mark after position encoding mentioned above.

[0075] In one of the embodiments, the encoding module is also used to input the position-encoded input mark into an encoder based on a visual transformer framework, to achieve interaction between template features and search features, interaction between tracking marks and search features, interaction between tracking marks and template features, and to output the encoding results corresponding to the tracking marks and search images in the last layer output of the encoder to the decoder.

[0076] In one embodiment, the transformer decoder in the decoding module includes layer decoder layer; each decoder layer contains an unmasked self-attention block, a multi-head attention block and a feed-forward network (FFN) block; in the first decoder layer: the tracking marker encoded features interact with each other through an unmasked multi-head attention mechanism as a query, and the interaction between the query and search image features is performed based on the multi-head attention block, and then processed by the feed-forward network block and output to the tracking prediction head to obtain the bounding box of the target.

[0077] In one of the embodiments, the decoding module is further used to input the tracking mark encoding features and the search image encoding features into the transformer decoder for decoding; the obtained decoding results are input into the tracking prediction head to obtain a bounding box sequence; the tracking prediction head is used to use the embedding-to-word network to map the output of the decoder layer to an integer in the vocabulary V, and then calculate the SOFTMAX probability through the softmax function, and finally sample the word from the vocabulary V according to the SOFTMAX probability to predict the bounding box sequence.

[0078] In one of the embodiments, the decoding module is further used to input the tracking mark coding features and the search image coding features into the first encoder layer of the transformer decoder, and the output of the first encoder layer is processed by the tracking prediction head to obtain the SOFTMAX probability of the four target bounding box values; if the SOFTMAX probability of the four target bounding box values ​​is not greater than a preset threshold, the output of the first decoder layer is used as the input of the second decoder layer, and the decoding and tracking result judgment are continued; if the SOFTMAX probability of the four target bounding box values ​​is greater than the preset threshold, the forward propagation process of the transformer decoder is terminated and the tracking result is output.

[0079] For the specific limitations of the visual target tracking device based on efficient sequence generation, please refer to the limitations of the visual target tracking method based on efficient sequence generation above, which will not be repeated here. The various modules in the above-mentioned visual target tracking device based on efficient sequence generation can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a visual target tracking method based on efficient sequence generation is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0081] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0082] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0083] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A visual target tracking method based on efficient sequence generation, characterized in that: The method comprises: Get the search image, template image, and randomly initialized tracking markers; The search image and the template image are divided into blocks and linearly projected to obtain a search visual embedding and a template visual embedding; The search visual embedding, the template visual embedding and the randomly initialized tracking mark are connected and position-encoded to obtain a position-encoded input mark; The encoder based on the visual transformer framework is used to perform global self-attention operations on all input tags after position encoding to obtain tracking tag encoding features and search image encoding features; The tracking mark coding feature is processed by a tracking prediction head to obtain a target tracking prediction result; If the target tracking prediction result meets the preset condition, the target tracking prediction result is used as the target tracking result; If the target tracking prediction result does not meet the preset conditions, a transformer decoder is used to decode the tracking mark coding features and the search image coding features; a tracking prediction head is used to process the decoding results to obtain a target tracking result; and the target tracking result is a bounding box sequence.

2. The visual target tracking method based on efficient sequence generation according to claim 1, characterized in that: The process of randomly initializing the tracking mark includes: using the VIT The token initialization method randomly initializes the tracking token.

3. The visual target tracking method based on efficient sequence generation according to claim 1, characterized in that: The search visual embedding, the template visual embedding and the randomly initialized tracking mark are connected and position encoded to obtain the position encoded input mark: ； in, is the position-encoded input token, are randomly initialized tracking markers, is the set of real numbers, For the dimension, To search for visual embeddings of images, is the visual embedding of the template image, is the number of blocks, is a positional encoding.

4. The visual target tracking method based on efficient sequence generation according to claim 1, characterized in that: The encoder based on the visual transformer framework is used to perform global self-attention operations on all input tags after position encoding to obtain tracking tag encoding features and search image encoding features, including: The position-encoded input mark is input into the encoder based on the visual transformer framework to realize the interaction between template features and search features, the interaction between tracking marks and search features, and the interaction between tracking marks and template features. The encoding results corresponding to the tracking marks and search images in the last layer output of the encoder are output to the decoder.

5. The visual target tracking method based on efficient sequence generation according to claim 1, characterized in that: The transformer decoder includes layer decoder layers; each decoder layer contains an unmasked multi-head self-attention block, a multi-head attention block and a feedforward network block; in the first decoder layer: The tracking marker encoding features interact with each other through an unmasked multi-head attention mechanism as a query, and the interaction between the query and search image features is performed based on the multi-head attention block, and then processed by the feedforward network block and output to the tracking prediction head to obtain the target's bounding box.

6. The visual target tracking method based on efficient sequence generation according to claim 1, characterized in that: Decoding the tracking mark encoding feature and the search image encoding feature using a transformer decoder; The decoding result is processed using a tracking prediction head to obtain a target tracking result; the target tracking result is a bounding box sequence, including: Inputting the tracking mark encoding feature and the search image encoding feature into a transformer decoder for decoding; The obtained decoding result is input into the tracking prediction head to obtain a bounding box sequence; the tracking prediction head is used to map the output of the decoder layer to an integer in the vocabulary V using an embedding-to-word network, and then calculate the SOFTMAX probability through a softmax function, and finally sample words from the vocabulary V according to the SOFTMAX probability to predict the bounding box sequence.

7. The visual target tracking method based on efficient sequence generation according to claim 5, characterized in that: Decoding the tracking mark encoding feature and the search image encoding feature using a transformer decoder; The decoding result is processed using a tracking prediction head to obtain a target tracking result; the target tracking result is a bounding box sequence, including: Inputting the tracking mark coding features and the search image coding features into the first encoder layer of the transformer decoder, and processing the output of the first encoder layer through the tracking prediction head to obtain the SOFTMAX probability of the four bounding box values ​​of the target; If the SOFTMAX probability of the four bounding box values ​​of the target is not greater than the preset threshold, the output of the first decoder layer is used as the input of the second decoder layer to continue decoding and tracking result judgment; If the SOFTMAX probability of the four bounding box values ​​of the target is greater than the preset threshold, the forward propagation process of the transformer decoder is terminated and the tracking result is output.

8. A visual target tracking device based on efficient sequence generation, characterized in that: The device comprises: Input module, used to obtain the search image, template image and randomly initialized tracking markers; A position encoding module is used to perform block division and linear projection on the search image and the template image respectively to obtain a search visual embedding and a template visual embedding; the search visual embedding, the template visual embedding and a randomly initialized tracking mark are connected and then position encoded to obtain an input mark after position encoding; The encoding module is used to perform global self-attention operation on all input tags after position encoding using an encoder based on the visual transformer framework to obtain tracking tag encoding features and search image encoding features; A decoding module is used to process the tracking mark coding feature using a tracking prediction head to obtain a target tracking prediction result; if the target tracking prediction result meets a preset condition, the target tracking prediction result is used as the target tracking result; if the target tracking prediction result does not meet the preset condition, a transformer decoder is used to decode the tracking mark coding feature and the search image coding feature; according to the decoding result, a tracking prediction head is used to process the target tracking result; the target tracking result is a bounding box sequence.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the visual target tracking method based on efficient sequence generation according to any one of claims 1 to 7 is implemented.

Images