Training method and device of visual target tracking model, tracking method and device

Abstract

The present disclosure relates to the field of visual target tracking and artificial intelligence, and provides a training method and device for a visual target tracking model, and a tracking method and device. The training method includes: obtaining multiple pairs of training samples; inputting the training samples into a feature embedding module to obtain template image features and search image features; using a feature extraction module to perform feature extraction operations to obtain a first extraction result and a second extraction result; using a feature fusion module to fuse the first extraction result and the second extraction result to obtain a fusion result; using a prediction head to perform prediction based on the fusion result to obtain a predicted tracking result; obtaining a training loss based on the predicted tracking result, true annotation information, and distilled annotation information, and training the model to obtain a trained visual target tracking model. The present disclosure solves the problem that the existing single-stream tracker has certain computational redundancy and limits the inference speed, and improves the accuracy of the model while retaining a high inference speed.

Description

Technical Field

[0001] The present disclosure relates to the fields of visual target tracking and artificial intelligence, and in particular to a training method and device for a visual target tracking model, and a tracking method and device. Background Art

[0002] Visual single target tracking is a basic task in computer vision. Taking visual tracking in video as an example, given the first frame image and the corresponding target bounding box in a video, it is required to predict the target bounding box in subsequent frames. Target tracking has important applications in machine vision, intelligent monitoring, unmanned driving and other fields.

[0003] In the related art, there are two-stream architecture and single-stream architecture. The two-stream architecture, as a traditional tracker architecture, has two branches, the template branch and the search branch, which are used to extract the features of the template image and the search image respectively, and predict the target bounding box and confidence in the search image based on the features of the two. The single-stream architecture, as an emerging tracker architecture, can use the data form of tokens based on the Transformer architecture, and can splice the tokens of the template image and the search image, and process them at one time. It is usually more accurate than the two-stream structure.

[0004] Although the existing single-stream tracker has an advantage in accuracy over the dual-stream tracker, it has a certain computational redundancy because the template features need to be recalculated for each frame, which limits the inference speed. Summary of the invention

[0005] The present disclosure provides a training method and device for a visual target tracking model, and a tracking method and device, to at least solve the problem that the existing single-stream tracker has certain computational redundancy and limits the reasoning speed. The present disclosure combines the advantages of the single-stream structure and the dual-stream architecture, and proposes a dual-stream tracker with a messenger token, which improves the speed and accuracy at the same time. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a training method for a visual target tracking model is provided, wherein the visual target tracking model comprises a feature embedding module, an information integration module, a feature extraction module, a feature fusion module and a prediction head, wherein the feature extraction module comprises a plurality of layers, wherein the training method comprises: obtaining a plurality of pairs of training samples, wherein each pair of training samples comprises a template sample image, a search sample image and sample annotation information, wherein the template sample image contains a visual target to be tracked, and the sample annotation information represents a bounding box of the visual target in the search sample image, wherein the sample annotation information comprises real annotation information and distilled annotation information; inputting the training sample into the feature embedding module to obtain a template image feature and a search image feature; utilizing the layers of the feature extraction module to perform a feature extraction operation on a first feature and a second feature to obtain a first extraction result and a second extraction result, wherein the first feature comprises the template image feature and a first messenger feature, and the second feature comprises the search image feature, the second messenger feature, and the second messenger feature. The invention relates to a method for extracting a visual target image of a search sample image, wherein the first messenger feature and the second messenger feature both include learnable parameters, the first messenger feature and the second messenger feature are the same, and the target token and the distilled token both include learnable parameters, wherein for two adjacent layers of the feature extraction module, the information integration module is used to determine the input features of the latter layer corresponding to the second messenger feature based on the output features of the previous layer corresponding to the first messenger feature and the output features corresponding to the second messenger feature; the feature fusion module is used to fuse the first extraction result and the second extraction result to obtain a fusion result; the prediction head is used to perform prediction based on the fusion result to obtain a predicted tracking result, wherein the predicted tracking result represents the bounding box of the visual target in the template sample image in the search sample image; the training loss is obtained based on the predicted tracking result, the real annotation information and the distilled annotation information, and the visual target tracking model is trained to obtain a trained visual target tracking model.

[0007] Optionally, the input of each layer of the feature extraction module includes a first input feature and a second input feature, and the output of each layer of the feature extraction module includes a first output feature and a second output feature, the first output feature includes a template image extraction feature and a first messenger extraction feature, and the second output feature includes a search image extraction feature and a second messenger extraction feature, wherein for each layer of the feature extraction module, the feature extraction operation includes: taking the first output feature output by the previous layer as the first input feature of the current layer, inputting it into the current layer, and obtaining the first output feature of the current layer; using the information integration module, integrating the first messenger extraction feature output by the previous layer with the second messenger extraction feature output by the previous layer, and obtaining a new messenger extraction feature; using the new messenger extraction feature to replace the second messenger extraction feature in the second output feature output by the previous layer, and obtaining a new second output feature; taking the new second output feature as the second input feature of the current layer, and inputting it into the current layer, and obtaining the second output feature of the current layer, wherein the first input feature and the second input feature of the first layer of the feature extraction module are the first feature and the second feature, respectively, and the first output feature and the second output feature of the last layer of the feature extraction module include the first extraction result and the second extraction result, respectively.

[0008] Optionally, the information integration module obtains the new messenger extraction feature by one of the following methods: directly using the second messenger extraction feature as the new messenger extraction feature; adding the first messenger extraction feature to the second messenger extraction feature to obtain the new messenger extraction feature; directly using the first messenger extraction feature as the new messenger extraction feature.

[0009] Optionally, the distilled annotation information is determined based on a teacher prediction result obtained by a pre-trained teacher model for predicting the training sample, and the predicted tracking result includes a target prediction result corresponding to the target token and a distilled prediction result corresponding to the distilled token, wherein the visual target tracking model is trained in the following manner: a first loss is obtained by comparing the target prediction result and the true annotation information; a second loss is obtained by comparing the distilled prediction result and the distilled annotation information; a prediction loss is determined based on the first loss and the second loss; and the parameters of the visual target tracking model are adjusted using the prediction loss to train the visual target tracking model.

[0010] Optionally, the real annotation information includes a real bounding box, the distilled annotation information includes a distilled bounding box, and the teacher prediction result includes a teacher bounding box, wherein the distilled bounding box is obtained in the following manner: based on a smoothing coefficient, the real bounding box and the teacher bounding box are weighted and summed to obtain the distilled bounding box, wherein the smoothing coefficient represents the proportion of the real bounding box in the distilled bounding box.

[0011] Optionally, the visual target tracking model also includes a fully connected layer and a sigmoid layer, wherein the smoothing coefficient is determined in the following manner: inputting the feature corresponding to the target token in the fusion result into the fully connected layer to obtain the fully connected feature; inputting the fully connected feature into the sigmoid layer to obtain the smoothing coefficient, wherein the smoothing coefficient is a scalar in the range of [0,1].

[0012] According to a second aspect of the present disclosure, a visual target tracking method is provided, the visual target tracking method comprising: acquiring a target template image and a target search image, wherein the target template image contains a visual target to be tracked; inputting the target template image and the target search image into a visual target tracking model, and using the visual target tracking model to predict a bounding box of the visual target in the target search image, wherein the visual target tracking model is trained according to the training method of the visual target tracking model described in the embodiments of the present disclosure.

[0013] According to a third aspect of the present disclosure, a training system for a visual target tracking model is provided, wherein the visual target tracking model comprises a feature embedding module, an information integration module, a feature extraction module, a feature fusion module and a prediction head, wherein the feature extraction module comprises a plurality of layers, wherein the training system comprises: a sample acquisition unit, configured to acquire a plurality of pairs of training samples, wherein each pair of training samples comprises a template sample image, a search sample image and sample annotation information, wherein the template sample image contains a visual target to be tracked, and the sample annotation information represents a bounding box of the visual target in the search sample image, wherein the sample annotation information comprises real annotation information and distilled annotation information; a first determination unit, configured to input the training sample into the feature embedding module to obtain a template image feature and a search image feature; a second determination unit, configured to perform a feature extraction operation on a first feature and a second feature using the layers of the feature extraction module to obtain a first extraction result and a second extraction result, wherein the first feature comprises the template image feature and the first messenger feature, and the second feature comprises the search image feature, the second messenger feature, and the second messenger feature. Features, target tokens and distilled tokens, the first messenger feature and the second messenger feature are both learnable parameters, the first messenger feature and the second messenger feature are the same, the target token and the distilled token both include learnable parameters, wherein, for two adjacent layers of the feature extraction module, the information integration module is used to determine the input features of the latter layer corresponding to the second messenger feature based on the output features of the previous layer corresponding to the first messenger feature and the output features corresponding to the second messenger feature; a sample fusion unit is configured to use the feature fusion module to fuse the first extraction result and the second extraction result to obtain a fusion result; a sample prediction unit is configured to use the prediction head to predict based on the fusion result to obtain a predicted tracking result, wherein the predicted tracking result represents the bounding box of the visual target in the template sample image in the search sample image; a training unit is configured to obtain a training loss based on the predicted tracking result, the real annotation information and the distilled annotation information, and train the visual target tracking model to obtain a trained visual target tracking model.

[0014] According to a fourth aspect of the present disclosure, a visual target tracking system is provided, comprising: an image acquisition unit, configured to acquire a target template image and a target search image, wherein the target template image contains a visual target to be tracked; and a visual prediction unit, configured to input the target template image and the target search image into a visual target tracking model, and predict a bounding box of the visual target in the target search image using the visual target tracking model, wherein the visual target tracking model is trained according to the training method of the visual target tracking model described in the present disclosure.

[0015] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the instructions executable by the processor, when executed by the processor, cause the processor to execute the training method of the visual target tracking model or the visual target tracking method according to an embodiment of the present disclosure.

[0016] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the visual target tracking model or the visual target tracking method according to the embodiment of the present disclosure.

[0017] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which, when executed by at least one processor, implement the training method of the visual target tracking model or the visual target tracking method according to the embodiments of the present disclosure.

[0018] The technical solution provided by the present disclosure brings at least the following beneficial effects:

[0019] By adopting the scheme disclosed in the present invention, messenger features with learnable parameters can be introduced respectively for the template image features and search image features obtained by the feature embedding module of the visual target tracking model, so that the corresponding messenger features can learn the information of the corresponding image feature extraction branch. In the feature extraction process, the introduction of such messenger features can improve the effect of feature extraction, so that the model can more accurately capture the visual information in the template image, thereby retaining the advantage of high inference speed of the dual-branch architecture that extracts features from the template image and the search image respectively, and can also improve the inference accuracy of the model.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0022] Figure 1 is a schematic flowchart of a training method for a visual target tracking model according to an exemplary embodiment of the present disclosure.

[0023] Figure 2 is a schematic block diagram of a visual target tracking model according to an exemplary embodiment of the present disclosure.

[0024] Figure 3 is a schematic block diagram of a feature extraction module in a visual target tracking model according to an exemplary embodiment of the present disclosure.

[0025] Figure 4 is a schematic flowchart of steps for performing feature extraction operations in a training method for a visual target tracking model according to an exemplary embodiment of the present disclosure.

[0026] Figure 5 is a schematic flowchart of a visual target tracking method according to an exemplary embodiment of the present disclosure.

[0027] Figure 6 is a schematic block diagram of a training system for a visual target tracking model according to an exemplary embodiment of the present disclosure.

[0028] Figure 7 is a schematic block diagram of a visual target tracking system according to an exemplary embodiment of the present disclosure.

[0029] Figure 8 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the present application will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0033] Starting from the SiamFC model, trackers based on twin structures have quickly become popular due to their simple structure and fast inference speed. Influenced by this, discriminative trackers also absorb its advantages. These trackers are typical two-stream structures, that is, they have two branches: template branch and search branch. The reason why these trackers use two branches is mainly due to structural limitations. Early trackers usually use two-dimensional convolutional networks to extract image features. In this case, in order to maintain the two-dimensional spatial structure of the image, only two branches can be used to extract features from the template image and the search image respectively.

[0034] Since the introduction of the ViT (Vision Transformer) model, the Transformer architecture has gradually become popular in the field of target tracking. It has been used in the feature fusion stage and feature extraction stage of the tracker. The trackers in these two periods still belong to the dual-stream structure. Since the Transformer architecture uses the data form of tokens, the tokens of the template image and the search image can be spliced ​​and processed at one time. This tracker belongs to the single-stream structure. In this regard, models such as OSTrack, MixFormer, and SimTrack first adopted the single-stream structure. They merged the feature extraction process and feature fusion process of the two branches to achieve accuracy beyond the dual-stream tracker. Since then, single-stream trackers have gradually become popular.

[0035] The single-stream tracker enables the search image and the template image to interact with each other earlier and dynamically extract target-oriented discriminative features. However, the single-stream tracker needs to re-extract the features of the template image for each inference, resulting in additional computational overhead and slow computation speed.

[0036] In view of the above problems, exemplary embodiments of the present disclosure propose a training method for a visual target tracking model, a visual target tracking method, a training system for a visual target tracking model, a visual target tracking system, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0037] In a first aspect of an exemplary embodiment of the present disclosure, a method for training a visual target tracking model is provided.

[0038] The training method of the visual target tracking model according to the exemplary embodiment of the present disclosure can be applied to scenarios where users interact with software. For example, the software can be loaded on the user terminal, and the user can input model training instructions on the user terminal. The user terminal can obtain the final visual target tracking model by executing the training method of the visual target tracking model according to the exemplary embodiment of the present disclosure.

[0039] Specifically, the visual target tracking model includes a feature embedding module, an information integration module, a feature extraction module, a feature fusion module and a prediction head, and the feature embedding module includes multiple layers.

[0040] The user terminal can obtain multiple pairs of training samples and sample annotation information of each pair of training samples, wherein each pair of training samples includes a template sample image and a search sample image, the template sample image contains a visual target to be tracked, and each sample annotation information represents the bounding box of the visual target in the search sample image.

[0041] The user terminal can input the training sample into the feature embedding module to obtain the template image feature and the search image feature; use the layers of the feature extraction module to perform feature extraction operations on the first feature and the second feature to obtain the first extraction result and the second extraction result, wherein the first feature includes the template image feature and the first messenger feature, the second feature includes the search image feature, the second messenger feature, the target token and the distillation token, the first messenger feature and the second messenger feature both include learnable parameters, the first messenger feature and the second messenger feature are the same, the target token and the distillation token both include learnable parameters, wherein for two adjacent layers of the feature extraction module, the information integration module is used to extract the first and second features. The block can determine the input features of the next layer corresponding to the second messenger features based on the output features of the previous layer corresponding to the first messenger features and the output features corresponding to the second messenger features; the feature fusion module can be used to fuse the first extraction result and the second extraction result to obtain a fusion result; the prediction head can be used to make a prediction based on the fusion result to obtain a predicted tracking result, wherein the predicted tracking result represents the bounding box of the visual target in the template sample image in the search sample image; the loss can be obtained based on the predicted tracking result, the real annotation information and the distilled annotation information, and the visual target tracking model can be trained to obtain a trained visual target tracking model.

[0042] The above-mentioned user terminal can be, for example, a tablet computer, a laptop computer, a digital assistant, a wearable device, etc. However, the implementation scenario of the above method is only an example scenario. The training method of the visual target tracking model according to the exemplary embodiment of the present disclosure can also be applied to other application scenarios. For example, the user can request a training model from the server through the network on the user terminal (for example, a mobile phone, a desktop computer, a tablet computer, etc.). The server can complete the request by executing the training method of the visual target tracking model according to the exemplary embodiment of the present disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform or a virtualization center.

[0043] In the training method of the visual target tracking model of the exemplary embodiment of the present disclosure, during the feature extraction process, the introduction of messenger features with learnable parameters can improve the effect of feature extraction, thereby retaining the advantage of high inference speed of the dual-stream architecture that extracts features from the template image and the search image respectively, while also improving the inference accuracy of the model.

[0044] The following will refer to Figures 1 to 5 An example of a training method for a visual target tracking model according to an embodiment of the present disclosure is described. The visual target tracking model may include a feature embedding module, an information integration module, a feature extraction module, a feature fusion module, and a prediction head.

[0045] like Figure 1 As shown, the training method of the visual target tracking model may include the following steps:

[0046] In step S110, multiple pairs of training samples may be obtained.

[0047] Here, each pair of training samples may include a template sample image, a search sample image, and sample annotation information. The template sample image contains a visual target to be tracked, and the sample annotation information represents the bounding box of the visual target in the search sample image. The sample annotation information may include real annotation information and distilled annotation information.

[0048] In visual target tracking, a template image may be an image containing a bounding box of a visual target to be tracked, such as the first frame of a video, which is used to mark the target that the tracker needs to track. A search image may be an image that needs to track the visual target in the template image, such as each frame after the first frame of a video. A visual tracker (or a visual target tracking model) may, for example, predict a bounding box (also called a bounding box) for the search image. In addition, the visual tracker may also obtain a confidence level, which may represent the reliability level of the prediction, such as the intersection-over-union ratio between the true bounding box and the predicted bounding box.

[0049] In this step, each pair of training samples may include a pair of template sample images and search sample images. The template sample image contains a bounding box for the visual target to be tracked, and the model may predict the bounding box of the visual target in the search sample image.

[0050] As an example, the training sample can be extracted from a video, for example, the template sample image is a certain video frame, and the search sample image can be a video frame that is separated from the template sample image by a predetermined number of frames. However, according to the embodiments of the present disclosure, the training samples can also be obtained by other means, for example, from a pre-built database, and there is no particular limitation on the acquisition method, content, format, etc. of the image itself.

[0051] In step S120, the training sample may be input into a feature embedding module to obtain template image features and search image features.

[0052] The template sample image and the search sample image can be simultaneously passed through a feature embedding module, which can include, for example, but is not limited to, a block division module and a linear projection module, wherein the block division module can divide the sample image into blocks of a predetermined size, such as 16x16 pixel blocks; and the linear projection module can perform linear projection on the divided blocks. In addition, in the feature embedding module, an absolute position code can be added to the image features, for example, an absolute position code can be added to the features obtained by linear projection.

[0053] The feature embedding module can output the template image features (also called template image tokens) based on the template sample image and output the search image features (also called search image tokens) based on the search sample image. In an embodiment of the present disclosure, the image features can be represented by tokens, such as Figure 2 As shown, the feature embedding module can output a template image feature token and a search image feature token.

[0054] As an example, in an embodiment of the present disclosure, the processing of the template image and the processing of the search image can be divided into two branches. The blocking module and the linear projection module in the feature embedding module can be shared by the two branches, and can process both the template image and the search image. However, the absolute position encoding added by the feature embedding module to the template image and the search image is different.

[0055] In step S130, a feature extraction module may be used to perform a feature extraction operation on the first feature and the second feature to obtain a first extraction result and a second extraction result.

[0056] In this step, the first feature may include a template image feature and a first messenger feature, the second feature may include a search image feature, a second messenger feature, a target token, and a distillation token, the first messenger feature and the second messenger feature are the same, and the first messenger feature and the second messenger feature may both be learnable parameters.

[0057] Based on the template image features and the search image features output by the feature embedding module, the first messenger feature and the second messenger feature can be added respectively, so as to obtain the first feature and the second feature. Here, the first messenger feature and the second messenger feature can both be randomly generated feature vectors. Before feature extraction, the two messenger features can be spliced ​​with the template image features and the search image features respectively, so that in the subsequent feature extraction process, the corresponding messenger features and the image features can continuously interact with each other, so that the messenger features contain information from the image, which makes the feature extraction process more information-rich, thereby improving the effect of feature extraction.

[0058] Specifically, in the feature extraction stage, the template branch for extracting features of the template image and the search branch for extracting features of the search image can respectively pass through a shared backbone network, i.e., a feature extraction module, which can include, for example, a ViT model for feature extraction. As an example, information transmission and fusion can be completed between the two branches through messenger features and information integration of the messenger features of the two branches, which will be described in detail in conjunction with the information integration module later.

[0059] In this step, the first messenger feature and the second messenger feature may be added to the template image feature and the search image feature, respectively. The first messenger feature and the second messenger feature may be used to characterize feature information other than the template image feature and the search image feature, respectively. The messenger feature may be, for example, one or more additional tokens, in which case the messenger feature may also be referred to as a messenger token. The messenger feature may have the same form as the image feature so that it can be spliced ​​to the image feature. Its initial parameters are randomly generated. After the training is completed, the learnable parameters of the first messenger feature and the second messenger feature are fixed, and they participate in the reasoning process of the model as initial messenger features.

[0060] For example, both template image features and search image features can be represented by tensors of (N, L, C) shape, where N represents batch size, L represents length, and C represents the number of channels. The messenger features can have the same batch size and number of channels as the image features, and the length can be pre-set. The initial messenger features (such as the first messenger features and the second messenger features mentioned above) can be obtained by initializing the tensor representation with a truncated normal distribution of such a shape.

[0061] By adding this messenger feature, target-specific information in the template image can be extracted in the feature extraction stage and can be passed to the search branch for extracting features of the search image in a subsequent process (e.g., through the information integration module to be described below), making the extracted features more robust.

[0062] As an example, the first messenger feature and the second messenger feature may be the same. Specifically, the first feature and the second feature may be obtained by concatenating the preset initial messenger feature to the template image feature and the search image feature, respectively. In the initial stage of training, the initial messenger feature may be randomly generated, and the initial messenger feature added to the template image feature and the search image feature may be the same; during the training process, the first messenger feature may learn the features of the template sample image, the second messenger feature may learn the features of the search sample image, and may also be integrated with the first messenger feature to receive information from the template sample image, which will be described in detail below.

[0063] In addition, the second feature may also include a target token and a distillation token. Both the initial target token and the distillation token may include user-defined randomly initialized learnable parameters. For example, the template image features and the search image features output by the feature embedding module may be (N, L, C)-shaped data. The second messenger features, target tokens, and distillation tokens may be added along the L dimension of the search image features. The initial target token and distillation token may be different. During the training process, the prediction of the target token is used to make a loss with the true annotation information, and the prediction of the distillation token is used to make a loss with the distillation annotation information. This will be described in detail below.

[0064] For two adjacent layers of the feature extraction module, the information integration module can be used to determine the input features of the latter layer corresponding to the second messenger features based on the output features of the previous layer corresponding to the first messenger features and the output features corresponding to the second messenger features.

[0065] The following will refer to Figure 3 and Figure 4 The following describes an example process in which a feature extraction module performs feature extraction to obtain a first extraction result and a second extraction result. Figure 3As shown, the feature extraction module includes multiple layers, the input of each layer of the feature extraction module includes a first input feature and a second input feature, the output of each layer of the feature extraction module includes a first output feature and a second output feature, the first output feature includes a template image extraction feature and a first messenger extraction feature, and the second output feature includes a search image extraction feature and a second messenger extraction feature. The second output feature may also include a target token and a distillation token. During the feature extraction process, the target token and the distillation token interact with the search image extraction feature and the second messenger feature, so that they are also updated layer by layer in the target token and the distillation token output by the last layer can be used as the tracking results of the tracking target in the search image.

[0066] As an example, Figure 4 As shown, for each layer of the feature extraction module, the feature extraction operation may include: step S410, the first output feature of the previous layer output can be used as the first input feature of the current layer, input to the current layer, to obtain the first output feature of the current layer; step S420, the information integration module can be used to obtain a new messenger extraction feature based on the first messenger extraction feature output by the previous layer and the second messenger extraction feature output by the previous layer; step S430, the new messenger extraction feature can be used to replace the second messenger extraction feature in the second output feature output by the previous layer, to obtain a new second output feature; step S440, the new second output feature can be used as the second input feature of the current layer, input to the current layer, to obtain the second output feature of the current layer.

[0067] Here, the first input feature and the second input feature of the first layer of the feature extraction module are the first feature and the second feature, respectively, and the first output feature and the second output feature of the last layer of the feature extraction module include the first extraction result and the second extraction result, respectively. For example, the template image extraction feature in the first output feature of the last layer can be used as the first extraction result, and the search image extraction feature, the target token, and the distillation token in the second output feature of the last layer can be used as the second extraction result.

[0068] Specifically, for each layer of the feature extraction module, the first output feature can be obtained by taking the first output feature of the previous layer as the first input feature of the current layer to obtain the first output feature of the current layer. Here, each layer of the feature extraction module may include a submodule, and the submodule may be, for example, a block module in the ViT model.

[0069] In addition, in the embodiment of the present disclosure, the feature extraction module may include multiple layers of sub-modules (blocks), and the input and output of each layer are connected to form a multi-layer extraction of image features, that is, the output of the current layer can be the output of the next layer. As the number of layers increases, deeper information can be extracted. In the embodiment of the present disclosure, for the template branch and the search branch, the feature extraction module is shared, and the feature extraction modules used by the two branches have the same structure and parameters.

[0070] like Figure 3 As shown, in the template branch, The layers can be expressed by the following formula (1):

[0071] (1)

[0072] in, Indicates Layers, Indicates input to Extract features from template images of layers. Indicates input to The first messenger extraction feature of the layer, Indicates The template image output by each layer extracts features. Indicates The first messenger output of each layer extracts features. Indicates The first output feature of the layer.

[0073] As an example, for each layer of the feature extraction module, the second output feature can be obtained in the following manner: the information integration module can be used to obtain a new messenger extraction feature based on the first messenger extraction feature output by the previous layer and the second messenger extraction feature output by the previous layer; the new messenger extraction feature can be used to replace the second messenger extraction feature in the second output feature output by the previous layer to obtain a new second output feature; the new second output feature can be used as the second input feature of the current layer to obtain the second output feature of the current layer.

[0074] Specifically, in an embodiment of the present disclosure, the information integration module can be used to receive messenger features from a template branch and messenger features from a search branch, generate new messenger extraction features, and thus obtain target-oriented information.

[0075] In this step, the information integration module can be used to perform an information transfer operation on the first messenger extraction feature from the template image and the second messenger extraction feature from the search image to obtain a new messenger extraction feature. The information transfer operation here may include adding the two messenger extraction features to obtain a new messenger extraction feature or selecting one of them as the new messenger extraction feature, which will be described in detail below.

[0076] like Figure 3 As shown, the information integration module can receive the first messenger extraction feature and the second messenger extraction feature, and output a new messenger extraction feature, the form of which can be the same as the first messenger extraction feature and the second messenger extraction feature, for example, the vector form of the three is the same.

[0077] like Figure 3 As shown, the second messenger extraction feature in the original second output feature can be replaced by the new messenger extraction feature to obtain a new second messenger extraction feature, and the second messenger extraction feature is input into the next layer of sub-module to continue extracting features.

[0078] In the above manner, the information of the template branch can be passed to the search branch through the information passing operation of the messenger feature, establishing a connection between the template image feature extraction and the search image feature extraction, so that the model can learn more template image information and improve the accuracy of the model.

[0079] like Figure 3 As shown in FIG. 1 , in the first-layer feature extraction module, the messenger features of the template branch and the search branch are shared, that is, the first messenger feature and the second messenger feature are both learnable initial messenger features; In the layer feature extraction module, the first messenger extraction feature of the template branch and the second messenger extraction feature of the search branch are subjected to information transmission operation through the information integration module to obtain a new messenger extraction feature, which can be input to the first layer feature extraction module together with the search image extraction feature of the current layer (or the current feature extraction module) of the search branch. Layer feature extraction module, get the The output of the layer feature extraction module.

[0080] Taking the second output feature including the search image extraction feature, the second messenger extraction feature, the target token and the distillation token as an example, in the search branch, the new messenger extraction feature can be input to the next layer together with the search image extraction feature, the target token and the distillation token of the current layer of the search branch to obtain the second output feature of the next layer. For example, the feature extraction module can be represented by the following equations (2) to (4):

[0081] (2)

[0082] (3)

[0083] (4)

[0084] in, Indicates input to Extract features from template images of layers; Indicates input to The second messenger extraction feature of the layer, that is, The second messenger extraction feature in the second output feature of the layer; Indicates that based on The new messenger extraction feature is obtained by combining the first messenger extraction feature in the first output feature of the layer and the second messenger extraction feature in the second output feature, that is, and Separate The messenger extraction features in each layer before and after information integration; and Respectively represent and Extract features from the search image output by each layer; and Respectively represent and The target token output by each layer; and Respectively represent and The distilled tokens output by each layer; Indicates The second messenger extraction feature in the second output feature of the feature extraction module is Indicates that the The second messenger extracted from the second output feature of the layer is used as input to the The second messenger of each layer extracts features, and integration represents the operation of the information integration module.

[0085] In addition, in the above step S410, it is described that the information transmission operation may include adding two messenger extraction features to obtain a new messenger extraction feature or selecting one of them as a new messenger extraction feature, which is described in detail here in combination with formula (2).

[0086] Specifically, the information integration module can be designed into different modes by selecting different integration operations in the above formula (2). Different modes can achieve the effect of improving tracking accuracy and can be selected according to needs.

[0087] As an example, the information integration module obtains new messenger extraction features by one of the following methods:

[0088] 1. Directly use the second messenger extraction features as new messenger extraction features.

[0089] This information transfer mode can be called a data-independent mode. Specifically, the messenger extraction features of the search branch can be independent of the template image features. For example, the above formula (2) can be written as .

[0090] In this mode, the model can directly learn the information that the template branch needs to pass and store it in the messenger features during the training phase, and can directly input the trained messenger features into the search branch during the inference phase.

[0091] 2. Add the first messenger extraction feature to the second messenger extraction feature to obtain a new messenger extraction feature.

[0092] This information transfer mode can be called an additive mode. Specifically, the new messenger extraction feature can be obtained by adding the first messenger extraction feature of the template branch and the second messenger extraction feature of the search branch. For example, the above formula (2) can be written as .

[0093] 3. Directly use the first messenger extraction features as new messenger extraction features.

[0094] This information transfer mode can be called a permutation mode. Specifically, the first messenger extraction feature of the template branch in the same layer can be directly used as the second messenger extraction feature of the search branch of the layer, that is, It all depends on , for example, the above formula (2) can be written as .

[0095] Here, the information integration module may be fixed, that is, it may not have learnable parameters, and its function is only to extract features from two messengers for information transmission (for example, adding the two or selecting one of them as the information transmission result).

[0096] In addition, it should be noted that although step S410 is described here first to determine the first output feature and steps S420 to S440 are described here to determine the second output feature, the order of determining the first output feature and the second output feature of the current layer is not limited to this. The second output feature may be determined first and then the first output feature, or the process of determining the second output feature and the process of determining the first output feature may be performed in parallel.

[0097] Return to reference Figure 1 In step S140, a feature fusion module may be used to fuse the first extraction result and the second extraction result to obtain a fusion result.

[0098] In step S150, a prediction head may be used to perform prediction based on the fusion result to obtain a predicted tracking result.

[0099] Here, the predicted tracking result may represent a bounding box of a visual target in the template sample image in the search sample image. As an example, the predicted tracking result may also include a confidence level.

[0100] As an example, the feature fusion module may be an encoder. Figure 3 As shown, the first extraction result may include the template image extraction feature in the first output feature of the last layer of the feature extraction module, and the second extraction result may include the search image extraction feature, the target token, and the distillation token in the second output feature of the last layer of the feature extraction module.

[0101] Specifically, the encoder is used to fuse the information of the two branches to obtain a fusion result. The fusion result may include features corresponding to the template image features, features corresponding to the search image features, features corresponding to the target tokens, and features corresponding to the distillation tokens, wherein the features corresponding to the target tokens may be obtained by interacting with the image features during the training process based on the initial target tokens, which may be used to obtain the target prediction results; the features corresponding to the distillation tokens may be obtained by interacting with the image features during the training process based on the initial distillation tokens, which may be used to obtain the distillation prediction results. The encoder may perform information fusion based on the final first extraction result and the second extraction result output by the feature extraction module (e.g., the part of the output of the last layer other than the messenger feature) to obtain fused template image features and search image features. The structure of the encoder may be composed of multiple layers of ViT blocks.

[0102] Here, the input of the encoder includes the first extraction result of the template branch and the second extraction result of the search branch, and may also include the target token and distillation token output by the search branch, but the input of the encoder does not include the messenger token. In addition, in the encoder, the target token and / or distillation token can interact with the template image features and search image features from the two branches to obtain a compact representation of the target position information.

[0103] Here, the target token is similar to the classification token in the ViT model, but it can directly predict the target state. The target token can be composed of two tokens, namely the bounding box token (bbox token) and the confidence token, which are used to predict the target bounding box and the confidence of the bounding box respectively.

[0104] Distilled tokens can also include two tokens, similar to the DeiT model, which can interact with other tokens through an attention mechanism, and their outputs are passed through the prediction head to predict the bounding box and confidence respectively.

[0105] The prediction head can receive the features corresponding to the distillation token and the features corresponding to the target token output from the encoder, and output the predicted target bounding box and confidence. Specifically, the output prediction tracking result includes the target prediction result corresponding to the target token and the distillation prediction result corresponding to the distillation token. The target prediction result and the distillation prediction result can include a bounding box and a confidence, respectively. The prediction head can be a lightweight structure, for example, it can be composed of 3 layers of MLPs. The prediction head can directly predict the bounding box and confidence from the target token or distillation token output by the feature fusion module.

[0106] In step S160, a training loss may be obtained based on the predicted tracking result, the real annotation information, and the distilled annotation information, and the visual target tracking model may be trained to obtain a trained visual target tracking model.

[0107] In this step, the training loss can be obtained by comparing the predicted tracking result and the sample annotation information. Based on the training loss, the parameters of the visual target tracking model can be adjusted to train the visual target tracking model.

[0108] As an example, in an embodiment of the present disclosure, a knowledge distillation method may be introduced. Specifically, a trained visual target tracking model may be used as a student model, and SwinTrack may be used as a teacher model. The teacher model may provide a supervision signal to the student model, thereby improving the prediction accuracy of the visual target tracking model.

[0109] Specifically, the sample annotation information may include real annotation information and distilled annotation information, the real annotation information may be obtained by manual annotation, and the distilled annotation information may be determined based on the teacher prediction result obtained by the pre-trained teacher model for the training sample. The teacher prediction result may include a teacher bounding box.

[0110] For example, the visual target tracking model can be trained in the following manner: obtain a first loss (or ordinary loss) by comparing the target prediction result and the true annotation information; obtain a second loss (or distillation loss) by comparing the distillation prediction result and the distillation annotation information; determine the training loss based on the first loss and the second loss; use the training loss to adjust the parameters of the visual target tracking model to train the visual target tracking model.

[0111] As an example, the real annotation information may include a real bounding box and a real confidence, the distilled annotation information may include a distilled bounding box and a distilled confidence, the target prediction result may include a target prediction bounding box and a target prediction confidence, and the distilled prediction result may include a distilled prediction bounding box and a distilled prediction confidence.

[0112] like Figure 3 As shown, the first loss may be a common loss, which may include, for example, a first bounding box regression loss and a first confidence loss. The second loss may be a knowledge distillation loss, which may include, for example, a second bounding box regression loss and a second confidence loss.

[0113] In this example, the prediction loss It can be expressed by the following formula (5):

[0114] (5)

[0115] in, represents the first bounding box regression loss, represents the first confidence loss, represents the second bounding box regression loss, represents the second confidence loss, , , and Here, confidence loss can also be called classification loss.

[0116] The first bounding box regression loss can be represented by the generalized intersection over union (GIoU) ​​loss between the bounding box predicted by the student model (i.e., the target prediction bounding box mentioned above) and the real bounding box in the real annotation information. The first confidence loss can also be called the first classification loss, which can be represented by the varifocal loss (VFL) between the confidence predicted by the student model (i.e., the target prediction confidence mentioned above) and the real confidence. The real confidence can be the intersection over union (IoU) between the predicted bounding box (i.e., the target prediction bounding box mentioned above) and the real bounding box.

[0117] As an example, the first bounding box regression loss and the first confidence loss It can be expressed by the following equations (6) and (7):

[0118] (6)

[0119] (7)

[0120] in, represents the generalized intersection and union ratio, represents the zoom loss, represents the intersection-and-union ratio, represents the target prediction bounding box, represents the true bounding box, Represents the target prediction confidence.

[0121] The second loss may be a knowledge distillation loss, which may include, for example, a second bounding box regression loss and a second confidence loss.

[0122] Similarly, the second bounding box regression loss can be represented by the generalized intersection-over-union loss between the distilled prediction bounding box and the distilled bounding box in the distilled annotation information. The second confidence loss can also be called the second classification loss, which can be represented by the zoom loss between the distilled prediction confidence and the distilled confidence. The distilled confidence can be the intersection-over-union ratio between the distilled prediction bounding box and the distilled bounding box.

[0123] As an example, the second bounding box regression loss and the second confidence loss It can be expressed by the following equations (8) and (9):

[0124] (8)

[0125] (9)

[0126] in, represents the distilled prediction bounding box, represents the distillation bounding box, Represents the distillation prediction confidence.

[0127] In the embodiments of the present disclosure, a label smoothing knowledge distillation method is proposed, which is different from traditional knowledge distillation. In traditional knowledge distillation, the output of the teacher model replaces the real label. In the embodiments of the present disclosure, by introducing the real bounding box into the teacher supervision box, on the one hand, the useful information introduced by the teacher model is retained, and on the other hand, the erroneous information of the teacher supervision box is corrected.

[0128] As an example, the distilled bounding box can be obtained in the following way: based on the smoothing coefficient, the real bounding box and the teacher bounding box are weighted and summed to obtain the distilled bounding box.

[0129] As mentioned above, the teacher bounding box can be predicted by a pre-trained teacher model for training samples, and the smoothing coefficient can represent the proportion of the real bounding box in the distilled bounding box.

[0130] For example, it can be expressed by the following formula (10):

[0131] (10)

[0132] in, represents the smoothing coefficient, represents the distillation bounding box, represents the true bounding box, In this example, in the above equations (8) and (9), the superscript “ " indicates label smoothing.

[0133] In the above example, the smoothing coefficient When different values ​​are taken, different label smoothing forms can be obtained.

[0134] In one example, the smoothing coefficient It can be a preset fixed value set manually, for example, which can be called a fixed smoothing coefficient. In this example, the distillation method can be considered as fixed labels smoothing knowledge distillation, which can be represented by the symbol represents a fixed smooth bounding box, using fixed bounding box regression loss and a fixed confidence loss They represent the corresponding regression loss and classification loss respectively.

[0135] In another example, the smoothing coefficient It can be determined based on the predicted tracking result of the visual target tracking model. In this example, the visual target tracking model may also include a fully connected layer and a sigmoid layer, and the smoothing coefficient may be determined in the following manner: inputting the feature corresponding to the target token in the fusion result into the fully connected layer to obtain the fully connected feature; inputting the fully connected feature into the sigmoid layer to obtain the smoothing coefficient, wherein the smoothing coefficient is a scalar in the range of [0,1].

[0136] Specifically, the smoothing coefficient It can be a value output by the model that varies with the input sample, which can be called an adaptive smoothing coefficient. In this example, the distillation method can be considered as adaptive label smoothing knowledge distillation. In this example, it can be represented by the symbol Represents an adaptive smooth bounding box, using adaptive bounding box regression loss and adaptive confidence loss Represent the corresponding regression loss and confidence loss respectively. Here, the superscript " ” stands for adaptive labelsmoothing.

[0137] In this example, the feature corresponding to the target token in the fusion result (such as the target token output by the encoder) can be followed by two fully connected layers and finally passed through a sigmoid layer to obtain a scalar, which can be used as an adaptive smoothing coefficient.

[0138] Although it is described above that the first loss is obtained by comparing the target prediction result and the true annotation information, and the second loss is obtained by comparing the distillation prediction result and the distillation annotation information to obtain the final training loss, the embodiments of the present disclosure are not limited to this, and the second loss can also be obtained by comparing the target prediction result and the distillation annotation information.

[0139] In the embodiments of the present disclosure, distillation tokens and distillation prediction results may not be used. Specifically, in the above process, the second feature may not include the distillation token, and the prediction tracking result may not include the distillation prediction result. The processing process may be similar to that described above.

[0140] In this case, instead of equations (8) and (9), the second bounding box regression loss and the second confidence loss It can be expressed by the following equations (11) and (12):

[0141] (11)

[0142] (12)

[0143] in, represents the distilled prediction bounding box, represents the distillation bounding box, Represents the distillation prediction confidence.

[0144] Through the above method, an adaptive label smoothing distillation training scheme can be provided, and based on the adaptive smoothing coefficient, the visual target tracking model can be efficiently trained.

[0145] The visual target tracking model proposed in the training method of the visual target tracking model according to the embodiment of the present disclosure can be called a MesTrack tracker, which has a dual-stream structure and improves the model accuracy by setting an information integration module, messenger features and target tokens, thereby combining the high precision of the single-stream architecture tracker and the fast reasoning performance of the dual-stream architecture tracker, thereby having both the fast reasoning capability of the dual-stream architecture and the high precision of the single-stream architecture.

[0146] In addition, in the training method according to the embodiment of the present disclosure, a messenger feature and an information integration module are proposed, which can efficiently transfer information from the template branch to the search branch. In addition, the integrity of the Transformer architecture can be maintained by setting the target token.

[0147] According to the embodiments of the present disclosure, the effectiveness of the messenger token and the information integration module in the above-mentioned training method, the effectiveness of the adaptive label smoothing knowledge distillation scheme, and the improvement of the model performance are verified, which will be described below respectively.

[0148] On the one hand, in order to verify the effectiveness of the messenger token and the information integration module, the MesTrack model with the messenger features and the information integration module removed, as well as three models with messenger features and the information integration module were trained on the two datasets LaSOT and LaSOT_ext. Among them, the information integration module used the data-independent integration mode (B1), the additive integration mode (B2) and the permutation integration mode (B3). The performance verification results of these four models are shown in Table 1 below:

[0149] Table 1

[0150]

[0151] In Table 1, SUC represents the success rate, PRE represents the precision, NPRE represents the normalized precision, unshared represents unshared, no represents no messenger feature, independent represents independent mode, additive represents additive mode, replacement represents replacement mode, and usual represents ordinary knowledge distillation. As shown in Table 1 above, the experimental results show that messenger tokens and information integration modules help improve tracking accuracy, among which the information integration module of the replacement integration mode has the most obvious improvement.

[0152] On the other hand, in order to verify the effectiveness of the adaptive label smoothing knowledge distillation scheme used in the present invention, a MesTrack model (A3) with messenger tokens and information integration modules removed was trained on the above dataset using the traditional distillation mode as a baseline, and its performance was compared with that of two MesTrack models (D2 and D3) with the same architecture trained using the label smoothing knowledge distillation strategy. Among them, D2 uses fixed label smoothing knowledge distillation and sets the fixed smoothing coefficient to 0.5, and D3 uses adaptive label smoothing knowledge distillation. The performance verification results of the adaptive label smoothing knowledge distillation scheme are shown in Table 2 above:

[0153] Table 2

[0154]

[0155] As shown in Table 2 above, the experimental results show that using adaptive label smoothing knowledge distillation loss to train the tracker architecture can improve the performance of the tracker.

[0156] On the other hand, the embodiments of the present disclosure have significantly improved performance compared to other trackers of the same period. Four MesTrack architecture trackers of different sizes and input resolutions can be compared with existing state-of-the-art trackers on four datasets: LaSOT, LaSOT_ext, TrackingNet, and GOT-10k.

[0157] Specifically, the embodiments of the present disclosure design four model variants of the MesTrack architecture, namely MesTrack-T, MesTrack-T-384, MesTrack-B and MesTrack-B-384, as shown in Table 3. These model variants represent the Tiny or Base scale of the MesTrack architecture and their corresponding 224×224 or 384×384 input resolution versions. In various variants of the MesTrack architecture, the number of encoder layers is equal to the sum of the number of encoder layers and the number of decoder layers in the SwinTrack of similar scale, and the number of encoder channels is consistent with the backbone network. The 384×384 resolution variant and the corresponding 224×224 variant are the same except for the input picture resolution and the teacher model. The various parameter settings, parameter amounts, floating-point operations and frame rates of the model are shown in Table 3 below as model variants, where FPS is measured on a single NVIDIAGeForce RTX3090 graphics card.

[0158] Table 3

[0159]

[0160] The comparison of these four model variants with the SOTA tracker under the corresponding storage or resolution conditions is shown in Table 4:

[0161] Table 4

[0162]

[0163] As shown in Table 4 above, the innovative tracker structure and distillation training scheme proposed in this disclosure can bring significant performance improvements compared to the best trackers of the same period.

[0164] In a second aspect of an exemplary embodiment of the present disclosure, a visual target tracking method is provided. The visual target tracking method can be applied to a scenario where a user interacts with software, for example, the software can be loaded on a user terminal, the user can input a visual target tracking instruction on the user terminal, and the user terminal can track a visual target by executing the visual target tracking method according to an exemplary embodiment of the present disclosure.

[0165] like Figure 5 As shown, the visual target tracking method may include: step S510, obtaining a target template image and a target search image, wherein the target template image contains a visual target to be tracked; step S520, inputting the target template image and the target search image into a visual target tracking model, and using the visual target tracking model to predict the bounding box of the visual target in the search sample image, wherein the visual target tracking model is trained according to the training method of the visual target tracking model described in the embodiments of the present disclosure.

[0166] The specific steps and details of the visual target tracking method according to the exemplary embodiment of the present disclosure can be implemented and understood similarly to the training method of the visual target tracking model described in the first aspect above, and accordingly, the visual target tracking method has the same or similar beneficial effects as the training method of the visual target tracking model described above, which will not be repeated here.

[0167] In a third aspect of the exemplary embodiments of the present disclosure, a training system for a visual target tracking model is provided, such as Figure 6 As shown, the visual target tracking model includes a feature embedding module, an information integration module, a feature extraction module, a feature fusion module and a prediction head, the feature extraction module includes multiple layers, and the training system includes a sample acquisition unit 610, a first determination unit 620, a second determination unit 630, a sample fusion unit 640, a sample prediction unit 650 and a training unit 660.

[0168] The sample acquisition unit 610 is configured to acquire multiple pairs of training samples, wherein each pair of training samples includes a template sample image, a search sample image, and sample annotation information, the sample annotation information represents the bounding box of the visual target to be tracked in the template sample image in the search sample image, wherein the sample annotation information includes real annotation information and distilled annotation information.

[0169] The first determining unit 620 is configured to input the training sample into the feature embedding module to obtain the template image features and the search image features.

[0170] The second determination unit 630 is configured to use each layer of the feature extraction module to perform a feature extraction operation on the first feature and the second feature to obtain a first extraction result and a second extraction result, wherein the first feature includes a template image feature and a first messenger feature, the second feature includes a search image feature, a second messenger feature, a target token and a distillation token, the first messenger feature and the second messenger feature both include learnable parameters, the first messenger feature and the second messenger feature are the same, and the target token and the distillation token both include learnable parameters, wherein for two adjacent layers of the feature extraction module, the information integration module is used to determine the input features of the latter layer corresponding to the second messenger feature based on the output features of the previous layer corresponding to the first messenger feature and the output features corresponding to the second messenger feature.

[0171] The sample fusion unit 640 is configured to fuse the first extraction result and the second extraction result using a feature fusion module to obtain a fusion result.

[0172] The sample prediction unit 650 is configured to use the prediction head to perform prediction based on the fusion result to obtain a predicted tracking result, wherein the predicted tracking result represents a bounding box of the visual target in the template sample image in the search sample image.

[0173] The training unit 660 is configured to obtain a training loss based on the predicted tracking result, the real annotation information and the distilled annotation information to obtain a trained visual target tracking model.

[0174] As an example, the feature extraction module includes multiple layers, the input of each layer of the feature extraction module includes a first input feature and a second input feature, the output of each layer of the feature extraction module includes a first output feature and a second output feature, the first output feature includes a template image extraction feature and a first messenger extraction feature, and the second output feature includes a search image extraction feature and a second messenger extraction feature. The second determination unit 630 is configured to use each layer of the feature extraction module to perform a feature extraction operation in the following manner: using the first output feature output by the previous layer as the first input feature of the current layer, inputting it to the current layer, and obtaining the first output feature of the current layer; using the information integration module, integrating the first messenger extraction feature output by the previous layer with the second messenger extraction feature output by the previous layer to obtain a new messenger extraction feature; using the new messenger extraction feature to replace the second messenger extraction feature in the second output feature output by the previous layer to obtain a new second output feature; using the new second output feature as the second input feature of the current layer, inputting it to the current layer, and obtaining the second output feature of the current layer, wherein the first input feature and the second input feature of the first layer of the feature extraction module are the first feature and the second feature, respectively, and the first output feature and the second output feature of the last layer of the feature extraction module include the first extraction result and the second extraction result, respectively.

[0175] As an example, the information integration module obtains a new messenger extraction feature by one of the following methods: directly using the second messenger extraction feature as a new messenger extraction feature; fusing the first messenger extraction feature with the second messenger extraction feature to obtain a new messenger extraction feature; directly using the first messenger extraction feature as a new messenger extraction feature.

[0176] As an example, the distilled annotation information is determined based on the teacher prediction results obtained by a pre-trained teacher model for the training samples, and the predicted tracking results include the target prediction results corresponding to the target token and the distilled prediction results corresponding to the distilled token. The training unit 660 is configured to train the visual target tracking model in the following manner: obtain a first loss by comparing the target prediction results and the true annotation information; obtain a second loss by comparing the distilled prediction results and the distilled annotation information; determine the training loss based on the first loss and the second loss; and use the training loss to adjust the parameters of the visual target tracking model to train the visual target tracking model.

[0177] As an example, the real annotation information includes the real bounding box, the distilled annotation information includes the distilled bounding box, and the teacher prediction result includes the teacher bounding box, wherein the distilled bounding box is obtained in the following manner: based on a smoothing coefficient, the real bounding box and the teacher bounding box are weighted and summed to obtain the distilled bounding box, wherein the smoothing coefficient represents the proportion of the real bounding box in the distilled bounding box.

[0178] As an example, a smoothing coefficient is determined based on a predicted tracking result of a visual target tracking model, wherein the visual target tracking model also includes a fully connected layer and a sigmoid layer, wherein the smoothing coefficient is determined in the following manner: a feature corresponding to a target token in a fusion result is input into the fully connected layer to obtain a fully connected feature; and the fully connected feature is input into the sigmoid layer to obtain a smoothing coefficient, wherein the smoothing coefficient is a scalar in the range of [0,1].

[0179] Regarding the device in the above-mentioned embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method. Each unit in the training device of the visual target tracking model can execute the corresponding steps in the method according to the training method of the visual target tracking model in the method embodiment of the first aspect above, which will not be elaborated here.

[0180] In a fourth aspect of the exemplary embodiments of the present disclosure, a visual target tracking system is provided, such as Figure 7 As shown, the visual target tracking system includes an image acquisition unit 710 and a visual prediction unit 720.

[0181] The image acquisition unit 710 is configured to acquire a target template image and a target search image, wherein the target template image contains a visual target to be tracked.

[0182] The visual prediction unit 720 is configured to input the target template image and the target search image into a visual target tracking model, and use the visual target tracking model to predict the bounding box of the visual target in the search sample image, wherein the visual target tracking model is trained according to the training method of the visual target tracking model of an embodiment of the present disclosure.

[0183] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0184] Figure 8 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Figure 8 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the training method of the visual target tracking model or the visual target tracking method as described in the above exemplary embodiments.

[0185] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0186] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0187] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0188] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0189] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0190] In an exemplary embodiment, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the visual target tracking model or the visual target tracking method as described in the above exemplary embodiments. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a B D-R, BD-RLTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0191] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes computer executable instructions, and when the computer executable instructions are executed by at least one processor, a training method for a visual target tracking model or a visual target tracking method according to an exemplary embodiment of the present disclosure is implemented.

[0192] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0193] In addition, it should be noted that although several examples of each step are described above with reference to specific drawings, it should be understood that the embodiments of the present disclosure are not limited to the combinations given in the examples, and the steps appearing in different drawings may be combined, and the order of execution of the steps may be changed, which is not exhaustive here.

[0194] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a visual target tracking model, characterized in that: The visual target tracking model includes a feature embedding module, an information integration module, a feature extraction module, a feature fusion module and a prediction head, the feature extraction module includes multiple layers, wherein the training method includes: Acquire multiple pairs of training samples, wherein each pair of training samples includes a template sample image, a search sample image, and sample annotation information, wherein the template sample image includes a visual target to be tracked, and the sample annotation information represents a bounding box of the visual target in the search sample image, wherein the sample annotation information includes real annotation information and distilled annotation information; Inputting the training sample into the feature embedding module to obtain template image features and search image features; Using each layer of the feature extraction module, perform a feature extraction operation on the first feature and the second feature to obtain a first extraction result and a second extraction result, wherein the first feature includes the template image feature and the first messenger feature, the second feature includes the search image feature, the second messenger feature, the target token and the distillation token, the first messenger feature and the second messenger feature are both learnable parameters, the first messenger feature and the second messenger feature are the same, and the target token and the distillation token both include learnable parameters, wherein for two adjacent layers of the feature extraction module, using the information integration module, based on the output feature of the previous layer corresponding to the first messenger feature and the output feature corresponding to the second messenger feature, determine the input feature of the next layer corresponding to the second messenger feature; Using the feature fusion module, the first extraction result and the second extraction result are fused to obtain a fusion result; Using the prediction head, prediction is performed based on the fusion result to obtain a predicted tracking result, wherein the predicted tracking result represents a bounding box of the visual target in the template sample image in the search sample image; A training loss is obtained based on the predicted tracking result, the real annotation information and the distilled annotation information, and the visual target tracking model is trained to obtain a trained visual target tracking model.

2. The training method according to claim 1, characterized in that: The input of each layer of the feature extraction module includes a first input feature and a second input feature, the output of each layer of the feature extraction module includes a first output feature and a second output feature, the first output feature includes a template image extraction feature and a first messenger extraction feature, the second output feature includes a search image extraction feature and a second messenger extraction feature, Wherein, for each layer of the feature extraction module, the feature extraction operation includes: The first output feature of the previous layer is used as the first input feature of the current layer, and is input into the current layer to obtain the first output feature of the current layer; Using the information integration module, a new messenger extraction feature is obtained based on the first messenger extraction feature output by the previous layer and the second messenger extraction feature output by the previous layer; Using the new messenger extraction feature to replace the second messenger extraction feature in the second output feature outputted by the previous layer, to obtain a new second output feature; The new second output feature is used as the second input feature of the current layer, and is input into the current layer to obtain the second output feature of the current layer. Among them, the first input feature and the second input feature of the first layer of the feature extraction module are the first feature and the second feature respectively, and the first output feature and the second output feature of the last layer of the feature extraction module include the first extraction result and the second extraction result respectively.

3. The training method according to claim 2, characterized in that: The information integration module obtains the new messenger extraction feature by one of the following methods: directly using the second messenger extraction feature as the new messenger extraction feature; Adding the first messenger extraction feature and the second messenger extraction feature to obtain the new messenger extraction feature; The first messenger extraction feature is directly used as the new messenger extraction feature.

4. The training method according to claim 1, characterized in that: The distillation annotation information is determined based on a teacher prediction result obtained by a pre-trained teacher model for predicting the training sample, and the prediction tracking result includes a target prediction result corresponding to the target token and a distillation prediction result corresponding to the distillation token. The visual target tracking model is trained in the following manner: Obtaining a first loss by comparing the target prediction result and the true annotation information; Obtaining a second loss by comparing the distillation prediction result and the distillation annotation information; Determine the training loss based on the first loss and the second loss; The training loss is used to adjust parameters of the visual object tracking model to train the visual object tracking model.

5. The training method according to claim 4, characterized in that: The real annotation information includes a real bounding box, the distilled annotation information includes a distilled bounding box, and the teacher prediction result includes a teacher bounding box, wherein the distilled bounding box is obtained by: Based on the smoothing coefficient, the real bounding box and the teacher bounding box are weighted and summed to obtain the distilled bounding box. The smoothing coefficient represents the proportion of the real bounding box in the distilled bounding box.

6. The training method according to claim 5, characterized in that: The visual target tracking model further includes a fully connected layer and a sigmoid layer, wherein the smoothing coefficient is determined by: Inputting the feature corresponding to the target token in the fusion result into the fully connected layer to obtain a fully connected feature; Input the fully connected features into the sigmoid layer to obtain the smoothing coefficient, The smoothing coefficient is a scalar in the range of [0, 1].

7. A visual target tracking method, characterized in that: The visual target tracking method comprises: Acquire a target template image and a target search image, wherein the target template image contains a visual target to be tracked; The target template image and the target search image are input into a visual target tracking model, and the visual target tracking model is used to predict the bounding box of the visual target in the target search image, wherein the visual target tracking model is trained according to the training method of the visual target tracking model according to any one of claims 1 to 6.

8. A training system for a visual target tracking model, characterized in that: The visual target tracking model includes a feature embedding module, an information integration module, a feature extraction module, a feature fusion module and a prediction head, wherein the feature extraction module includes multiple layers, wherein the training system includes: A sample acquisition unit is configured to acquire multiple pairs of training samples, wherein each pair of training samples includes a template sample image, a search sample image, and sample annotation information, wherein the template sample image includes a visual target to be tracked, and the sample annotation information represents a bounding box of the visual target in the search sample image, wherein the sample annotation information includes real annotation information and distilled annotation information; A first determining unit is configured to input the training sample into the feature embedding module to obtain a template image feature and a search image feature; A second determination unit is configured to use each layer of the feature extraction module to perform a feature extraction operation on the first feature and the second feature to obtain a first extraction result and a second extraction result, wherein the first feature includes the template image feature and the first messenger feature, the second feature includes the search image feature, the second messenger feature, the target token and the distillation token, the first messenger feature and the second messenger feature are both learnable parameters, the first messenger feature and the second messenger feature are the same, and the target token and the distillation token both include learnable parameters, wherein for two adjacent layers of the feature extraction module, the information integration module is used to determine the input feature of the next layer corresponding to the second messenger feature based on the output feature of the previous layer corresponding to the first messenger feature and the output feature corresponding to the second messenger feature; a sample fusion unit, configured to fuse the first extraction result and the second extraction result using the feature fusion module to obtain a fusion result; A sample prediction unit is configured to use the prediction head to perform prediction based on the fusion result to obtain a predicted tracking result, wherein the predicted tracking result represents a bounding box of a visual target in the template sample image in the search sample image; The training unit is configured to obtain a training loss based on the predicted tracking result, the real annotation information and the distilled annotation information, and train the visual target tracking model to obtain a trained visual target tracking model.

9. A visual target tracking system, characterized in that: The visual target tracking system comprises: An image acquisition unit is configured to acquire a target template image and a target search image, wherein the target template image contains a visual target to be tracked; A visual prediction unit is configured to input the target template image and the target search image into a visual target tracking model, and use the visual target tracking model to predict the bounding box of the visual target in the target search image, wherein the visual target tracking model is trained according to the training method of the visual target tracking model according to any one of claims 1 to 6.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, Wherein, when the processor executable instructions are executed by the processor, the processor is prompted to execute the training method of the visual target tracking model according to any one of claims 1 to 6 or the visual target tracking method according to claim 7.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the visual target tracking model according to any one of claims 1 to 6 or the visual target tracking method according to claim 7.

12. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the training method of the visual target tracking model according to any one of claims 1 to 6 or the visual target tracking method according to claim 7 is implemented.

Images