Target tracker training methods and target tracking methods

Abstract

This invention provides a target tracker training method and a target tracking method. The training method includes: training a neural network model based on multiple template-search image pairs, each with its own labels. Each template-search image pair includes a target template frame and a search frame. The labels indicate whether the target template frame and the search frame match in each template-search image pair and the true target location. The neural network model includes an encoder, a decoder, and a prototype memory network. The trained neural network model is then used to determine the target tracker for outputting the target tracking result corresponding to the search frame in the template-search image pair. The target tracking result includes a confidence level representing the matching result and a target location prediction result in the case of a match. This invention can perform accurate target state reasoning by utilizing positioning uncertainty, ensuring reliable target appearance matching and target tracking results even when the target appearance changes drastically.

Description

Technical Field

[0001] This invention relates to the field of visual target tracking technology, and in particular to a target tracker training method and a target tracking method. Background Art

[0002] Visual object tracking (VOT) is an important task in computer vision with various practical applications, such as intelligent video surveillance, autonomous driving, and robotics. Given an initially labeled bounding box of an object in the first frame, the goal of visual object tracking is to predict the bounding box of that object throughout the entire video. The tracker should construct a discriminative object appearance to distinguish the object from background clutter in real time and be able to flexibly address the challenge of tracking failures caused by changes in the object's appearance.

[0003] For decades, matching-based methods have been the mainstream solution due to their good balance between accuracy and efficiency. These methods treat visual object tracking from the perspective of similarity comparison, estimating the target state by learning a two-stream network to compute the similarity between a template and a search image.

[0004] Transformer-based target trackers have become the mainstream tracking paradigm due to their accuracy and efficiency. Despite substantial progress, two major drawbacks remain: First, during target motion, the appearance of the target can change drastically over time. For example, the target may become blurred, occluded, or its angle may change due to motion (deformation). Unreliable target localization and tracking predictions can accumulate throughout the video. Some existing methods, such as LTMU, Siam R-CNN, and KeepTrack, mitigate this problem by introducing online meta-updates, object re-detection, or online appearance association mechanisms. However, these methods rely on the assumption of local smoothing to maintain temporal consistency of the target (finding the target position in the next frame within a local range of the target's position in the previous frame). For fast-moving targets, this can easily lead to suboptimal solutions, misleading target appearance matching and ultimately causing tracking failure. Second, the target may become blurred, occluded, or deformed during motion, increasing the uncertainty in the target bounding box localization. Most methods typically treat target tracking as a deterministic coordinate regression problem, severely neglecting the uncertainty of target localization. This uncertainty, in challenging scenarios, hinders trackers from maintaining reliable target state predictions. Other methods, such as UAST and UAF, employ distributed regression representations and Monte Carlo dropout techniques to model localization uncertainty; however, how to effectively utilize the uncertainty output to obtain reliable target estimates remains unclear, and their tracking performance is poor for blurred, occluded, and deformed targets. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a target tracker training method and a target tracking method to eliminate or improve one or more defects existing in the prior art.

[0006] A first aspect of the present invention provides a method for training a target tracker, the method comprising the following steps:

[0007] The neural network model is trained using multiple templates with their own first and second labels and search image pairs.

[0008] Each template-search image pair includes a target template frame and a search frame. The first label and the second label are used to indicate whether the target template frame and the search frame in each template-search image pair match, and the actual target position in the search frame, respectively. The neural network model includes an encoder, a decoder, and a prototype memory network. The encoder encodes the template-search image pair and outputs the encoded target template features and the encoded search image features. The decoder decodes the encoded search image features and outputs the target localization uncertainty and the target position prediction result. The prototype memory network constructs a target prototype memory library based on the encoded target template features, the encoded search image features corresponding to the encoded target template features, the target localization uncertainty, and the target position prediction result. Based on the target prototype memory library, a confidence level is generated to represent the matching result. The target prototype memory library is constructed when the confidence level indicates that the target template frame and the search frame in the corresponding template-search image pair match and are consistent with the first label of the template-search image pair.

[0009] The trained neural network model is determined as a target tracker for outputting the target tracking results corresponding to the search frames in the template and search image pair. The target tracking results include confidence scores representing the matching results and target position prediction results in the matching case.

[0010] In some embodiments of the present invention, the decoder includes a feature pyramid network, a target localization branch, and a target localization uncertainty prediction branch.

[0011] The feature pyramid network is used to upsample the features of the encoded search image;

[0012] The target localization branch is used to estimate the target position in the upsampled encoded search image features and output the target position prediction result accordingly.

[0013] The target localization uncertainty prediction branch is used to generate target localization uncertainty based on the upsampled encoded search image features.

[0014] In some embodiments of the present invention, the prototype memory network includes a global average pooling layer, a confidence inversion module, a splicing module, a convolutional layer, a masking module, a weighted operation module, a target prototype memory, a memory reader, a target prototype aggregation module, and a multilayer perceptron.

[0015] The global average pooling layer is used to compress the encoded target template features and output the target prototype accordingly.

[0016] The confidence inversion module is used to convert the target positioning uncertainty into a confidence feature;

[0017] The splicing module is used to splice the encoded search image features and the corresponding confidence features, and output the spliced ​​features accordingly.

[0018] The convolutional layer is used to encode the visual semantics and location confidence in the concatenated features, and outputs the encoded features accordingly.

[0019] The masking module is used to mask the target location prediction result and output the target location mask result accordingly.

[0020] The weighted operation module is used to perform weighted operations on the target prototype, the corresponding encoded features, and the target position mask result, and output the target weighted prototype accordingly.

[0021] The target prototype memory is constructed from multiple historical target weighted prototypes, which are a subset of target weighted prototypes selected from multiple output target weighted prototypes.

[0022] The memory reader is used to extract from the target prototype memory a number of historical target weighted prototypes that have a high similarity to the output target weighted prototype;

[0023] The target prototype aggregation module is used to concatenate multiple historical target weighted prototypes that have a high similarity to the output target weighted prototype into a target combination prototype, and aggregate the target combination prototype and the corresponding target weighted prototype based on the cross attention mechanism to output the target aggregation prototype.

[0024] The multilayer perceptron is used to generate a confidence score based on the target aggregate prototype. If the confidence score is greater than a preset threshold and the matching result and corresponding template are consistent with the first label of the search image pair, then the target weighted prototype corresponding to the confidence score can be used as a historical target weighted prototype.

[0025] In some embodiments of the present invention, the neural network model is iteratively trained by minimizing the target localization uncertainty loss function of the decoder and the cross-entropy loss function of the prototype memory network.

[0026] In some embodiments of the present invention, the expression for the target localization uncertainty loss function is:

[0027]

[0028] Among them, L uc σ represents the target location uncertainty loss function; μ represents the predicted target position coordinates; σ represents the standard deviation of the predicted target position coordinates, i.e., the target location uncertainty; μ gt The actual value representing the target's location coordinates, i.e., the second label;

[0029] The expression for the cross-entropy loss function is:

[0030]

[0031] Among them, L pro N represents the cross-entropy loss function; B This represents the number of template-search image pairs in a batch, where i represents the index of the template-search image pair in the batch, i = 1, 2, ..., N. B ;y i p represents the first label of the i-th template and the search image pair; i This represents the confidence level of the i-th template and the search image in the corresponding target aggregation prototype.

[0032] A second aspect of the present invention provides a target tracking method, the method comprising the following steps:

[0033] Obtain the video generated for the target motion;

[0034] The targets in the initial frames of the video are marked, and the marked initial frames are used as target template frames, while the unmarked frames in the video are used as search frames.

[0035] The target template frame and each search frame are respectively used as multiple templates and search image pairs and input into the target tracker, so that the target tracker outputs the target tracking result for each search frame. The target tracking result includes a confidence level for representing the matching result and a target position prediction result in the case of matching. When the confidence level indicates that the target template frame and the search frame in the corresponding template and search image pair match, the target prototype memory is updated. The target tracker is pre-trained based on the target tracker training method described in the first aspect above.

[0036] In some embodiments of the present invention, in the step of inputting the target template frame and each search frame as multiple templates and search image pairs into the target tracker so that the target tracker outputs the target tracking results of each search frame, if the output confidence level is greater than a preset threshold, the target prototype memory is updated based on the target weighted prototype corresponding to the confidence level.

[0037] A third aspect of the present invention provides an electronic device comprising: a computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, wherein when the computer instructions are executed by the processor, the device implements the steps of the target tracker training method described in the first aspect above, or implements the steps of the target tracking method described in the second aspect above.

[0038] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target tracker training method described in the first aspect above, or implements the steps of the target tracking method described in the second aspect above.

[0039] A fifth aspect of the present invention provides a computer program product comprising computer instructions which, when executed by a processor, implement the steps of the target tracker training method described in the first aspect, or the steps of the target tracking method described in the second aspect.

[0040] The target tracker training method and target tracking method of the present invention can obtain reliable target estimation and tracking results by using positioning uncertainty to perform accurate target state reasoning. Even when the target appearance changes drastically, reliable target appearance matching results and target tracking results can still be guaranteed.

[0041] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0042] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0043] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0044] Figure 1 This is a flowchart illustrating a target tracker training method in one embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the network structure of a target tracker in one embodiment of the present invention;

[0046] Figure 3 This is a flowchart illustrating a target tracking method in one embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram comparing the target tracking results of a target tracker in one embodiment of the present invention with those of several existing mainstream trackers. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0049] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0050] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0051] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0052] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0053] To address the challenge of target tracking failure caused by changes in the target's appearance during movement (such as becoming blurred, occluded, or deformed), and to effectively utilize the positioning uncertainty output to obtain reliable target estimation, target matching, and target tracking results, this invention provides a target tracker training method and a target tracking method based on uncertainty awareness. By utilizing positioning uncertainty information for accurate target state reasoning, reliable target appearance matching and target tracking results can still be guaranteed even when the target's appearance changes drastically.

[0054] Figure 1This is a schematic flowchart of a target tracker training method according to an embodiment of the present invention. Figure 1 As shown, the target tracker training method includes the following steps:

[0055] Step S110: Train a neural network model based on multiple templates and search image pairs, each with its own first and second labels.

[0056] Each template-search image pair includes a target template frame and a search frame. The first label and the second label are used to indicate whether the target template frame and the search frame in each template-search image pair match, and the actual target position in the search frame, respectively. The neural network model includes an encoder, a decoder, and a prototype memory network. The encoder encodes the template-search image pair and outputs the encoded target template features and the encoded search image features. The decoder decodes the encoded search image features and outputs the target localization uncertainty and the target position prediction result. The prototype memory network constructs a target prototype memory library based on the encoded target template features, the encoded search image features corresponding to the encoded target template features, the target localization uncertainty, and the target position prediction result. Based on the target prototype memory library, a confidence level is generated to represent the matching result. The target prototype memory library is constructed when the confidence level indicates that the target template frame and the search frame in the corresponding template-search image pair match and are consistent with the first label of the template-search image pair.

[0057] Specifically, the training dataset used to train the model consists of multiple pairs of templates and search images, each with its own first and second labels. In practice, this training dataset can be obtained as follows: First, acquire videos of the motion of multiple different objects. These videos can be captured simultaneously for the motion of multiple different objects, or they can be multiple videos captured separately for the motion of each of the multiple different objects. From these multiple different objects, determine one object as the target object to be tracked, and then treat the other objects as interference objects. Then, two frames are randomly selected multiple times from the acquired video. One frame contains the target object and is used as the target template frame. The other frame may or may not contain the target object and is used as the search frame. Every two target template frames and search frames form a template-search image pair. In a template-search image pair, if both the target template frame and the search frame contain the target object, then the target template frame and the search frame match (the template-search image pair is a positive sample); if the target template frame contains the target object but the search frame does not, then the target template frame and the search frame do not match (the template-search image pair is a negative sample). Based on this, a first label is set for each template-search image pair to indicate whether the target template frame and the search frame match or do not match. A second label is set by labeling the search frame with a target bounding box representing the true location of the target object. Finally, a training dataset consisting of multiple positive samples and multiple negative samples is formed.

[0058] Figure 2 This is a schematic diagram of the network structure of a target tracker in one embodiment of the present invention. Figure 2 As shown, the encoder employs a visual Transformer network architecture. The Transformer network segments the template and search image pair into local patches and embeds them into a set of visual tokens, which are aggregated through multiple cross-attention blocks. By performing a scaled dot product on the query-key tokens and interacting with the softmax normalization of the value tokens, the feature interaction between the template and search image pair can be enhanced. This allows for the accurate capture of discriminative features used to precisely locate the target object and clearly distinguish the target object from the background, thus establishing a highly discriminative feature representation.

[0059] In some embodiments, the decoder includes a feature pyramid network, a target localization branch, and a target localization uncertainty prediction branch. The feature pyramid network is used to upsample the encoded search image features. The target localization branch is used to estimate the target position in the upsampled encoded search image features and output the corresponding target position prediction result. The target localization uncertainty prediction branch is used to generate the target localization uncertainty based on the upsampled encoded search image features.

[0060] In some embodiments, the prototype memory network includes a global average pooling layer, a confidence inversion module, a splicing module, a convolutional layer, a masking module, a weighted operation module, a target prototype memory, a memory reader, a target prototype aggregation module, and a multilayer perceptron.

[0061] The global average pooling layer is used to compress the encoded target template features and output the target prototype accordingly.

[0062] The confidence inversion module is used to convert the target positioning uncertainty into a confidence feature;

[0063] The splicing module is used to splice the encoded search image features and the corresponding confidence features, and output the spliced ​​features accordingly.

[0064] The convolutional layer is used to encode the visual semantics and location confidence in the concatenated features, and outputs the encoded features accordingly.

[0065] The masking module is used to mask the target location prediction result and output the target location mask result accordingly.

[0066] The weighted operation module is used to perform weighted operations on the target prototype, the corresponding encoded features, and the target position mask result, and output the target weighted prototype accordingly.

[0067] The target prototype memory is constructed from multiple historical target weighted prototypes, which are a subset of target weighted prototypes selected from multiple output target weighted prototypes.

[0068] The memory reader is used to extract from the target prototype memory a number of historical target weighted prototypes that have a high similarity to the output target weighted prototype;

[0069] The target prototype aggregation module is used to concatenate multiple historical target weighted prototypes that have a high similarity to the output target weighted prototype into a target combination prototype, and aggregate the target combination prototype and the corresponding target weighted prototype based on the cross attention mechanism to output the target aggregation prototype.

[0070] The multilayer perceptron is used to generate a confidence score based on the target aggregate prototype. If the confidence score is greater than a preset threshold and the matching result and corresponding template are consistent with the first label of the search image pair, then the target weighted prototype corresponding to the confidence score can be used as a historical target weighted prototype.

[0071] In some embodiments, the neural network model is iteratively trained by minimizing the target localization uncertainty loss function of the decoder and the cross-entropy loss function of the prototype memory network.

[0072] like Figure 2 As shown, this embodiment uses an Uncertainty-aware Localization Decoder (ULD) to model the localization uncertainty of the target bounding box corners. Let B = (x tl ,y tl ,x br ,y br ) represents the corner points of the bounding box of the target object, where (x tl ,y tl () represents the top left corner, (x) br ,y br The bottom right corner (can also be represented by the bottom left and top right corners) is represented by ). Since this method can independently optimize each coordinate, the coordinates of the bounding box B are simplified to x. The encoded search image features F... s The input is fed into a Feature Pyramid Network (FPN) to upsample the encoded search image features to a higher resolution. The upsampled features are then fed into the target bounding box corner localization branch and the target localization uncertainty prediction branch. The target bounding box corner localization branch uses multiple sequentially connected convolutional layers, normalization layers, and activation function layers (Conv-BN-ReLU layers, with convolutional, normalization, and activation function layers also sequentially connected) to generate a 2×H×W (H is height, W is width, C is channel or depth, C=2) target location prediction probability map to estimate the positions of the top-left and bottom-right corners (or bottom-left and top-right corners) of the target bounding box. The target localization uncertainty prediction branch generates a 4×H×W target localization uncertainty distribution map F. uc The four channels represent the location uncertainty of the four corner coordinates of the target bounding box. Specifically, the location uncertainty is represented by a univariate Gaussian distribution, as shown in the following formula:

[0073]

[0074] Here, μ represents the learnable parameters, μ represents the predicted or estimated values ​​of the target bounding box corner coordinates (target location coordinates), σ represents the standard deviation of the predicted or estimated target location coordinates, and it is important to note that this standard deviation σ also measures the positioning uncertainty of the target bounding box corners (target positioning uncertainty), and x represents the target bounding box corner coordinates (target location coordinates). If the standard deviation σ approaches 0, it means that the model is extremely confident in the predicted target bounding box corner positions.

[0075] The true values ​​of the target bounding box corner coordinates (target position coordinates) are represented by a simple Dirac delta distribution, as shown in the following formula:

[0076] P gt (x)=δ(x-μ gt )

[0077] Where, μ gt The true values ​​representing the corner coordinates of the target bounding box (target position coordinates) are called the second label. During model training, to predict the parameter θ, the target localization uncertainty distribution P is minimized. θ The Dirac distribution P of (x) and the true values ​​of the target location coordinates gt The KL divergence between (x) is calculated as follows:

[0078]

[0079] The KL divergence in the above equation measures the probability difference between the predicted target location uncertainty and the true target location coordinates. Therefore, a target location uncertainty loss function is introduced for the regression and estimation of the target bounding box corner coordinates (target location coordinates), as follows:

[0080]

[0081] Among them, L uc H(P) represents the loss function for target localization uncertainty. gt (x) is P gt The entropy of (x). Because H(P) gt (x)) and log(2π) / 2 do not depend on the estimated parameters, therefore the target localization uncertainty loss function can be simplified to the following expression:

[0082]

[0083] Here, parameters μ and σ are predicted by the target bounding box corner localization branch and the target localization uncertainty prediction branch in the decoder ULD, respectively. Unlike previous methods that directly predict the relative offset or corner coordinates of the target bounding box, this decoder simultaneously estimates the coordinates of the target bounding box corners and the localization uncertainty. The loss function L in this expression... uc It can be seen that the goal of ULD is to minimize localization uncertainty and to bring the target bounding box corners back to their true values. If the target state estimation is inaccurate, i.e., μ-μ... gt It will get bigger, so in order to make L uc As the target localization uncertainty loss function decreases, the tracker tends to predict a larger localization uncertainty σ, leading to tracking failure. By minimizing the target localization uncertainty loss function described above, the decoder is iteratively trained to make its output localization uncertainty very small, and the predicted values ​​of the output target position coordinates (target bounding box corner coordinates) approximate the true values.

[0084] Although the Uncertainty-Aware Localizer (ULD) can predict the localization uncertainty of the target bounding box corners (target location), how to fully utilize this uncertainty information to maintain reliable target state estimation remains unclear. To address this, this method proposes a Prototype Memory Network (PMN), which achieves reliable target localization and tracking by using historical template representations (historical target weighted prototypes). Figure 2 As shown, in order to construct the prototype representation (also called the template prototype representation) of the target object, the encoded target template features are first processed through a global average pooling layer. The compression formula is as follows:

[0085] P = GAP(F) t )

[0086] Where P represents the target prototype, and GAP(·) represents the global average pooling operator. The output target prototype P∈R C ×1×1 It can be viewed as a compact prototype representation that encodes the channel weights of the target object. Then, the target localization uncertainty distribution map F is transformed using the confidence inversion module (CIM). uc ∈R 4×H×W Convert to confidence feature F c ∈R 1×H×W The formula is as follows:

[0087] F c =Sigmoid(Conv(1-F) uc ))

[0088] Where Conv(·) denotes the Conv-BN-ReLU layer. Correspondingly, the output confidence feature F... c The reliability of spatial localization in the feature domain of the encoded search image was measured. For F c For each pixel in the image, a higher response indicates that the location corresponding to that pixel is more likely to belong to the foreground target object with a higher confidence level. Next, the stitching module (C) combines the confidence features F... c The encoded search image features F corresponding to the input s The concatenated features are then processed by a convolutional layer (F) to encode the concatenated features. This convolutional layer simultaneously encodes the visual semantics and location confidence of the search image, outputting the encoded features F, as shown in the following formula:

[0089] F = Conv 1×1 (Concat(F c F s ))

[0090] Among them, Conv 1×1 It is a 1×1 convolutional layer, designed to recover the size of the feature channels. To enhance the foreground region (containing the target object), the masking module generates a target-specific target location mask map M∈R from the target location prediction probability map output by the decoder. 1×H×W The specific masking operation involves marking each pixel within the target bounding box on the target location prediction probability map as 0, and marking other areas as -∞. Then, the weighted operation module multiplies the target prototype P with the encoded feature F, and combines this with the target-specific target location mask map M to enhance the encoded feature F. The calculation formula is as follows:

[0091]

[0092] Among them, P * ∈R C×1×1 This represents the reweighted target prototype, i.e., the target-weighted prototype, which highlights features with high confidence in the foreground region.

[0093] To maintain reliable target localization and target state prediction, a target prototype memory is constructed to model the temporal dependency of the embedded target prototypes. Specifically, K historical frames (template-search image pairs) are taken as guides and projected into target-weighted prototypes (as historical target-weighted prototypes), thereby establishing a target prototype memory M = {P0, P1, ..., P...}. K Next, the memory reader performs a memory read operation, calculating the target weighted prototype P. * The cosine similarity between the target and each historical target weighted prototype is used to retrieve and extract the target weighted prototype P from the target prototype memory. * The most relevant historical target weighted prototype is calculated using the following formula:

[0094]

[0095] Among them, P i This is the i-th prototype in the target prototype memory M. Next, k historical target weighted prototypes with high similarity are selected to determine the reliability of the target weighted prototype. Generally, if the target prototype differs significantly from the top k candidate prototypes in the similarity ranking, it indicates a large appearance change between the target prototype and the most relevant historical prototype, making it difficult to maintain a reliable target prototype for robust tracking. Therefore, this method concatenates the selected top k candidate prototypes into a target combined prototype P. g ∈R C×k and using a lightweight Transformer in P * and P gA cross-attention mechanism is executed between them to aggregate the target prototype representation, as shown in the following formula:

[0096]

[0097] in, This represents the target prototype after aggregation, i.e., the target aggregated prototype; φ q (·), φ k (·) and φ v (·) represent learnable linear functions used to project the target prototype into a query-key-value structure. The cross-attention mechanism in the above formula models the temporal correlation between the target-weighted prototype and the target-combined prototype. To identify its reliability, it can be based on the target-aggregated prototype using two multilayer perceptron (MLP) layers. To generate a two-dimensional confidence score, this score can be trained using the cross-entropy loss function, the formula for which is as follows:

[0098]

[0099] Among them, L pro N represents the cross-entropy loss function; B This represents the number of template-search image pairs in a batch, where i represents the index of the template-search image pair in the batch, i = 1, 2, ..., N. B ;y i p represents the first label of the i-th template and the search image pair; i This represents the confidence level or confidence score of the target aggregate prototype corresponding to the i-th template and search image pair. If the confidence level is greater than a preset threshold, it is considered reliable, meaning that the search frame in the corresponding template and search image pair matches the target template frame, that is, the search frame contains the target object to be tracked, and the confidence level p i The matching result and the corresponding first label y are represented. i When the matching results are consistent, the target weighted prototype corresponding to this confidence level can be used as a historical target weighted prototype for constructing the target prototype memory. If the confidence level is less than a preset threshold, it is considered unreliable, meaning that the search frame and target template frame in the corresponding template and search image pair do not match, i.e., the search frame does not contain the target object to be tracked, and therefore the target prototype and target weighted prototype cannot be obtained from it. By minimizing the above cross-entropy loss function, the prototype memory network is iteratively trained so that the result of whether the target template frame and search frame in the template and search image pair, represented by the confidence level output, matches, is consistent with the first label set for the template and search image pair.

[0100] Step S120: The trained neural network model is determined as a target tracker for outputting the target tracking results corresponding to the search frames in the template and search image pair. The target tracking results include the confidence level for representing the matching results and the target position prediction results in the matching case.

[0101] The target tracker is obtained by training the neural network model using a training dataset consisting of multiple positive and multiple negative samples. The trained prototype memory network (PMN) is used to output the confidence score representing whether the corresponding template matches the target template frame and the search frame in the search image pair. The trained decoder (ULD) is used to output the target location prediction probability map (target bounding box position) when the template matches the target template frame and the search frame in the search image pair, thus achieving accurate target tracking even when the target's appearance changes significantly.

[0102] Figure 3 This is a schematic flowchart of a target tracking method according to one embodiment of the present invention. The present invention also provides a target tracking method, such as... Figure 3 As shown, the method includes the following steps:

[0103] Step S310: Obtain the video generated for the target motion.

[0104] Step S320: Mark the target in the initial frame of the video, and use the marked initial frame as the target template frame, and use the unmarked frames in the video as each search frame.

[0105] In this step, the bounding boxes of the targets in the initial frame are marked to obtain the initial frame with the target bounding box marked, and this frame is used as the target template frame. The remaining unmarked frames in the video are used as the search frames.

[0106] Step S330: The target template frame and each search frame are respectively input into the target tracker as multiple templates and search image pairs, so that the target tracker outputs the target tracking result for each search frame. The target tracking result includes a confidence level for representing the matching result and a target position prediction result in the case of matching. When the confidence level indicates that the target template frame and the search frame in the corresponding template and search image pair match, the target prototype memory is updated. The target tracker is pre-trained based on the target tracker training method described in any of the foregoing embodiments.

[0107] In some embodiments, in step S330, where the target template frame and each search frame are input into the target tracker as multiple templates and search image pairs respectively, so that the target tracker outputs the target tracking results for each search frame, if the output confidence level is greater than a preset threshold, the target prototype memory is updated based on the target weighted prototype corresponding to that confidence level. In this embodiment, if the confidence level is less than the preset threshold, there may be two possible results: one is that the input search frame and the target template frame corresponding to that confidence level do not match, and the other is that the search range of the input search frame corresponding to that confidence level is small and the target object is not searched and tracked. Correspondingly, the search range in the search frame is expanded and then re-inputted into the target tracker along with the target template frame for detection and online tracking. During the update of the target prototype memory, if the maximum storage space of the target prototype memory is reached, the oldest target weighted prototype stored therein is popped out to store the updated target weighted prototype that is closest to the target state of the current search frame.

[0108] The target tracker of this invention is an uncertainty-aware Transformer tracker, referred to as UncTrack. During online target tracking in video, UncTrack effectively utilizes the positioning uncertainty output by ULD (Undefined Device) through PMN (Profile Name) for accurate target state inference and online tracking. Furthermore, it feeds back high-confidence, reliable samples to the target prototype memory for memory updates, adaptively updating the appearance representation of the target or template. This allows it to adapt to drastic changes in the target object's appearance, optimizes the estimation of the target object's tracking state, improves the robustness of the target tracker, and ultimately ensures reliable target estimation output by ULD. In other words, the target tracker of this invention achieves accurate online target tracking even under conditions of drastic changes in the target object's appearance (such as blurring, occlusion leading to incomplete target, and target deformation due to angle changes caused by target movement), and achieves a new level of performance.

[0109] Finally, the performance of the target tracker UncTrack of this invention was tested in four typical and complex target tracking scenarios. The performance and target tracking results of UncTrack were compared with those of several existing mainstream trackers (GRM, DiMP, ATOM, OSTrack, STMTrack, MixFormer, and ROMTrack). Figure 4 As shown. By Figure 4 It can be seen that, compared with the various existing trackers, the target tracking results output by the UncTrack target tracker are closest to the actual state of the target during the motion process. That is, the overlap between the target bounding box position predicted by UncTrack and the actual target bounding box position (the bounding box represented by GT) is the highest, achieving excellent performance and establishing a new advanced technical standard.

[0110] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the aforementioned target tracker training method or target tracking method.

[0111] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned target tracker training method or target tracking method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0112] This invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the aforementioned target tracker training method or target tracking method.

[0113] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0114] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0115] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a target tracker, characterized in that, The method includes: A neural network model is trained based on multiple template-search image pairs, each with its own first and second labels. Each template-search image pair includes a target template frame and a search frame. The first and second labels represent whether the target template frame and search frame match in each template-search image pair, and the actual target location in the search frame, respectively. The neural network model includes an encoder, a decoder, and a prototype memory network. The encoder encodes the template-search image pairs, outputting encoded target template features and encoded search image features. The decoder decodes the encoded search image features, outputting target localization uncertainty and target location prediction results. The prototype memory network constructs a target prototype memory based on the encoded target template features, the corresponding encoded search image features, target localization uncertainty, and target location prediction results. Based on the target prototype memory, a confidence level representing the matching result is generated. The target prototype memory is constructed when the confidence level indicates that the target template frame and search frame match in the corresponding template-search image pair and are consistent with the first label of the template-search image pair. The trained neural network model is determined as a target tracker for outputting the target tracking results corresponding to the search frames in the template and search image pair. The target tracking results include a confidence level representing the matching result and a target position prediction result in the matching case. The prototype memory network includes a global average pooling layer, a confidence inversion module, a splicing module, a convolutional layer, a masking module, a weighted operation module, a target prototype memory, a memory reader, a target prototype aggregation module, and a multilayer perceptron. The global average pooling layer is used to compress the encoded target template features and output the target prototype accordingly. The confidence inversion module is used to convert the target positioning uncertainty into a confidence feature; The splicing module is used to splice the encoded search image features and the corresponding confidence features, and output the spliced ​​features accordingly. The convolutional layer is used to encode the visual semantics and location confidence in the concatenated features, and outputs the encoded features accordingly. The masking module is used to mask the target location prediction result and output the target location mask result accordingly. The weighted operation module is used to perform weighted operations on the target prototype, the corresponding encoded features, and the target position mask result, and output the target weighted prototype accordingly. The target prototype memory is constructed from multiple historical target weighted prototypes, which are a subset of target weighted prototypes selected from multiple output target weighted prototypes. The memory reader is used to extract from the target prototype memory a number of historical target weighted prototypes that have a high similarity to the output target weighted prototype; The target prototype aggregation module is used to concatenate multiple historical target weighted prototypes that have a high similarity to the output target weighted prototype into a target combination prototype, and aggregate the target combination prototype and the corresponding target weighted prototype based on the cross attention mechanism to output the target aggregation prototype. The multilayer perceptron is used to generate a confidence score based on the target aggregate prototype. If the confidence score is greater than a preset threshold and the matching result and corresponding template are consistent with the first label of the search image pair, then the target weighted prototype corresponding to the confidence score can be used as a historical target weighted prototype.

2. The method according to claim 1, characterized in that, The decoder includes a feature pyramid network, a target localization branch, and a target localization uncertainty prediction branch. The feature pyramid network is used to upsample the features of the encoded search image; The target localization branch is used to estimate the target position in the upsampled encoded search image features and output the target position prediction result accordingly. The target localization uncertainty prediction branch is used to generate target localization uncertainty based on the upsampled encoded search image features.

3. The method according to claim 1, characterized in that, The neural network model is iteratively trained by minimizing the target localization uncertainty loss function of the decoder and the cross-entropy loss function of the prototype memory network.

4. The method according to claim 3, characterized in that, The expression for the target localization uncertainty loss function is: Among them, L uc This represents the loss function for target location uncertainty. The predicted value represents the target's location coordinates; The standard deviation of the predicted coordinates of the target location represents the uncertainty in target positioning. The actual value representing the target's location coordinates, i.e., the second label; The expression for the cross-entropy loss function is: Among them, L pro N represents the cross-entropy loss function; B This represents the number of template-search image pairs in a batch, where i represents the index of the template-search image pair in the batch, i=1,2,…,N. B ; This represents the first label of the i-th template and the search image pair; This represents the confidence level of the i-th template and the search image in the corresponding target aggregation prototype.

5. A target tracking method, characterized in that, The method includes: Obtain the video generated for the target motion; The targets in the initial frames of the video are marked, and the marked initial frames are used as target template frames, while the unmarked frames in the video are used as search frames. The target template frame and each search frame are respectively input into the target tracker as multiple templates and search image pairs, so that the target tracker outputs the target tracking result for each search frame. The target tracking result includes a confidence level for representing the matching result and a target position prediction result in the case of matching. When the confidence level indicates that the target template frame and the search frame in the corresponding template and search image pair match, the target prototype memory is updated. The target tracker is pre-trained based on the target tracker training method as described in any one of claims 1 to 4.

6. The method according to claim 5, characterized in that, In the step of inputting the target template frame and each search frame as multiple templates and search image pairs into the target tracker so that the target tracker outputs the target tracking results for each search frame, if the output confidence level is greater than a preset threshold, the target prototype memory is updated based on the target weighted prototype corresponding to the confidence level.

7. An electronic device comprising a processor, a memory, and computer instructions stored in the memory, characterized in that, The processor is configured to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method as described in any one of claims 1 to 6.

9. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 6.

Images