Target tracker training method and target tracking method

Abstract

The invention provides a target tracker training method and a target tracking method, and the training method comprises the steps: training a neural network model based on a plurality of template and search image pairs with respective labels, each template and search image pair comprising a target template frame and a search frame, the labels are used for representing whether each template is matched with a target template frame and a search frame in a search image pair or not and representing a real target position, and the neural network model comprises an encoder, a decoder and a prototype memory network; and determining the trained neural network model as a target tracker used for outputting a target tracking result corresponding to a template and a search frame in the search image pair, wherein the target tracking result comprises a confidence coefficient used for representing a matching result and a target position prediction result under the matching condition. According to the method, accurate target state reasoning can be carried out by utilizing positioning uncertainty, and reliable target appearance matching and target tracking results can still be ensured under the condition that the appearance of the target changes drastically.

Description

Technical Field

[0001] The present invention relates to the technical field of visual target tracking, and in particular to a target tracker training method and a target tracking method. Background Art

[0002] Visual object tracking (VOT) is an important task in computer vision with multiple practical applications such as intelligent video surveillance, autonomous driving, and robotics. Given an initial annotated bounding box of a target in the first frame, the goal of the visual object tracking task is to predict the bounding box of the target throughout the entire video. The tracker should construct a discriminative target appearance to distinguish the target object from background distractors in real time and be flexible to the challenge of target tracking failure caused by changes in the target object's appearance.

[0003] For decades, matching-based methods have become the mainstream solution due to their good balance between accuracy and efficiency. Such methods approach visual object tracking from the perspective of similarity comparison, by learning a two-stream network to compute the similarity between the template and the search image to estimate the object state.

[0004] Transformer-based object trackers have become the mainstream tracking paradigm due to their accuracy and efficiency. Despite some substantial progress, there are still two defects: First, during the motion of the target, the appearance of the target object will change dramatically over time, such as blurring, occlusion, resulting in incompleteness of the target object, and changes in the angle (deformation) of the target object due to motion, etc., and unreliable target positioning and tracking prediction results may accumulate throughout the video. Some existing methods, such as LTMU, Siam R-CNN, and KeepTrack, can alleviate this problem by introducing online meta-updates, object re-detection, or online appearance association mechanisms. However, these methods rely on local smoothness assumptions to maintain the temporal consistency of the target object (searching for the target position in the next frame within the local range of the target position in the previous frame). For fast-moving targets, it is easy to fall into suboptimal solutions, thereby misleading the target appearance matching and ultimately leading to tracking failure. Second, the target may become blurred, occluded, and deformed during motion, which increases the positioning uncertainty of the target bounding box. Most methods usually treat target tracking as a deterministic coordinate regression problem, seriously ignoring the problem of target positioning uncertainty, which prevents the tracker from maintaining reliable target state prediction in challenging scenarios. Some methods such as UAST and UAF use distribution regression representation and Monte Carlo dropout technology to model positioning uncertainty, but it is still unclear how to effectively use uncertainty output to obtain reliable target estimation, and the tracking effect of blurred, occluded and deformed targets is not good. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a target tracker training method and a target tracking method to eliminate or improve one or more defects in the prior art.

[0006] A first aspect of the present invention provides a target tracker training method, the method comprising the following steps:

[0007] A neural network model is trained based on a plurality of templates and search images each having a first label and a second label,

[0008] Wherein, each template and search image pair includes a target template frame and a search frame, and the first label and the second label are respectively used to indicate whether the target template frame and the search frame in each template and search image pair match, and the real target position in the search frame; the neural network model includes an encoder, a decoder and a prototype memory network, the encoder is used to encode the template and search image pair, and correspondingly output the encoded target template features and the encoded search image features; the decoder is used to decode the encoded search image features, and correspondingly output the target positioning uncertainty and the target position prediction result; the prototype memory network is used to construct a target prototype memory library according to the encoded target template features and the encoded search image features corresponding to the encoded target template features, the target positioning uncertainty and the target position prediction result, and generate a confidence for representing the matching result based on the target prototype memory library, and construct the target prototype memory library when the confidence indicates that the target template frame and the search frame in the corresponding template and search image pair match and are consistent with the first label of the template and search image pair;

[0009] The trained neural network model is determined as a target tracker for outputting target tracking results corresponding to a search frame in a template and a search image pair, wherein the target tracking results include a confidence level for indicating a matching result and a target position prediction result in a matching situation.

[0010] In some embodiments of the present invention, the decoder includes a feature pyramid network, a target location branch, and a target location uncertainty prediction branch.

[0011] The feature pyramid network is used to upsample the encoded search image features;

[0012] The target location branch is used to estimate the target position in the upsampled encoded search image features, and output the target position prediction result accordingly;

[0013] The target location uncertainty prediction branch is used to generate target location uncertainty based on the upsampled encoded search image feature correspondence.

[0014] In some embodiments of the present invention, the prototype memory network includes a global average pooling layer, a confidence inversion module, a splicing module, a convolutional layer, a mask module, a weighted operation module, a target prototype memory bank, a memory reader, a target prototype aggregation module and a multi-layer perceptron.

[0015] The global average pooling layer is used to compress the encoded target template features and output the target prototype accordingly;

[0016] The confidence inversion module is used to convert the target positioning uncertainty into a confidence feature;

[0017] The splicing module is used to splice the encoded search image features and the corresponding confidence features, and output the spliced ​​features accordingly;

[0018] The convolution layer is used to encode the visual semantics and the location confidence in the concatenated features, and output the encoded features accordingly;

[0019] The mask module is used to mask the target position prediction result and output the target position mask result accordingly;

[0020] The weighted operation module is used to perform weighted operation on the target prototype and the corresponding encoded features and the target position mask result, and output the corresponding target weighted prototype;

[0021] The target prototype memory library is constructed by a plurality of historical target weighted prototypes, and the plurality of historical target weighted prototypes are a part of target weighted prototypes selected from the outputted plurality of target weighted prototypes;

[0022] The memory reader is used to extract from the target prototype memory bank a plurality of historical target weighted prototypes having a high similarity to the output target weighted prototype;

[0023] The target prototype aggregation module is used to splice multiple historical target weighted prototypes with high similarity to the output target weighted prototype into a target combination prototype, and aggregate the target combination prototype and the corresponding target weighted prototype based on the cross attention mechanism, and output the target aggregation prototype accordingly;

[0024] The multilayer perceptron is used to generate confidence based on the target aggregate prototype. If the confidence is greater than a preset threshold and the matching result represented and the corresponding template are consistent with the first label of the search image pair, the target weighted prototype corresponding to the confidence can be used as the historical target weighted prototype.

[0025] In some embodiments of the present invention, the neural network model is iteratively trained by minimizing the target positioning uncertainty loss function of the decoder and the cross entropy loss function of the prototype memory network.

[0026] In some embodiments of the present invention, the expression of the target positioning uncertainty loss function is:

[0027]

[0028] Among them, L uc represents the target positioning uncertainty loss function; μ represents the predicted value of the target position coordinates; σ represents the standard deviation of the predicted value of the target position coordinates, that is, the target positioning uncertainty; μ gt Represents the true value of the target position coordinates, i.e., the second label;

[0029] The expression of the cross entropy loss function is:

[0030]

[0031] Among them, L pro represents the cross entropy loss function; N B Represents the number of template and search image pairs in a batch, i represents the sequence number of template and search image pairs in a batch, i = 1, 2, ..., N B ;y i represents the first label of the i-th template and search image pair; p i Represents the confidence of the target aggregate prototype corresponding to the i-th template and search image pair.

[0032] A second aspect of the present invention provides a target tracking method, the method comprising the following steps:

[0033] Get the video generated for the target motion;

[0034] Marking the target in the initial frame of the video, and using the marked initial frame as the target template frame, and using the unmarked frames in the video as various search frames;

[0035] The target template frame and each search frame are respectively input into the target tracker as multiple template and search image pairs, so that the target tracker outputs the target tracking results of each search frame accordingly, and the target tracking results include a confidence level for representing the matching result and a target position prediction result in the case of a match. When the confidence level indicates that the target template frame and the search frame in the corresponding template and search image pair match, the target prototype memory library is updated, wherein the target tracker is pre-trained based on the target tracker training method described in the first aspect above.

[0036] In some embodiments of the present invention, the target template frame and each search frame are respectively input into the target tracker as multiple template and search image pairs, so that in the step that the target tracker outputs the target tracking results of each search frame accordingly, if the output confidence is greater than a preset threshold, the target prototype memory bank is updated based on the target weighted prototype corresponding to the confidence.

[0037] The third aspect of the present invention provides an electronic device, which includes: a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the device implements the steps of the target tracker training method described in the first aspect, or implements the steps of the target tracking method described in the second aspect.

[0038] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target tracker training method described in the first aspect, or implements the steps of the target tracking method described in the second aspect.

[0039] The fifth aspect of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the target tracker training method described in the first aspect, or implement the steps of the target tracking method described in the second aspect.

[0040] The target tracker training method and target tracking method of the present invention can obtain reliable target estimation and tracking results by utilizing positioning uncertainty to perform accurate target state inference, and can still ensure reliable target appearance matching results and target tracking results when the target appearance changes dramatically.

[0041] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0042] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0044] Figure 1 Schematic diagram of the process of a target tracker training method in one embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the network structure of a target tracker in one embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a target tracking method according to an embodiment of the present invention;

[0047] Figure 4 The figure is a schematic diagram comparing target tracking results of a target tracker in one embodiment of the present invention and multiple existing mainstream trackers. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0049] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0050] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0051] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0052] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0053] In order to address the challenge of target tracking failure caused by changes in the target's appearance during motion (becoming blurred, occluded, deformed, etc.), and effectively utilize positioning uncertainty output to obtain reliable target estimation, target matching and target tracking results, an embodiment of the present invention provides a target tracker training method and a target tracking method based on uncertainty perception, which uses positioning uncertainty information to perform accurate target state inference, and can still ensure reliable target appearance matching results and target tracking results when the target appearance changes drastically.

[0054] Figure 1FIG. 1 is a flow chart of a target tracker training method according to an embodiment of the present invention. Figure 1 As shown, the target tracker training method includes the following steps:

[0055] Step S110, training a neural network model based on a plurality of templates with respective first labels and second labels and search images,

[0056] Wherein, each template and search image pair includes a target template frame and a search frame, and the first label and the second label are respectively used to indicate whether the target template frame and the search frame in each template and search image pair match, and the actual target position in the search frame; the neural network model includes an encoder, a decoder and a prototype memory network, the encoder is used to encode the template and search image pair, and correspondingly output the encoded target template features and the encoded search image features; the decoder is used to decode the encoded search image features, and correspondingly output the target positioning uncertainty and the target position prediction result; the prototype memory network is used to construct a target prototype memory library according to the encoded target template features and the encoded search image features corresponding to the encoded target template features, the target positioning uncertainty and the target position prediction result, and generate a confidence for representing the matching result based on the target prototype memory library, and construct the target prototype memory library when the confidence indicates that the target template frame and the search frame in the corresponding template and search image pair match and are consistent with the first label of the template and search image pair.

[0057] Specifically, the training data set used to train the model is formed by a plurality of templates and search image pairs each having a first label and a second label. In the specific implementation process, the training data set can be obtained in the following manner: first, a video shot for the motion of a plurality of different objects is obtained. The video can be shot for the motion of a plurality of different objects at the same time, or can be a plurality of videos shot for the motion of each object in a plurality of different objects. One of the objects is determined as the target object to be tracked from the plurality of different objects, and the other objects are regarded as interference objects. Then, two frames of images are randomly selected from the acquired video multiple times, one of which contains the target object and is used as the target template frame, and the other contains or does not contain the target object and is used as the search frame. Every two target template frames and search frames form a template and search image pair, and it is considered that in a template and search image pair, if the target template frame and the search frame both contain the target object, then the target template frame and the search frame match (the template and the search image pair is a positive sample); if the target template frame contains the target object and the search frame does not contain the target object, then the target template frame and the search frame do not match (the template and the search image pair is a negative sample), and accordingly, a first label is set for each template and search image pair to mark whether the target template frame and the search frame therein match or do not match, and a second label is set by marking the search frame therein with a target bounding box representing the true position of the target object, and finally a training data set consisting of multiple positive samples and multiple negative samples is formed.

[0058] Figure 2 FIG. 1 is a schematic diagram of the network structure of a target tracker in one embodiment of the present invention. Figure 2 As shown in the figure, the encoder adopts the visual Transformer network architecture. The Transformer network divides the template and search image pairs into local patches and embeds them into a set of visual tokens, which are aggregated through multiple cross-attention blocks. By scaling the query-key token and interacting with the softmax normalization of the value token, the feature interaction between the template and the search image pair can be enhanced, so that the discriminative features used to accurately locate the target object and clearly distinguish the target object from the background can be accurately captured to establish a highly discriminative feature representation.

[0059] In some embodiments, the decoder includes a feature pyramid network, a target positioning branch and a target positioning uncertainty prediction branch, wherein the feature pyramid network is used to upsample the encoded search image features; the target positioning branch is used to estimate the target position in the upsampled encoded search image features and output the target position prediction result accordingly; the target positioning uncertainty prediction branch is used to generate target positioning uncertainty based on the upsampled encoded search image features.

[0060] In some embodiments, the prototype memory network includes a global average pooling layer, a confidence inversion module, a splicing module, a convolutional layer, a mask module, a weighted operation module, a target prototype memory bank, a memory reader, a target prototype aggregation module and a multi-layer perceptron.

[0061] The global average pooling layer is used to compress the encoded target template features and output the target prototype accordingly;

[0062] The confidence inversion module is used to convert the target positioning uncertainty into a confidence feature;

[0063] The splicing module is used to splice the encoded search image features and the corresponding confidence features, and output the spliced ​​features accordingly;

[0064] The convolution layer is used to encode the visual semantics and the location confidence in the concatenated features, and output the encoded features accordingly;

[0065] The mask module is used to mask the target position prediction result and output the target position mask result accordingly;

[0066] The weighted operation module is used to perform weighted operation on the target prototype and the corresponding encoded features and the target position mask result, and output the corresponding target weighted prototype;

[0067] The target prototype memory library is constructed by a plurality of historical target weighted prototypes, and the plurality of historical target weighted prototypes are a part of target weighted prototypes selected from the outputted plurality of target weighted prototypes;

[0068] The memory reader is used to extract from the target prototype memory bank a plurality of historical target weighted prototypes having a high similarity to the output target weighted prototype;

[0069] The target prototype aggregation module is used to splice multiple historical target weighted prototypes with high similarity to the output target weighted prototype into a target combination prototype, and aggregate the target combination prototype and the corresponding target weighted prototype based on the cross attention mechanism, and output the target aggregation prototype accordingly;

[0070] The multilayer perceptron is used to generate confidence based on the target aggregate prototype. If the confidence is greater than a preset threshold and the matching result represented and the corresponding template are consistent with the first label of the search image pair, the target weighted prototype corresponding to the confidence can be used as the historical target weighted prototype.

[0071] In some embodiments, the neural network model is iteratively trained by minimizing the target positioning uncertainty loss function of the decoder and the cross entropy loss function of the prototype memory network.

[0072] like Figure 2 As shown, this embodiment uses an uncertainty-aware localization decoder (ULD) to model the localization uncertainty of the corner points of the target bounding box. Let B = (x tl ,y tl ,x br ,y br ) is the corner point representation of the bounding box of the target object, where (x tl ,y tl ) indicates the upper left corner, (x br ,y br ) represents the lower right corner (it can also be represented by the lower left corner and the upper right corner); since this method can optimize each coordinate independently, the coordinate of the bounding box B is simplified to x. The encoded search image feature F s Input into the feature pyramid network (FPN) so that the feature pyramid network upsamples the encoded search image features to a higher resolution. Then the upsampled features are respectively input into the target bounding box corner location branch and the target location uncertainty prediction branch, wherein the target bounding box corner location branch uses multiple groups of sequentially connected convolutional layers, normalization layers, and activation function layers (Conv-BN-ReLU layers, convolutional layers, normalization layers, and activation function layers are also sequentially connected) to generate a 2×H×W (H is height, W is width, C is channel or depth, C=2) target location prediction probability map to estimate the position of the upper left corner and lower right corner (or, lower left corner and upper right corner) of the target bounding box; the target location uncertainty prediction branch generates a 4×H×W target location uncertainty distribution map F uc , where the four channels represent the positioning uncertainty of the four corner coordinates of the target bounding box. Specifically, the positioning uncertainty is represented by a univariate Gaussian distribution, and the formula is as follows:

[0073]

[0074] Among them, mortar represents a learnable parameter, μ represents the predicted value or estimated value of the target bounding box corner coordinates (target position coordinates); σ represents the standard deviation of the predicted value or estimated value of the target position coordinates. It should be noted that the standard deviation σ also measures the positioning uncertainty of the target bounding box corners (target positioning uncertainty); x represents the target bounding box corner coordinates (target position coordinates). If the standard deviation σ tends to 0, it means that the model is extremely confident in the predicted target bounding box corner positions.

[0075] The true value of the target bounding box corner coordinates (target position coordinates) is represented by a simple Dirac delta distribution, as follows:

[0076] P gt (x) = δ(x-μ gt )

[0077] Among them, μ gt Represents the true value of the target bounding box corner coordinates (target position coordinates), that is, the second label. During the model training process, in order to predict the parameter θ, minimize the target positioning uncertainty distribution P θ (x) and the Dirac distribution P of the true value of the target position coordinate gt The KL divergence between (x) is calculated as:

[0078]

[0079] The KL divergence in the above equation measures the probability difference between the predicted target location uncertainty and the true value of the target location coordinates. Therefore, the target location uncertainty loss function is introduced for the regression and estimation of the target bounding box corner coordinates (target location coordinates), as follows:

[0080]

[0081] Among them, L uc represents the target positioning uncertainty loss function, H(P gt (x)) is P gt (x). Since H(P gt (x)) and log(2π) / 2 do not depend on the estimated parameters, so the target positioning uncertainty loss function can be simplified to the following expression:

[0082]

[0083] Among them, the parameters μ and σ are predicted by the target bounding box corner location branch and the target location uncertainty prediction branch in the decoder ULD, respectively. Unlike the previous method of directly predicting the relative offset or corner coordinates of the target bounding box, this decoder estimates the coordinates of the target bounding box corners and the location uncertainty at the same time. The loss function L of this expression is uc It can be seen that the goal of ULD is to minimize the positioning uncertainty and regress the corner points of the target bounding box to the true value. gt will become larger, so in order to make L uc When σ becomes smaller, the tracker tends to predict a larger positioning uncertainty σ, which leads to tracking failure. By minimizing the above target positioning uncertainty loss function, the decoder is iteratively trained so that the positioning uncertainty of its output becomes very small and the predicted value of the output target position coordinates (target bounding box corner coordinates) is close to the true value.

[0084] Although the uncertainty-aware localization decoder (ULD) can predict the localization uncertainty of the target bounding box corners (target position), it is still unclear how to make full use of this uncertainty information to maintain a reliable target state estimation. To this end, this method proposes a prototype memory network (PMN), which achieves reliable target localization and tracking by using a historical template representation (historical target weighted prototype). Figure 2 As shown in Figure 1, in order to construct the prototype representation of the target object (also called the template prototype representation), the encoded target template features are firstly processed through the global average pooling layer. For compression, the formula is as follows:

[0085] P=GAP(F t )

[0086] Where P represents the target prototype and GAP(·) represents the global average pooling operator. The output target prototype P∈R C ×1×1 It can be regarded as a compact prototype representation that encodes the channel weights of the target object. Then, the target positioning uncertainty distribution map F is transformed into uc ∈R 4×H×W Converted to confidence feature F c ∈R 1×H×W , the formula is as follows:

[0087] F c =Sigmoid(Conv(1-F uc ))

[0088] Where Conv(·) represents the Conv-BN-ReLU layer. Correspondingly, the output confidence feature F c It measures the spatial localization reliability of the input encoded search image feature domain. c For each pixel in , a higher response indicates that the position corresponding to the pixel is more likely to belong to the foreground target object with a higher confidence. Then, the concatenation module (C) converts the confidence feature F c The encoded search image feature F corresponding to the input s The concatenated features are concatenated and encoded through a convolutional layer (F). The convolutional layer encodes the visual semantics and location confidence of the search image at the same time, and outputs the encoded feature F. The formula is as follows:

[0089] F=Conv 1×1 (Concat(F c , F s ))

[0090] Among them, Conv 1×1 It is a 1×1 convolutional layer, which aims to restore the size of the feature channel. In order to enhance the foreground area (including the target object), the mask module generates a target-specific target position mask map M∈R from the target position prediction probability map output by the decoder. 1×H×W , the specific mask operation is to mark each pixel in the target bounding box on the target position prediction probability map as 0, and mark other areas as -∞. The weighted operation module then multiplies the target prototype P with the encoded feature F, and combines the target-specific target position mask map M to enhance the encoded feature F. The calculation formula is as follows:

[0091]

[0092] Among them, P * ∈R C×1×1 Represents the re-weighted target prototype, i.e., the target weighted prototype, which highlights the features with high confidence in the foreground region.

[0093] In order to maintain reliable target localization and target state prediction, a target prototype memory is constructed to model the temporal dependency of the embedded target prototypes. Specifically, K historical frames (template and search image pairs) are taken as guides and projected as target weighted prototypes (as historical target weighted prototypes), thereby building a target prototype memory M = {P 0 , P 1 , ..., P K}. Then the memory reader performs the memory reading operation and calculates the target weighted prototype P * The cosine similarity between the weighted prototypes of the target and each historical target is used to retrieve and extract the target weighted prototype P from the target prototype memory bank. * The most relevant historical target weighted prototype is given by the following formula:

[0094]

[0095] Among them, P i is the i-th prototype in the target prototype memory M. Next, k historical target weighted prototypes with high similarity are selected to determine whether the target weighted prototype is reliable. Generally, if the target prototype is far away from the first k candidate prototypes in the descending similarity order, it means that the appearance of the target prototype has changed greatly from the most relevant historical prototype, which makes it difficult to maintain a reliable target prototype for robust tracking. Therefore, this method splices the first k candidate prototypes selected into a target combined prototype P g ∈R C×k , and use lightweight Transformer in P* and P g The cross attention mechanism is performed between them to aggregate the target prototype representation. The formula is as follows:

[0096]

[0097] in, represents the target prototype after aggregation, that is, the target aggregation prototype; φ q (·),φ k (·) and φ v (·) represent learnable linear functions, respectively, which are used to project the target prototype into a query-key-value structure. The operation of the cross-attention mechanism in the above formula models the temporal correlation between the target weighted prototype and the target combined prototype. In order to identify its reliability, two multi-layer perceptrons (MLP layers) are used to aggregate the target prototypes. To generate a two-dimensional confidence (confidence score), which can be trained by the cross entropy loss function, the formula of which is as follows:

[0098]

[0099] Among them, L pro represents the cross entropy loss function; N B Represents the number of template and search image pairs in a batch, i represents the sequence number of template and search image pairs in a batch, i = 1, 2, ..., N B ;y i represents the first label of the i-th template and search image pair; p i represents the confidence or confidence score of the target aggregate prototype corresponding to the i-th template and search image pair. If the confidence is greater than the preset threshold, the confidence is considered to be credible, that is, the search frame in the corresponding template and search image pair matches the target template frame, that is, the search frame contains the target object to be tracked, and the confidence p i The matching result and the corresponding first label y i When the matching results represented are consistent, the target weighted prototype corresponding to the confidence can be used as the historical target weighted prototype for constructing the target prototype memory library; if the confidence is less than the preset threshold, the confidence is considered unreliable, that is, the search frame and the target template frame in the corresponding template and search image pair do not match, that is, the search frame does not contain the target object to be tracked, and the target prototype and the target weighted prototype cannot be obtained from it. By minimizing the above cross entropy loss function, the prototype memory network is iteratively trained so that the result of whether the target template frame and the search frame in the template and search image pair represented by the confidence output by it matches is consistent with the first label set corresponding to the template and search image pair.

[0100] Step S120, determining the trained neural network model as a target tracker for outputting target tracking results corresponding to the search frame in the template and search image pair, wherein the target tracking results include a confidence level for representing the matching result and a target position prediction result in a matching situation.

[0101] The target tracker is obtained by training the neural network model with a training data set consisting of multiple positive samples and multiple negative samples. The trained prototype memory network PMN is used to output the confidence of the result indicating whether the target template frame and the search frame in the corresponding template and search image pair match. The trained decoder ULD is used to output the target position prediction probability map (target bounding box position) when the target template frame and the search frame in the template and search image pair match, that is, to achieve accurate target tracking when the target appearance changes greatly.

[0102] Figure 3 FIG. 1 is a flow chart of a target tracking method in an embodiment of the present invention. The present invention also provides a target tracking method, such as Figure 3 As shown, the method comprises the following steps:

[0103] Step S310, obtaining a video generated for target motion.

[0104] Step S320, marking the target in the initial frame of the video, and using the marked initial frame as the target template frame, and using the unmarked frames in the video as various search frames.

[0105] In this step, the bounding box of the target in the initial frame is marked, and the initial frame with the target bounding box mark is obtained and used as the target template frame, and the remaining unmarked frames in the video are used as various search frames.

[0106] Step S330, the target template frame and each search frame are respectively input into the target tracker as multiple template and search image pairs, so that the target tracker outputs the target tracking results of each search frame accordingly, and the target tracking results include a confidence level for representing the matching result and a target position prediction result in the case of a match, and when the confidence level indicates that the target template frame and the search frame in the corresponding template and search image pair match, the target prototype memory library is updated, wherein the target tracker is pre-trained based on the target tracker training method as described in any of the aforementioned embodiments.

[0107] In some embodiments, in step S330, the target template frame and each search frame are respectively input into the target tracker as multiple templates and search image pairs, so that the target tracker outputs the target tracking results of each search frame, and if the output confidence is greater than a preset threshold, the target prototype memory is updated based on the target weighted prototype corresponding to the confidence. In this embodiment, if the confidence is less than the preset threshold, there may be two corresponding results: one is that the search frame input corresponding to the confidence does not match the target template frame, and the other is that the search range of the search frame input corresponding to the confidence is small, and the target object is not searched and tracked. Correspondingly, the search range in the search frame is expanded and re-input into the target tracker together with the target template frame for detection and online tracking. In the process of updating the target prototype memory, if the maximum storage space of the target prototype memory is reached, the oldest target weighted prototype stored therein is popped out to store the updated target weighted prototype closest to the target state of the current search frame.

[0108] The target tracker of the present invention is an uncertainty-aware Transformer tracker, called UncTrack. In the process of online tracking of targets in videos, UncTrack effectively utilizes the positioning uncertainty output by ULD through PMN to perform accurate target state reasoning and online tracking, and feeds back high-confidence and credible samples to the target prototype memory library for memory update, adaptively updates the appearance representation of the target or template, can adapt to drastic changes in the appearance of the target object, optimizes the estimation of the tracking state of the target object, improves the robustness of the target tracker, and finally enables the ULD to output reliable target estimates. That is, the target tracker of the present invention realizes accurate online target tracking under the condition of drastic changes in the appearance of the target object (target deformation such as incomplete target due to blurring or occlusion, and angle changes caused by target movement), and reaches a new performance level.

[0109] Finally, the performance of the target tracker UncTrack of the present invention is tested in four typical and complex target tracking scenarios, and the performance and target tracking results of the target tracker UncTrack are compared with those of multiple existing mainstream trackers (GRM, DiMP, ATOM, OSTrack, STMTrack, MixFormer and ROMTrack). Figure 4 As shown. Figure 4 It can be seen that compared with the above existing trackers, the target tracking result output by the target tracker UncTrack is closest to the actual state of the target during the motion process, that is, the target bounding box position predicted by UncTrack has the highest overlap with the real target bounding box position (bounding box represented by GT), achieving excellent performance and establishing a new advanced technology standard.

[0110] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the aforementioned target tracker training method or target tracking method.

[0111] The embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the steps of the aforementioned target tracker training method or target tracking method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0112] An embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the aforementioned target tracker training method or target tracking method.

[0113] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0114] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0115] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A target tracker training method, characterized in that: The method comprises: A neural network model is trained based on a plurality of template and search image pairs each having a first label and a second label, wherein each template and search image pair includes a target template frame and a search frame, and the first label and the second label are used to indicate whether the target template frame and the search frame in each template and search image pair match, and the actual target position in the search frame; the neural network model includes an encoder, a decoder and a prototype memory network, the encoder is used to encode the template and search image pair, and correspondingly output the encoded target template features and the encoded search image features; the decoder is used to decode the encoded search image features, and correspondingly output the target positioning uncertainty and the target position prediction result; the prototype memory network is used to construct a target prototype memory library according to the encoded target template features and the encoded search image features corresponding to the encoded target template features, the target positioning uncertainty and the target position prediction result, and correspondingly generate a confidence for representing the matching result based on the target prototype memory library, and construct the target prototype memory library when the confidence indicates that the target template frame and the search frame in the corresponding template and search image pair match and are consistent with the first label of the template and search image pair; The trained neural network model is determined as a target tracker for outputting target tracking results corresponding to a search frame in a template and a search image pair, wherein the target tracking results include a confidence level for indicating a matching result and a target position prediction result in a matching situation.

2. The method according to claim 1, characterized in that The decoder includes a feature pyramid network, a target positioning branch, and a target positioning uncertainty prediction branch. The feature pyramid network is used to upsample the encoded search image features; The target location branch is used to estimate the target position in the upsampled encoded search image features, and output the target position prediction result accordingly; The target location uncertainty prediction branch is used to generate target location uncertainty based on the upsampled encoded search image feature correspondence.

3. The method according to claim 1, characterized in that The prototype memory network includes a global average pooling layer, a confidence inversion module, a splicing module, a convolutional layer, a mask module, a weighted operation module, a target prototype memory bank, a memory reader, a target prototype aggregation module and a multi-layer perceptron. The global average pooling layer is used to compress the encoded target template features and output the target prototype accordingly; The confidence inversion module is used to convert the target positioning uncertainty into a confidence feature; The splicing module is used to splice the encoded search image features and the corresponding confidence features, and output the spliced ​​features accordingly; The convolution layer is used to encode the visual semantics and the location confidence in the concatenated features, and output the encoded features accordingly; The mask module is used to mask the target position prediction result and output the target position mask result accordingly; The weighted operation module is used to perform weighted operation on the target prototype and the corresponding encoded features and the target position mask result, and output the corresponding target weighted prototype; The target prototype memory library is constructed by a plurality of historical target weighted prototypes, and the plurality of historical target weighted prototypes are a part of target weighted prototypes selected from the outputted plurality of target weighted prototypes; The memory reader is used to extract from the target prototype memory bank a plurality of historical target weighted prototypes having a high similarity to the output target weighted prototype; The target prototype aggregation module is used to splice multiple historical target weighted prototypes with high similarity to the output target weighted prototype into a target combination prototype, and aggregate the target combination prototype and the corresponding target weighted prototype based on the cross attention mechanism, and output the target aggregation prototype accordingly; The multilayer perceptron is used to generate confidence based on the target aggregate prototype. If the confidence is greater than a preset threshold and the matching result represented and the corresponding template are consistent with the first label of the search image pair, the target weighted prototype corresponding to the confidence can be used as the historical target weighted prototype.

4. The method according to claim 1, characterized in that The neural network model is iteratively trained by minimizing the target positioning uncertainty loss function of the decoder and the cross entropy loss function of the prototype memory network.

5. The method according to claim 4, characterized in that The expression of the target positioning uncertainty loss function is: Among them, L uc represents the target positioning uncertainty loss function; μ represents the predicted value of the target position coordinates; σ represents the standard deviation of the predicted value of the target position coordinates, that is, the target positioning uncertainty; μ gt Represents the true value of the target position coordinates, i.e., the second label; The expression of the cross entropy loss function is: Among them, L pro represents the cross entropy loss function; N B Represents the number of template and search image pairs in a batch, i represents the sequence number of template and search image pairs in a batch, i = 1, 2, ..., N B ;y i represents the first label of the i-th template and search image pair; p i Represents the confidence of the target aggregate prototype corresponding to the i-th template and search image pair.

6. A target tracking method, characterized in that: The method comprises: Get the video generated for the target motion; Marking the target in the initial frame of the video, and using the marked initial frame as the target template frame, and using the unmarked frames in the video as various search frames; The target template frame and each search frame are respectively input into the target tracker as multiple template and search image pairs, so that the target tracker outputs the target tracking results of each search frame accordingly, and the target tracking results include a confidence level for representing the matching result and a target position prediction result in the case of a match. When the confidence level indicates that the target template frame and the search frame in the corresponding template and search image pair match, the target prototype memory library is updated, wherein the target tracker is pre-trained based on the target tracker training method as described in any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The target template frame and each search frame are respectively input into the target tracker as multiple template and search image pairs, so that the target tracker outputs the target tracking results of each search frame accordingly. If the output confidence is greater than a preset threshold, the target prototype memory bank is updated based on the target weighted prototype corresponding to the confidence.

8. An electronic device comprising a processor, a memory and computer instructions stored in the memory, characterized in that: The processor is used to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images