Single-target tracking method and device incorporating existence judgment with multiple image inputs and historical trajectory enhancement

Abstract

This invention discloses a single-target tracking method and apparatus that incorporates existence determination through multi-image input and historical trajectory enhancement. A network model for single-target tracking is constructed, comprising a modified micro-sliding converter, a bounding box embedding module, a multi-layer encoder module, a multi-layer decoder module, and a multi-layer perceptron. The network model is trained in two stages by inputting consecutive frames of images with known target bounding boxes into the single-target tracking network model. Then, the trained network model is used to process consecutive frames of images to be tracked, where only the first frame has a known target bounding box, and the remaining frames have unknown target bounding boxes. This invention can fuse target appearance information and target motion trajectory information through multi-image input and multi-bounding box input, making the tracking results more robust and accurate. The model effectively improves the speed and accuracy of target tracking, can determine the existence of the target, and has wide applications.

Description

Technical Field

[0001] This invention belongs to the field of single-target tracking, specifically involving a single-target tracking method and apparatus that incorporates existence judgment with multiple image inputs and historical trajectory enhancement. Background Technology

[0002] Single-object tracking is one of the core research directions in computer vision, its task being to continuously locate the position and scale of a specific target in a video sequence. This technology is widely used in scenarios such as autonomous driving, intelligent security, drone monitoring, and robot navigation, and is a key support for realizing dynamic environment perception and decision-making. With the development of deep learning, single-object tracking methods have evolved from traditional filtering models to deep neural network-based methods, resulting in a variety of mainstream frameworks.

[0003] Early methods, represented by correlation filtering such as MOSSE and KCF, achieved fast tracking by learning discriminative filters online. The core idea of ​​these methods was to quickly locate the target position in subsequent frames of a video sequence by learning a discriminative filter, offering high speed. However, their feature representation capabilities and robustness to deformation and occlusion were limited, resulting in inconsistent tracking performance. Subsequently, Siamese network-based methods became mainstream, such as SiamFC and SiamRPN, which achieved a balance between speed and accuracy by training a similarity matching network offline. However, these methods typically rely on the first frame template or dynamically updated templates, ignoring the continuity of target appearance changes between frames and the temporal correlation of motion trajectories. This leads to model drift in complex scenes (such as target occlusion, rapid movement, and interference from similar objects).

[0004] In recent years, the Transformer architecture has been introduced into the tracking field, such as the STARK model, which improves the ability to capture long-term temporal dependencies by modeling the target-background relationship through a self-attention mechanism. However, existing methods still have the following limitations: First, most models use single-frame or dual-frame input, failing to fully utilize the appearance and motion information of historical frames, making it difficult to cope with the challenges of drastic changes in target appearance or long-term occlusion; second, tracking strategies based on local search regions are prone to failure when the target quickly moves out of the search range, and there is a lack of effective re-detection mechanisms to ensure tracking continuity.

[0005] In addition, since the evaluation metrics for single-object tracking datasets do not penalize prediction errors in frames without targets, all current single-object trackers assume that a target exists in every frame. However, in real-world applications, targets may disappear from view, and existing trackers cannot determine whether a target exists in such cases, which greatly limits the application scenarios of the trackers. Summary of the Invention

[0006] In view of the problems existing in the prior art, the purpose of the present invention is to provide a single target tracking method and apparatus that incorporates existence judgment with multiple image inputs and historical trajectory enhancement.

[0007] This invention designs a multi-image feature extraction and fusion process, extracting and fusing features from multiple image blocks to obtain a more expressive and robust unified appearance feature. Simultaneously, it designs a historical trajectory and multi-image feature fusion process, fusing historical trajectory information and image appearance information to obtain a bounding box embedding rich in both appearance and historical trajectory information. Finally, it designs a target location and target confidence decoding process, decoding the bounding box embedding rich in appearance and historical trajectory information into target confidence and target bounding box coordinates. By integrating these three parts, this invention can quickly and accurately track targets based on multiple images and historical trajectory information, and determine whether the target is within the field of view through target confidence. It can be widely applied in scenarios such as autonomous driving, intelligent security, drone monitoring, and robot navigation.

[0008] The technical solution of the present invention is as follows: I. A single-target tracking method incorporating existence judgment with multiple image inputs and historical trajectory enhancement, the method comprising the following steps: For a single target, a single-target tracking network model is constructed that is designed for single-target tracking, has the characteristics of discriminating the existence of the target, and supports multiple image inputs. The training is carried out in two stages by inputting the consecutive frames of images with known target bounding boxes into the single-target tracking network model. Then, the trained single-target tracking network model is used to input the consecutive frames of images to be tracked, where only the target bounding box of the first frame is known and the target bounding boxes of the remaining frames are unknown, into the single-target tracking network model for tracking processing.

[0009] The single-target tracking network model includes a modified micro-sliding converter, a bounding box embedding module, a multi-layer encoder module, a multi-layer decoder module, and a multi-layer perceptron. The specific processing is as follows: S1. Initialize the historical appearance feature library and the historical bounding box library; S2. The second appearance embedding feature I2 is constructed by combining the appearance features of the current frame image with the current historical appearance feature library and template image, and then processing them with a modified tiny swin transformer and a multi-layer encoder module. S3. Simultaneously input the bounding boxes of the historical bounding box library and the template image into the bounding box embedding module for embedding processing to obtain the bounding box embedding features corresponding to each bounding box, and then concatenate them to obtain the bounding box embedding feature B. S4. Then, input the second appearance embedding feature I2 and the bounding box embedding feature B into the multi-layer decoder module for feature fusion to generate a bounding box embedding fusion feature B1 containing historical trajectory information and image feature information. S5. Finally, the bounding box is embedded in the fusion feature B1 and input into the multilayer perceptron for processing and prediction to obtain the predicted bounding box of the current frame image, thus realizing target tracking in the current frame image, and then feeding back to update the appearance feature library and the historical bounding box library. S6. Repeat steps S2 to S5 continuously to perform iterative target tracking processing.

[0010] The template image is the image of the first frame.

[0011] Step S1 specifically involves: The initial historical appearance feature library is constructed by extracting appearance features from the first frame image after modification using a tiny swintransformer, with a total of n-1 features set; that is, the appearance features of the first frame image are repeated 7 times (n=8 times) and connected together.

[0012] The initial historical bounding box library is constructed using the known bounding boxes of the first frame image as bounding box labels, with a total of n-1 bounding boxes set. That is, the bounding boxes of the first frame image are repeated 7 times (n=8 times) and connected together.

[0013] The historical appearance feature library and the historical bounding box library will be updated and iterated in the future. The appearance features in the historical appearance feature library and the bounding boxes in the historical bounding box library correspond to each other in order. That is, the i-th appearance feature in the historical appearance feature library and the i-th bounding box in the historical bounding box library both correspond to the prediction result obtained from the i-th frame image.

[0014] Step S2 specifically involves: obtaining the appearance features of the current frame image by sequentially cropping image blocks through a preset template bounding box ratio and extracting features through a modified tiny swin transformer. Then, the appearance features of the current frame image are combined in real time with the current historical appearance feature library and the appearance features of the template image to construct the first appearance embedding feature I1 of multidimensional data. The first appearance embedding feature I1 is then input into the multi-layer encoder module to perform feature fusion of multiple image features to obtain the second appearance embedding feature I2.

[0015] The modified tiny swin transformer is mainly composed of three network stages connected in sequence. Each stage downsamples the image by half the input image. Compared with the original tiny swin transformer (which has 4 network stages and downsamples four times, resulting in lower resolution), the extracted image features have higher spatial resolution and contain more fine-grained features.

[0016] In specific implementation, the appearance features of the template image, the various appearance features in the historical appearance feature library, and the appearance features of the current frame image are arranged and spliced ​​in order to form a multi-image feature as the first appearance embedding feature I1.

[0017] The multi-image features of the first appearance embedding feature I1 are fused using a multi-layer encoder module to obtain the second appearance embedding feature I2, which integrates complementary or related information from multiple images to generate a more expressive and robust unified feature representation.

[0018] The multi-layer encoder module mainly consists of six consecutively connected encoder layers. Each encoder layer includes a first multi-head attention layer, a first feedforward network, and two normalization layers. The input of the first normalization layer is used as the input of the encoder layer. The output of the first normalization layer is simultaneously used as the key K, value V, and query vector Q of the first multi-head attention layer for processing. The features output by the first multi-head attention layer and the input of the encoder layer are added together to obtain the first attention feature. The first attention feature is then passed through the second normalization layer and input to the first feedforward network. The features output by the first feedforward network and the first attention feature are added together to obtain the first feedforward feature, which is used as the output of the encoder layer.

[0019] The encoder module consists of a first multi-head attention layer and a first feedforward network. It takes feature embeddings from multiple images as input and fuses the features of different images through an attention mechanism to generate a more expressive and robust unified feature representation.

[0020] The specific topology of the first multi-head attention layer is a "split-parallel-fusion" topology. The input is first linearly mapped to generate three feature matrices Q, K, and V, and then split into multiple parallel "attention heads" along the feature dimensions. Each head independently calculates the global relevance scores of Q and K and performs a weighted summation on V. Finally, the outputs of all parallel branches are concatenated and fused into a single matrix through a linear layer, thereby realizing the interaction and extraction of global contextual information within the sequence.

[0021] The first feedforward network has a specific topology of "dimensionality increase-activation-dimensionality reduction" with position-by-position fully connected topology. It independently and identically applies a two-layer multilayer perceptron to each vector: first, it significantly increases the feature dimension through a first-layer linear mapping and then passes it through a nonlinear activation function, and then it compresses the dimension back to its original size through a second-layer linear mapping. In this way, it enhances the nonlinear representation ability of local features at each position without changing the sequence length and dimension.

[0022] Therefore, a multi-layer decoder module is used to perform feature fusion on the bounding box embedding fusion feature and the second appearance embedding feature of the image feature to generate a bounding box embedding that includes historical trajectory information and image feature information.

[0023] In this invention, the bounding box represents the target tracking frame, that is, the target area obtained by tracking, which is mainly composed of the four position coordinates of the four corners of the frame.

[0024] Specifically, a single bounding box includes cx, cy, w, and h, where cx and cy are the horizontal and vertical coordinates of the center of the bounding box, and w and h are the width and height of the bounding box. All four coordinates are normalized to [0,1], and then quantized to integers in [0,1000) before being input into the bounding box embedding module.

[0025] The bounding box embedding module consists of an embedding layer, which converts four coordinates into four coordinate embeddings and then concatenates them to obtain the bounding box embedding.

[0026] The multi-layer decoder module mainly consists of six consecutively connected decoder layers. Each decoder layer includes a masked multi-head attention layer, a second multi-head attention layer, a second feedforward network, and three normalization layers. The input of the third normalization layer is used as the input of the encoder layer. The output of the third normalization layer is simultaneously used as the key K, value V, and query vector Q of the masked multi-head attention layer for processing. The features output by the masked multi-head attention layer are added together with the input of the encoder layer to obtain the masked attention features. The masked attention features are then passed through a fourth normalization layer and input to the second multi-head attention layer. The features output by the second multi-head attention layer are added together with the masked attention features to obtain the second attention features. The second attention features are then passed through a fifth normalization layer and input to the second feedforward network. The features output by the second feedforward network and the second attention features are added together to obtain the second feedforward features, which are then used as the output of the decoder layer.

[0027] The masked multi-head attention layer forces each bounding box embedding to focus only on the bounding box embeddings before it through masked causal masking, effectively modeling historical trajectories. The entire decoder module takes feature embeddings from multiple images and multiple bounding box embeddings as input, and fuses image features and historical trajectory information through an attention mechanism.

[0028] The specific topology of the masked multi-head attention layer is a unidirectional constrained topology of "masking the future-parallel-fusion". Its basic topology is exactly the same as that of the standard multi-head self-attention layer. The only structural difference lies in the feature branch: after calculating the original dot product score of the key K and the query vector Q, and before passing through the Softmax activation function, an additional "lower triangular mask matrix" operation is chained. This topological modification forces the association score of all "future" nodes after the current position in the sequence to be set to negative infinity (making their weight probability zero), physically blocking the backward flow of information, ensuring that the decoder strictly follows the autoregressive principle that only depends on historical context, and finally performs multi-head feature concatenation and linear fusion.

[0029] The first feedforward network and the second feedforward network have the same topology. The only difference between the first multi-head attention layer and the second multi-head attention layer is that the sources of the key K and the value V are different.

[0030] The specific topology of the multilayer perceptron is a cascaded topology of "fully connected stacking - unidirectional transmission - nonlinear mapping". It consists of an input layer, a hidden layer, and an output layer connected in series. In terms of spatial topology, all nodes (neurons) between adjacent layers are densely connected pairwise, forming a weight network in the form of a completely bipartite graph. In terms of data flow, information propagates strictly unidirectionally from front to back. When passing through each layer, "linear matrix multiplication plus bias" is performed, followed by a "nonlinear activation function". Through this layer-by-layer, multi-dimensional continuous nonlinear transformation, deep feature extraction and fitting of complex input signals are achieved.

[0031] The multilayer perceptron takes the last bounding box embedding as input and outputs the bounding box information of the target in the current image patch and the target confidence. The target confidence information is used to determine whether the target exists in the current frame. The image patch bounding box information is restored to the original image coordinate system through coordinate transformation to obtain the target's position information.

[0032] The confidence level of the current feature is determined by comparing the target confidence level with the feature update threshold. If the confidence level is greater than the threshold, it means that the feature contains a lot of target information, which is helpful for subsequent tracking. The appearance feature library and historical trajectory library are then updated. Otherwise, no update is performed.

[0033] The model described in this invention effectively improves the speed and accuracy of target tracking and can determine the existence of a target. It can be widely applied in scenarios such as autonomous driving, intelligent security, drone monitoring, and robot navigation.

[0034] Step S5 specifically involves: decoding the target location and target confidence level. Multilayer perceptron is used to decode the bounding box embedding fusion features to predict the bounding box and confidence of the target in the image patch of the current frame image. Then, the bounding box coordinates of the image patch are restored to the target bounding box of the original image after coordinate transformation (box), which are used as the target bounding box and target confidence. Then, the feature library is updated via sliding: the target confidence level is compared with the preset feature update threshold. If the target confidence is greater than the feature update threshold, the target bounding box is used as the tracking and recognition result of the current frame image. The appearance features of the current frame image and the target bounding box are added as bounding box labels to the end of the appearance feature library and the historical trajectory library, respectively. The earliest historical appearance feature in the appearance feature library and the earliest bounding box in the historical trajectory library are removed for updating. If the target confidence is not greater than the feature update threshold, it is considered that there is no target in the current frame, and the historical appearance feature library and historical trajectory library are not updated.

[0035] The process of inputting consecutive frames of images with known target bounding boxes into a single-target tracking network model for training is divided into two stages: The first phase of training is as follows: The appearance features of the first frame image and the appearance features of n consecutive random frames are used to construct the first-stage historical appearance feature library. The bounding box of only the first frame image and n preset fixed filling bounding boxes are repeatedly connected to construct the first-stage historical bounding box library. The known target bounding boxes of the n consecutive random frames are used as the predicted target bounding boxes. The first-stage historical appearance feature library and the first-stage historical bounding box library are then input into the single target tracking network model to establish the first-stage loss function for training. For example, the input consists of one frame of template image and eight consecutive frames of search images containing the target. The input also includes the bounding box corresponding to the quantized template image and seven identical filled bounding boxes. The four coordinates of the filled bounding boxes are all set to 1000. The target location and target confidence decoding module embeds the eight bounding boxes into the probability distribution of the target bounding box coordinates corresponding to the eight frames of search image blocks and the cross-union ratio between the predicted bounding box and the bounding box label as the target confidence. The mathematical expectation of the probability distribution is calculated to obtain the four coordinates of the bounding box. The target confidence loss, bounding box L1 loss, and bounding box GIOU loss are constructed to form the first stage training loss and are then trained.

[0036] The second phase of training is as follows: The appearance features of the first frame image and the appearance features of the random n consecutive frames are used to construct the second-stage historical appearance feature library. The bounding box of only the first frame image and the known target bounding box of the random n consecutive frames are concatenated to construct the second-stage historical bounding box library. The known target bounding box of the random n consecutive frames is used as the predicted target bounding box. The second-stage historical appearance feature library and the second-stage historical bounding box library are then input into the single-target tracking network model to establish the second-stage loss function for training.

[0037] For example, the input consists of one frame of template image and eight consecutive frames of search images containing the target. The input also includes the bounding boxes of the quantized template image and the bounding boxes corresponding to the previous seven search images. The target location and target confidence decoding module embeds the eight bounding boxes into the probability distribution of the target bounding box coordinates corresponding to the eight search image blocks, and uses the cross-union ratio (CUI) between the predicted bounding box and the bounding box label as the target confidence. The mathematical expectation of the probability distribution is calculated to obtain the four coordinates of the bounding box. The target confidence loss, bounding box L1 loss, and bounding box GIOU loss are constructed to form the second-stage training loss and are then trained.

[0038] Losses L in the first and second stages track Established as: L track =(λ obj / n)×∑ n i=1 (C i -IOU (box) i ', box i )) 2 +(λ L1 / n)×∑ n i=1 ||box i '-box i ||1+(λ giou / n)×∑ n i=1 (1-IOU(box i ', box i )+(A(box i ', box i )-U(box i ', box i )) / A(box i ',box i )) Among them, C i The target confidence score of the i-th target bounding box obtained from the current prediction. i 'The i-th bounding box label in the historical bounding box library, box iFor the i-th target bounding box obtained from the current prediction, IOU (box) i ', box i ) indicates the bounding box i 'with enclosed box' i The intersection-union ratio, ||.||1 denotes the calculation of the L1 norm, A(box) i ', box i ) indicates the bounding box i 'with enclosed box' i The area of ​​the minimum closure region, U(box) i ', box i ) indicates the bounding box i 'with enclosed box' i The area of ​​the union of the bounding boxes; i represents the index of the predicted data dimension, n represents the total number of predicted data dimensions, i.e., the total number of bounding boxes; λ obj λ represents the target loss weight. L1 λ represents the bounding box L1 loss weights. giou This represents the bounding box GIOU loss weight; Then with a loss of L track The minimum target is used to train a single-object tracking network model to obtain the parameters of the micro-sliding converter, bounding box embedding module, multi-layer encoder module, multi-layer decoder module and multi-layer perceptron.

[0039] The target tracking described in this invention can typically be robot end-effector tracking or tracking of people in surveillance images.

[0040] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention can fuse target appearance information and target motion trajectory information through multiple image inputs and multiple bounding box inputs, making the tracking results more robust and accurate.

[0041] Secondly, the model of this invention uses a modified micro-sliding converter as the backbone network, which is two levels smaller than the traditional VIT backbone network with a base size. This ensures the tracking speed of the model and fully meets the requirements for real-time use.

[0042] Finally, the target confidence can be decoded through the target location and target confidence module, enabling the tracker to determine whether the target appears in the current frame, thus expanding the application scenarios of the tracker. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the overall process framework of the single-target tracking method that incorporates existence detection with multiple image inputs and historical trajectory enhancement, as provided in the embodiment. Figure 2 This is a structural diagram of the multilayer encoder module provided in the embodiment; Figure 3 This is a structural diagram of the multi-layer decoder module provided in the embodiment; Figure 4 This is a flowchart illustrating the first stage of training of the network model provided in the embodiment. Figure 5 This is a flowchart illustrating the second-stage training process of the network model provided in the embodiment. Figure 6 This is an example image of image block cropping provided in the embodiment; Figure 7 This is a target tracking example diagram provided in the embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0046] Currently, mainstream trackers are mainly divided into two categories based on their model input: one type uses only the first frame as a template for tracking; the other type, in addition to the first frame, uses an additional template that is dynamically updated at fixed intervals. Both approaches ignore the continuity of object appearance changes and object motion between frames, which plays a decisive role in tracking scenes with many similar objects. Based on this idea, this invention adopts a structure with multiple images and multiple bounding boxes as input, and generates the target bounding box for the next frame in an autoregressive manner. This structure is similar to machine translation tasks in natural language processing, and can effectively model the continuity between frames, achieving excellent results in tracking tasks. In addition, since current single-object tracking methods assume that each frame contains the target for bounding box prediction, their use is limited in application scenarios where the target is missing. Therefore, this invention outputs the target confidence score through a target location and target confidence decoding module, providing users with a basis for judging whether the target exists, greatly expanding the application scenarios of this invention. Ultimately, this invention can better meet the requirements of personal and industry application scenarios, providing a real-time, high-quality single-object tracking solution.

[0047] like Figure 1 As shown in the embodiment, a single-target tracking process integrating multiple image inputs with existence judgment and historical trajectory enhancement is provided. This process can store multiple image features and historical trajectory information through historical appearance feature libraries and historical trajectory libraries, fully modeling target appearance information and motion trajectory information, thereby quickly obtaining highly accurate target localization. Simultaneously, it determines target existence by outputting target confidence. Specifically, it includes the following steps: Step 1, Initialization: The template image is cropped into image blocks according to the template bounding box and then the image blocks are input into the modified micro-sliding converter to extract image features, which are used to initialize the initial image features and multiple image features in the historical appearance feature library; the template bounding box is used to initialize the initial bounding box and multiple bounding boxes in the historical trajectory library.

[0048] In the embodiment, the image patch extraction process is as follows: Figure 6 As shown, the red box represents the bounding box used to crop the image patch. The square cropping area is calculated centered on the bounding box and multiplied by 16 times its area. Figure 6 The green box is then used to scale the image patch to obtain an image patch T∈R that meets the input requirements of the modified micro-sliding converter. H×W×C Where H is the image height, W is the image width, and C is the number of image channels, typically 3. The image block T is input to the modified micro-sliding transformer to obtain the feature I∈R. H / 16×W / 16×C1 Used to initialize the initial image feature I0 and I1~I in the historical appearance feature library. n-1The same template bounding box is used to initialize the initial bounding box box0 and boxes1~box in the historical trajectory library. n-1 .

[0049] Step 2, Image Feature Extraction and Fusion: After cropping image blocks based on the target bounding box of the previous frame, the modified micro-sliding converter is used to extract image features, and then the multi-layer encoder module is used to fuse the features of multiple images.

[0050] In the embodiment, the image cropping process is as follows: Figure 6 As shown, the red box represents the target bounding box of the previous frame. A square cropping area is calculated centered on the bounding box of the previous frame, multiplied by 16 times its area. Figure 6 The green box is then used to scale the image patch to obtain an image patch S∈R that meets the input requirements of the modified micro-sliding converter. (H×W×C) The image patch S is input into the modified micro-sliding converter to obtain feature I. n ∈R H / 16×W / 16×C1 , convert I0~I n Flattening and piecing them together yields the first appearance embedding feature I1∈R [H / 16×W / 16×(n+1)]×C1 .

[0051] To achieve the fusion of features from multiple images, the first appearance embedding feature I1 is input into the encoder module, and the encoder module structure is as follows: Figure 2 As shown, the encoder module consists of six stacked encoder layers. Each encoder layer includes a multi-head attention layer and a feedforward network. Layer normalization is used to normalize the features. The first appearance embedding feature I1 is processed by the encoder module to obtain a more expressive second appearance embedding feature I2∈R. [(H / 16)×(W / 16)×(n+1)]×C1 .

[0052] Step 3, Fusion of historical trajectory and multiple image features: Input the initial bounding box and multiple bounding boxes from the historical trajectory library into the bounding box embedding module to obtain bounding box embeddings and concatenate them to obtain bounding box embedding features B. Then, use the multi-layer decoder module to perform feature fusion on the bounding box embeddings and image features to generate bounding box fusion embedding features containing historical trajectory information and image feature information.

[0053] In this embodiment, a single bounding box includes cx, cy, w, and h, where cx and cy are the x and y coordinates of the center of the bounding box, and w and h are the width and height of the bounding box. All four coordinates are normalized to [0,1], and then quantized to integers in the range [0,1000). These coordinates are then input to the bounding box embedding module, which consists of an embedding layer that converts the four coordinates into four coordinate embeddings E∈R. C1 / 4 Then, they are pieced together to form a bounding box embedded in B0~B0. n-1 ∈R C1By concatenating all the bounding box embeddings, we obtain the bounding box embedding feature B∈R. n×C1 .

[0054] To achieve the fusion of historical trajectory and multi-image features, the second appearance embedding feature I2 and the bounding box embedding feature B are input into the decoder module. The decoder module structure is as follows: Figure 3 As shown, the decoder module consists of six stacked decoder layers. Each decoder layer includes a masked multi-head attention layer, a multi-head attention layer, and a feedforward network. Layer normalization is used to normalize the features. The masked multi-head attention layer forces each bounding box embedding to focus only on the previous bounding box embedding through causal masking, which is used to learn historical trajectory information. The multi-head attention layer uses the bounding box embedding feature B as the query and the second appearance embedding feature I2 as the key and value, allowing the bounding box embedding feature B to learn appearance information from the second appearance embedding feature I2. Finally, the output bounding box fusion embedding feature B1 is obtained by projection through the feedforward network, realizing the fusion of historical trajectory and multi-image features.

[0055] Step 4, Decode target location and target confidence: Use a multilayer perceptron to decode the bounding box embedding to obtain the bounding box information of the target in the current image block and the target confidence. Then, use coordinate transformation to restore the bounding box coordinates of the image block to the target bounding box of the original image to obtain the target location and target confidence.

[0056] In this embodiment, the bounding box fusion embedded feature B1 is input into a multilayer perceptron for decoding to obtain the target confidence C. n ∈R and bounding box probability distribution box n ∈R 4000 C n The value is between 0 and 1, representing the probability that the target exists in the current frame, used to determine whether there is a target in the current frame. n This includes probability distributions for four coordinates: cx, cy, w, and h. Each probability distribution contains 1000 dimensions, and the final cx, cy, w, and h are obtained by calculating the expected value of the corresponding probability distribution.

[0057] Step 5, Feature library sliding update: Compare the target confidence with the feature update threshold. If it is greater than the threshold, add the latest image features and bounding boxes to the historical appearance feature library and the historical trajectory library respectively, and delete the earliest image features and bounding boxes in the historical appearance feature library and the historical trajectory library.

[0058] In this embodiment, to obtain the latest appearance information and motion trajectory information, it is necessary to update the historical appearance feature database and the historical trajectory database. However, during the update, it is important to determine whether the target exists. If appearance features and bounding boxes of non-existent targets are used to update the historical appearance feature database and the historical trajectory database, it will pollute the two feature databases and cause tracking drift. Therefore, only when the target confidence level C is reached... n >thresh, only then can the image features I n and the box n The historical appearance feature library and historical trajectory library are added, while the image feature I1 and bounding box box1 are removed from both. The historical appearance feature library reuses the results of previous image feature extraction, thus preserving the target appearance information while saving computational resources, effectively improving the accuracy and speed of the tracker.

[0059] The aforementioned single-object tracking method, which incorporates existence judgment with multiple image inputs and historical trajectory enhancement, utilizes a modified micro-sliding transformer, bounding box embedding module, encoder module, decoder module, and multilayer perceptron. These components undergo two-stage training to optimize model parameters before application. The first-stage image cropping, the two-stage appearance feature extraction, and the first-stage bounding box embedding differ from those used during inference. The first-stage training image cropping uses a randomly scaled bounding box of the current frame. The two-stage appearance feature extraction requires extracting multiple image features simultaneously (while inference, only the latest image frame's features are extracted each time). The first-stage bounding box embedding uses padding to replace the subsequent n-1 bounding boxes (box1~box). n-1 .

[0060] like Figure 4 As shown, the purpose of the first stage of training is to make the model focus on learning the appearance features of the target being modeled. Too much input box information will cause the model to rely on box information, resulting in the learned visual features not reflecting the appearance features of the target being tracked. Therefore, padding is used to replace the last n-1 bounding boxes box1~box n-1 The values ​​cx, cy, w, and h of the bounding box are all set to a fixed value of 1000; when cropping image blocks, the bounding box annotations of the current image are first randomly scaled, and then... Figure 6 The first step involves cropping image patches. This step aims to increase the diversity of image patches, enabling the model to learn more robust appearance features and improve its generalization ability. During the first training phase, the modified micro-sliding transformer extracts features from all image patches for subsequent feature fusion. After processing by the modified micro-sliding transformer, encoder module, bounding box embedding module, decoder module, and multilayer perceptron, the image patches and bounding boxes yield the target confidence scores C1~C1. nAnd the enclosed boxes box1~box n The target confidence loss, bounding box L1 loss, and bounding box GIOU loss are constructed to form the first-stage training loss and are trained. The target confidence loss is used to provide supervision of target existence, while the bounding box L1 loss and bounding box GIOU loss are used to provide supervision of bounding box positioning accuracy.

[0061] Specifically, the loss in the first training phase L track1 Represented as: L track1 =(λ obj / n)×∑ n i=1 (C i -IOU (box) i ', box i )) 2 +(λ L1 / n)×∑ n i=1 ||box i '-box i ||1+(λ giou / n)×∑ n i=1 [1-IOU(box i ', box i )+(A(box i ', box i )-U(box i ', box i )) / A(box i ',box i )] Where C i The target confidence score output by the model, box i 'For the bounding box label, box i The bounding box predicted by the model, IOU (box) i ', box i ) indicates box i 'with box i The intersection-union ratio, ||.||1 denotes the calculation of the L1 norm, A(box) i ', box i ) indicates box i 'with box i The area of ​​the minimum closure region, U(box) i ', box i ) indicates box i 'with box i The area of ​​the union of the sets.

[0062] like Figure 5 As shown, the purpose of the second stage of training is to incorporate historical trajectory information of the target, allowing the model to learn this historical trajectory information and to fuse it with appearance information. The bounding box embedding in the second stage of training uses input bounding boxes box0~box0. n-1 The bounding box label corresponding to the current image is used, and when cropping image blocks, in addition to the template image block T, other image blocks S1~S2 are also used. n The bounding box tag corresponding to the previous frame image is used, as shown in the details below. Figure 6 As shown, box1~box n-1 The offset relative to the image center implies the target's motion information relative to the previous frame. During the second training phase, the modified micro-sliding transformer needs to extract features from all image patches for subsequent feature fusion. The image patches and bounding boxes are processed by the modified micro-sliding transformer, encoder module, bounding box embedding module, decoder module, and multilayer perceptron to obtain the target confidence scores C1~C1. n And the enclosed boxes box1~box n The target confidence loss, bounding box L1 loss, and bounding box GIOU loss are constructed to form the second-stage training loss and are trained. The target confidence loss is used to provide supervision of target existence, while the bounding box L1 loss and bounding box GIOU loss are used to provide supervision of bounding box positioning accuracy.

[0063] Specifically, the loss in the second training phase L track2 Represented as: L track2 =(λ obj / n)×∑ n i=1 (C i -IOU (box) i ', box i )) 2 +(λ L1 / n)×∑ n i=1 ||box i '-box i ||1+(λ giou / n)×∑ n i=1 [1-IOU(box i ', box i )+(A(box i ', box i )-U(box i ', box i )) / A(box i ',box i )] Where C i The target confidence score output by the model, box i 'For the bounding box label, box i The bounding box predicted by the model, IOU (box) i ', box i ) indicates box i 'with box i The intersection-union ratio, ||.||1 denotes the calculation of the L1 norm, A(box) i ', box i ) indicates box i 'with box i The area of ​​the minimum closure region, U(box) i ', box i ) indicates box i 'with box i The area of ​​the union of the predicted data dimensions, where i represents the index of the predicted data dimension and n represents the total number of predicted data dimensions; λ obj λ represents the target loss weight. L1 λ represents the L1 loss weights. giou This represents the GIOU loss weight.

[0064] After the above two stages of training, one will be able to pass Figure 1 The flowchart shown takes an image patch and historical bounding boxes as input and outputs the target confidence score and the target bounding box. Figure 7 The final tracking results are displayed, showing that the tracking is accurate and can handle cases where the target does not exist.

[0065] Based on the same inventive concept, the embodiment also provides a single-target tracking device that incorporates existence judgment for multi-image input and historical trajectory enhancement, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned single-target tracking method that incorporates existence judgment for multi-image input and historical trajectory enhancement, specifically including the following steps: Step 1, Initialization: The template image is cropped into image blocks according to the template bounding box and then the image blocks are input into the modified micro-sliding converter to extract image features, which are used to initialize the initial image features and multiple image features in the historical appearance feature library; the template bounding box is used to initialize the initial bounding box and multiple bounding boxes in the historical trajectory library. Step 2, Image Feature Extraction and Fusion: After cropping an image block based on the target bounding box of the previous frame, the modified micro-sliding converter is used to extract image features. Then, the multi-layer encoder module is used to fuse the features of multiple images. Specifically, the image block is cropped according to the bounding box of the previous frame. The image block is then input into the modified micro-sliding converter module to extract image features. These features are then concatenated with the image features stored in the historical appearance feature library and the initial image features to obtain the first appearance embedding feature I1. The multi-layer encoder module is then used to fuse the first appearance embedding feature I1 to integrate complementary or related information from multiple images and generate a more expressive and robust unified feature representation, the second appearance embedding feature I2. Step 3, historical trajectory and multi-image feature fusion: Input the initial bounding box and multiple bounding boxes in the historical trajectory library into the bounding box embedding module to obtain bounding box embeddings and concatenate them to obtain bounding box embedding feature B. Then, use the multi-layer decoder module to perform feature fusion on the bounding box embedding feature B and the second appearance embedding feature I2 to generate bounding box embedding fusion feature B1 containing historical trajectory information and image feature information. Step 4, Decode target location and target confidence: Use a multilayer perceptron to decode the bounding box embedding fusion feature B1 to obtain the bounding box information of the target in the current image block and the target confidence. Then, use coordinate transformation to restore the bounding box coordinates of the image block to the target bounding box of the original image to obtain the target location and target confidence. Step 5, Feature library sliding update: Compare the target confidence with the feature update threshold. If it is greater than the threshold, add the latest image features and bounding boxes to the historical appearance feature library and the historical trajectory library respectively, and delete the earliest image features and bounding boxes in the historical appearance feature library and the historical trajectory library.

[0066] The single-target tracking device with multi-image input and historical trajectory enhancement that incorporates existence judgment provided in this embodiment is a hardware device. At the hardware level, in addition to a processor and memory, it also includes an internal bus, network interface, memory, and other hardware required for the operation. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the single-target tracking method with multi-image input and historical trajectory enhancement that incorporates existence judgment in steps 1-5 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0067] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described single-target tracking method incorporating existence judgment and historical trajectory enhancement, specifically including the following steps: Step 1, Initialization: The template image is cropped into image blocks according to the template bounding box and then the image blocks are input into the modified micro-sliding converter to extract image features, which are used to initialize the initial image features and multiple image features in the historical appearance feature library; the template bounding box is used to initialize the initial bounding box and multiple bounding boxes in the historical trajectory library. Step 2, Image Feature Extraction and Fusion: After cropping an image block based on the target bounding box of the previous frame, the modified micro-sliding converter is used to extract image features. Then, the multi-layer encoder module is used to fuse the features of multiple images. Specifically, the image block is cropped according to the bounding box of the previous frame. The image block is then input into the modified micro-sliding converter module to extract image features. These features are then concatenated with the image features stored in the historical appearance feature library and the initial image features to obtain the first appearance embedding feature I1. The multi-layer encoder module is then used to fuse the first appearance embedding feature I1 to integrate complementary or related information from multiple images and generate a more expressive and robust unified feature representation, the second appearance embedding feature I2. Step 3, historical trajectory and multi-image feature fusion: Input the initial bounding box and multiple bounding boxes in the historical trajectory library into the bounding box embedding module to obtain bounding box embeddings and concatenate them to obtain bounding box embedding feature B. Then, use the multi-layer decoder module to perform feature fusion on the bounding box embedding feature B and the second appearance embedding feature I2 to generate bounding box embedding fusion feature B1 containing historical trajectory information and image feature information. Step 4, Decode target location and target confidence: Use a multilayer perceptron to decode the bounding box embedding fusion feature B1 to obtain the bounding box information of the target in the current image block and the target confidence. Then, use coordinate transformation to restore the bounding box coordinates of the image block to the target bounding box of the original image to obtain the target location and target confidence. Step 5, Feature library sliding update: Compare the target confidence with the feature update threshold. If it is greater than the threshold, add the latest image features and bounding boxes to the historical appearance feature library and the historical trajectory library respectively, and delete the earliest image features and bounding boxes in the historical appearance feature library and the historical trajectory library.

[0068] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0069] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A single-target tracking method incorporating existence judgment with multiple image inputs and historical trajectory enhancement, characterized in that, The method includes the following steps: A single-target tracking network model is constructed, featuring target presence detection and multi-image input capabilities. The model is trained in two phases by inputting consecutive frames of images with known target bounding boxes into the single-target tracking network model, which includes a modified micro-sliding transformer, a bounding box embedding module, a multi-layer encoder module, a multi-layer decoder module, and a multi-layer perceptron. Then, the trained single-target tracking network model is used to input consecutive frames of images to be tracked, where only the first frame has a known target bounding box and the remaining frames have unknown target bounding boxes, for tracking processing.

2. The single-target tracking method incorporating existence judgment with multi-image input and historical trajectory enhancement according to claim 1, characterized in that: The single-target tracking network model specifically processes the following: S1. Initialize the historical appearance feature library and the historical bounding box library; S2. The second appearance embedding feature I2 is constructed by combining the appearance features of the current frame image with the current historical appearance feature library and template image, and then processing them with the modified micro-sliding converter and multi-layer encoder module. S3. Simultaneously input the bounding boxes of the historical bounding box library and the template image into the bounding box embedding module to obtain the bounding box embedding features corresponding to each bounding box, and then concatenate them to obtain the bounding box embedding feature B. S4. Then, input the second appearance embedding feature I2 and the bounding box embedding feature B into the multi-layer decoder module for feature fusion to generate a bounding box embedding fusion feature B1 containing historical trajectory information and image feature information. S5. Finally, the bounding box is embedded into the fusion feature B1 and input into the multilayer perceptron to obtain the predicted bounding box of the current frame image, and then fed back to update the appearance feature library and the historical bounding box library. S6. Repeat steps S2 to S5 continuously to perform iterative target tracking processing.

3. The single-target tracking method incorporating existence judgment with multi-image input and historical trajectory enhancement according to claim 2, characterized in that: The specific steps of step S1 are as follows: the initial historical appearance feature library is constructed by extracting appearance features from the modified image of the first frame using a micro-sliding converter, and a total of n-1 features are set; the initial historical bounding box library is constructed by using the bounding boxes of the image of the first frame as bounding box labels, and a total of n-1 bounding boxes are set.

4. The single-target tracking method incorporating existence judgment with multi-image input and historical trajectory enhancement according to claim 2, characterized in that: The specific steps of step S2 are as follows: the current frame image is processed by a micro-sliding converter after being cropped and modified by a preset template bounding box ratio to obtain the appearance features of the current frame image. Then, the appearance features of the current historical appearance feature library and the template image are combined in real time to construct the first appearance embedding feature I1. The first appearance embedding feature I1 is input into the multi-layer encoder module for feature fusion to obtain the second appearance embedding feature I2.

5. A single-target tracking method incorporating existence judgment with multi-image input and historical trajectory enhancement according to claim 2 or 4, characterized in that: The multi-layer encoder module mainly consists of six consecutively connected encoder layers. In each encoder layer, the input of the first normalization layer is used as the input of the encoder layer. The output of the first normalization layer is simultaneously used as the key K, value V, and query vector Q of the first multi-head attention layer for processing. The features output by the first multi-head attention layer and the input of the encoder layer are added together to obtain the first attention feature. The first attention feature is then passed through the second normalization layer and input into the first feedforward network. The output features of the first feedforward network and the first attention feature are added together to obtain the output of the encoder layer.

6. The single-target tracking method incorporating existence judgment with multi-image input and historical trajectory enhancement according to claim 2, characterized in that: The multi-layer decoder module mainly consists of six consecutive decoder layers. In each decoder layer, the input of the third normalization layer is used as the input of the encoder layer. The output of the third normalization layer is simultaneously used as the key K, value V, and query vector Q of the mask multi-head attention layer for processing. The features output by the mask multi-head attention layer are added together with the input of the encoder layer to obtain the mask attention features. The attention features are input into the second multi-head attention layer after passing through the fourth normalization layer. The features output by the second multi-head attention layer and the mask attention features are added together to obtain the second attention features. The second attention features are then input into the second feedforward network after passing through the fifth normalization layer. The features output by the second feedforward network and the second attention features are added together to obtain the output of the decoder layer.

7. A single-target tracking method incorporating existence judgment with multi-image input and historical trajectory enhancement according to claim 2, characterized in that: Step S5 specifically involves: using a multilayer perceptron to decode the bounding box embedding fusion features, predicting the bounding box and confidence score of the target in the current frame image, which are then used as the target bounding box and target confidence score; and finally comparing the target confidence score with a preset feature update threshold. If the target confidence is greater than the feature update threshold, the target bounding box is used as the result of the current frame. The appearance features and target bounding box of the current frame are added to the end of the appearance feature library and the historical trajectory library, respectively, and the earliest historical appearance features and bounding boxes are removed for updating. Otherwise, it is assumed that there is no target in the current frame, and the historical appearance feature library and the historical trajectory library are not updated.

8. The single-target tracking method incorporating existence judgment and historical trajectory enhancement according to claim 1, characterized in that: The process of inputting consecutive frames of images with known target bounding boxes into the single-target tracking network model is divided into two stages for training. Specifically, the appearance features of the first frame and the appearance features of random consecutive n frames are used to construct the current stage historical appearance feature library. The bounding boxes of the first frame are used to construct the current stage historical bounding box library. The known target bounding boxes of random consecutive n frames are used as the predicted target bounding boxes. The current stage historical appearance feature library and historical bounding box library are then input into the single-target tracking network model to establish the current stage loss function for training. The difference between the two stages is that the first stage involves repeatedly connecting the bounding box of only the first frame with n pre-set fixed filling bounding boxes to build the first stage historical bounding box library, while the second stage involves connecting the bounding box of only the first frame with the known target bounding boxes of n random consecutive frames to build the second stage historical bounding box library.

9. The single-target tracking method with multiple image inputs incorporating existence judgment and historical trajectory enhancement according to claim 8, characterized in that, Loss L in the first and second stages track Established as: L track =(λ obj / n)×∑ n i=1 (C i -IOU(box i ', box i )) 2 +(λ L1 / n)×∑ n i=1 ||box i '-box i ||1+(λ giou / n)×∑ n i=1 (1-IOU(box i ', box i )+(A(box i ', box i )-U(box i ', box i )) / A(box i ',box i )) Among them, C i The target confidence score of the i-th target bounding box obtained from the current prediction. i 'The tag for the i-th historical bounding box in the historical bounding box library, box i For the i-th target bounding box obtained from the current prediction, IOU (box) i ', box i ) indicates the history box i 'with target bounding box' i The intersection-union ratio, ||.||1 denotes the calculation of the L1 norm, A(box) i ', box i ) indicates the history box i 'with target bounding box' i The area of ​​the minimum closure region, U(box) i ', box i ) indicates the history box i 'with target bounding box' i The area of ​​the union of the two data dimensions; i represents the index of the predicted data dimension, and n represents the total number of predicted data dimensions; λ obj λ represents the target loss weight. L1 λ represents the bounding box L1 loss weights. giou This represents the bounding box GIOU loss weight; Then with a loss of L track The single-object tracking network model is trained with the minimum target as the objective to obtain the parameters of the micro-sliding converter, bounding box embedding module, multi-layer encoder module, multi-layer decoder module, and multi-layer perceptron.

10. A single-target tracking device integrating multiple image inputs with existence judgment and historical trajectory enhancement, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the method of any one of claims 1-9.

Images