Target tracking method and system of prediction model device

By introducing Transformer model and position and bounding box coding in the target tracking method, combining global information inference and linear layer generation weights, the problem of overfitting and insufficient feature utilization of discriminant models is solved, and more efficient and accurate target tracking is achieved.

CN118968128AInactive Publication Date: 2024-11-15WUHAN TEXTILE UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410967901.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The target tracking method based on the discriminant model has overfitting problems, resulting in a degradation of prediction performance. The target state information in the training frame is ignored during the test phase, and the feature representation is not effectively enhanced by the target feature.

Method used

The Transformer model is used to combine position information encoding and bounding box information encoding, and global information inference is performed through feature extraction network and Transformer encoder to generate enhanced features, and a linear layer is used to generate border regression and target classification weights to improve the accuracy of target positioning and classification. .

Benefits of technology

It significantly improves the accuracy and robustness of target tracking, can predict target positions and bounding boxes more accurately, enhances the model's migration and real-time performance, and is suitable for complex backgrounds and multi-scale target processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968128A_ABST
    Figure CN118968128A_ABST
Patent Text Reader

Abstract

The invention belongs to but not limited to the technical field of target tracking, and discloses a target tracking method and system for a predictive modeler, and the method comprises the steps: obtaining an image in a video sequence, and inputting the image into a feature extraction network for feature extraction; bounding box information codes and position information codes are added to target information in the training frames, codes are also added to the test frames, and feature mixing is carried out on the test frames and the depth images; combining the features from the test branch and the training branch, inputting the combined features into a Transform encoder, and generating enhanced features through global information reasoning between a training frame and a test frame; the Transform decoder is used for predicting the weight of the target model through the output of the Transform encoder; and the obtained information passes through a linear layer to obtain a frame regression weight and a target classification weight, and the obtained weights are combined with the extracted test features to carry out bounding box regression and realize target classification. Compared with a traditional optimized model prediction method, the method has the advantage that the position and the bounding box of the target are predicted more accurately by expanding the model predictor.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of target tracking technology, and in particular relates to a target tracking method and system for a prediction modeler. Background Art

[0002] Object tracking is an important task in the field of computer vision video processing. Its purpose is to continuously identify and locate objects in video sequences. Object tracking has a wide range of application scenarios, such as monitoring systems, autonomous driving, intelligent transportation, and many other fields. Existing object tracking methods are mainly divided into two categories: generative models and discriminative models. The generative model method constructs an appearance model of the target and searches for the area in the video sequence that best matches the model for tracking. The discriminative model trains a classifier to distinguish between the target and the background, thereby achieving object tracking.

[0003] Traditional tracking based on discriminative model prediction uses the target model to classify the target object in the test frame, and uses the training frame to obtain the weight of the target model from the model, where the prediction of the discriminative tracker is very important so that the subsequent target model can accurately predict. However, there is a problem with the traditional tracking method based on the discriminative model, that is, the performance of the prediction model is degraded due to the overfitting of the training samples by the discriminative process. In addition, the discriminative model only uses the features extracted from the test frame to identify and locate the target in the test phase, ignoring the target state information contained in the training frame, so that the model does not effectively use the target features to enhance the feature representation in the test phase.

[0004] The Chinese patent with publication number CN117974722A discloses a "system and method for single target tracking based on improved attention mechanism and Transformer", which is characterized in that the extracted features are passed through the feature integration module to obtain global and local integrated feature information. Then, the spatiotemporal feature enhancement module in the multi-head attention mechanism of the improved Transformer is used to capture the spatiotemporal context features inside the image.

[0005] Combining the above two ideas, this method integrates the idea of ​​discriminative model and combines Transformer. This method proposes a target tracking method based on Transformer model prediction module. Compared with the former feature integration module, Transformer can better capture the global context information of the target, but can use the target information to dynamically build a more discriminative feature space for each frame, thereby improving the accuracy and robustness of tracking. And further expand the model predictor, add two encodings to estimate the second set of weights for precise bounding box regression, better learn the detailed features and location information of the target, and more accurately predict the location and bounding box of the target. And the architecture of the tracker of this method depends on training and testing information, and can directly extract features and weights from the input video image frame, making the model transferable and practical.

[0006] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are: there is a problem with the tracking method based on the discriminant model, that is, the performance of the prediction model is reduced due to the overfitting of the training samples in the discriminant process. In addition, the discriminant model only uses the features extracted from the test frame to identify and locate the target in the test phase, ignoring the target state information contained in the training frame, so that the model does not effectively use the target features to enhance the feature representation in the test phase. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a target tracking method and system of a prediction model. Transformer can better capture the global context information of the target, thereby improving the accuracy and robustness of tracking, and at the same time add position information encoding and bounding box information encoding enhancement, so that the model can more accurately regress the bounding box and better learn the detailed features of the target. At the same time, the entire prediction module is transferable.

[0008] The present invention is implemented as follows: a target tracking method of a prediction modeler, comprising:

[0009] S1, obtain three frames of images in the video sequence, two of which are used as training images and one as a test image, and input them into the feature extraction network for feature extraction;

[0010] S2, add bounding box information encoding and insertion position information encoding to the Gaussian map centered on the target information in the training frame, add encoding to the test frame as well, and perform feature mixing with the depth image of the test frame;

[0011] S3, combines the features from the test and training branches and inputs them into the Transformer encoder, generating enhanced features by reasoning about the global information between the test frame and the training frame;

[0012] S4, Transformer decoder predicts the target model weights through the output of Transformer encoder;

[0013] S5, the predicted target model weight obtained by the prediction model is passed through a linear layer to obtain a bounding box regression weight and a target classification weight.

[0014] S6 receives the bounding box regression weight and target classification weight from the prediction module, as well as the test features extracted by the Transformer, performs bounding box regression on the test features, and implements target classification.

[0015] Further, step S1 includes the following sub-steps:

[0016] (1.1) The entire target tracking framework of this method includes a test branch and a training branch.

[0017] (1.2) Obtain three different frames of images in the video sequence, two of which are used as training images and one as a test image.

[0018] (1.3) Use ResNet50 as the feature extraction module, where ResNet50 has two basic blocks: Convolutional Block and IdentityBlock;

[0019] The typical structure of Convolutional Block is to reduce the number of channels through 1×1 convolution layer to reduce the computational complexity; then use 3×3 convolution layer to learn and extract features, and finally use 1×1 convolution layer to keep the dimension of feature map consistent;

[0020] The Identity Block consists of three convolutional layers, of which the first and third are 1×1 convolutional layers. The first convolutional layer is used to reduce or restore the number of channels, and the third is used to restore the number of channels of the feature map to keep the dimension consistent. The second layer is a 3x3 convolutional layer, which is used to learn and extract features. The input and output of the Identity Block have the same dimension, and the input is directly added to the output through a jump connection, which retains the information of the original input.

[0021] ResNet50 builds the entire network by stacking Convolutional Block and Identity Block, thereby alleviating the gradient vanishing problem in deep networks. This allows the network to be deeper. The residual connection allows the network to learn more effective feature representations and the training is more stable.

[0022] Further, step S2 includes the following sub-steps:

[0023] (2.1) Add a target position encoding to the training frame, which allows the model predictor to add the target state information in the training frame when predicting the target model. The specific operation is as follows: Set a learnable Token Embedding to represent the foreground so that the model predictor can incorporate the target state information from the training frame. Combine it with the Gaussian graph y centered at the target position i Point product is used as position information encoding. The specific target encoding function formula is as follows:

[0024] ψ(y i ,R fg )=y i ·R fg

[0025] Among them, ψ(y i ,R fg ) represents the target encoding function, and the features are obtained by encoding the target position. is a learnable embedding Token Embedding representation of the prospect, where · represents the dot product, represents a Gaussian map centered at the location of the training object in the training image.

[0026] Then superimpose the target code with the original feature. The specific superposition method is as follows:

[0027] v i =x i +ψ(y i ,R fg )

[0028] The target code ψ(y i ,R fg ) and the features obtained by the feature extraction module of training image 1 and training image 2 and Add together, and the training features obtained are Contains the encoded target state information, where H and W are the height and width of the input training image, and C is the number of channels.

[0029] (2.2) For symmetry, a test code is added to identify the features corresponding to the test frame.

[0030] v test =x test +μ(R test )

[0031] Among them, x test is the test feature obtained by the feature extraction module of the test image, μ(R test ) is x testRepeating token R for each block test , because the specific target position is unknown in the test frame, TokenR test Superimposed on each patch position.

[0032] (2.3) Add another encoding to merge the information about the target bounding box, calculate the distance between each point on the training feature map and the four edges of the target box, and obtain a densely distributed target box, where the training frame represents the bounding box containing the target object in training frame i, and represents the position and size of the target object in training frame i, where and represents the coordinates of the center of the bounding box, and Represents the width and height of the bounding box, and the feature map x i Each position (j x ,j y ) coordinates are mapped to the coordinates in the image domain (k x ,k y ). The calculation formula is as follows:

[0033]

[0034] Where s is the stride of the feature extraction network. Calculate each remapped position to the left of the bounding box l i , the right side of the bounding box r i , the upper edge of the bounding box i , the bounding box bottom b i The normalized distance of is used to obtain the bounding box. The formula is as follows:

[0035]

[0036] Among them, H im =s·H,W im =s·W, corresponding to the spatial dimension of the image block and the depth feature x used to extract i The stride of the backbone network is d i =(l i ,t i ,r i ,b i ),and

[0037] (2.4) Use MLP multi-layer perceptron to encode the bounding box and change the dimension from 4 to C so that the subsequent fusion feature dimensions are consistent. After the dimension is increased, the training image features, test image features, and their target position encodings are superimposed together. The specific formula is as follows:

[0038]

[0039] Among them, ψ(y i ,R fg ) is the feature obtained by encoding the target position in the above step (2.1), x i is the training image frame feature obtained through the feature extraction network, is the target box d i Using Multilayer Perceptron By combining the three together, the model can better capture the spatial and contextual information of the target, while also obtaining the target state information of the training image frame to enhance the features.

[0040] Further, step S3 includes the following sub-steps:

[0041] (3.1) will get training features containing multiple codes Contains coded test features Reshape these superimposed features into And expand all m training features v along the first dimension i and test feature v test Connect them together. It turns it into a two-dimensional sequence and processes it through the Transformer encoder, where the sequence length is H·W and C is the feature dimension.

[0042] [t1,…,t m ,t test ]=T enc ([v1,…,v m ,v test ])

[0043] The Transformer encoder consists of a multi-head self-attention module, which enables it to perform global reasoning across the entire frame or even multiple training and test frames. After this encoder, the target state can be identified as foreground and background regions respectively, and the Transformer can distinguish between the two regions.

[0044] Further, step S4 includes the following sub-steps:

[0045] (4.1) The Transformer decoder receives the t of the Transformer encoder. i and t test , which is obtained by combining the characteristics of the entire training sample and the test sample. The details are as follows:

[0046] w=T dec ([t1,…,t m ,t test ],R fg )

[0047] Among them, input t1,…,t m ,t test It is obtained by joint reasoning of training samples and test samples, and uses the same learnable foreground embedding R fg The query is fed into the Transformer decoder, which enables the resulting weights to make the prediction model more discriminative.

[0048] Further, step S5 includes the following sub-steps:

[0049] (5.1) The weight w output by the Transformer decoder passes through a linear layer to obtain the weight ω for bounding box regression bbreg and target classification ω cls .

[0050] (5.2) Weight ω cls The predicted target model is directly used for subsequent target classification, and the target model classifies the target by applying these weights to the test features.

[0051] (5.3) Weight ω for bounding box regression bbreg Conditioning the test features enables the network to regress the bounding box based on the target state of the current frame.

[0052] Further, step S6 includes the following sub-steps:

[0053] (6.1) In order to make the encoder output the test feature t test It is target perception. An attention map is obtained through self-attention calculation. The test features are weighted using the attention map. The weighted feature expression is as follows:

[0054] Z weighted =Attention(t test *ω bbreg )⊙t test

[0055] Where ⊙ represents element-by-element multiplication, The weighted features are then input into the convolutional neural network (CNN), which finally uses an exponential activation function to produce normalized bounding box predictions under the same ltrb representation. The bounding box of the target is regressed at each position in the feature map, and the boundary of the target relative to that position is represented by four distances (left, top, right, and bottom). By combining the feature extraction of the Transformer encoder and the anchor-free bounding box regression method, efficient bounding box regression is achieved by performing convolution operations directly on the feature map, while utilizing the global context information of the Transformer to improve detection accuracy.

[0056] (6.2) Use the Transformer decoder to output the weight ω generated by the linear layer cls And the test feature map output by the encoder, analogous to the DCF discriminant model, by learning a filter, and obtaining the target classification score h(w,t test ). The specific formula is as follows:

[0057] h(w,t test )=w*t test

[0058] Where w is the weight of the filter output by the pre-decoder, t test is the test feature after the training feature enhancement in the encoder, * is the convolution operation, which combines the output features of the Transformer decoder and encoder, generates classification weights using a linear layer, and obtains the target classification score through similarity calculation. This method combines the global context understanding ability of the Transformer with the efficiency of the convolution operation to achieve effective target classification.

[0059] Another object of the present invention is to provide a target tracking system of a prediction modeler for implementing the target tracking method of the prediction modeler, comprising:

[0060] A training image acquisition module is used to acquire three frames of images in a video sequence, two of which are used as training images and one as a test image, and input them into a feature extraction network for feature extraction;

[0061] A feature mixing module is used to add bounding box information encoding and insertion position information encoding to the Gaussian map centered on the target information in the training frame, add encoding to the test frame, and perform feature mixing with the test frame depth image;

[0062] The enhanced feature acquisition module is used to combine the features from the test and training branches and input them into the Transformer encoder to generate enhanced features by inferring the global information between the test frame and the training frame;

[0063] A target model weight prediction output module is used to predict the target model weight using the output of the Transformer encoder using the Transformer decoder;

[0064] The border regression weight and target classification weight acquisition module is used to obtain the border regression weight and target classification weight by passing the predicted target model weight obtained by the prediction model through a linear layer.

[0065] The target classification module is used to receive the bounding box regression weights and target classification weights from the prediction module, as well as the test features extracted by the Transformer, perform bounding box regression on the test features, and realize target classification.

[0066] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the target tracking method of the prediction modeler.

[0067] Another object of the present invention is to provide an information data processing terminal, which includes the target tracking system of the prediction modeler.

[0068] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0069] First, compared with traditional optimized model prediction methods, the model predictor is extended to more accurately predict the location and bounding box of the target. Transformer can capture global relationships with smaller inductive biases, enabling it to learn more powerful target model predictions. The model predictor is further extended to add location information encoding and target box encoding, which can better learn the fine-grained features and location information of the target.

[0070] Second, compared with the traditional optimized model predictor method, the present invention introduces target position coding and target bounding box coding, and the model can more accurately capture the fine-grained features and position information of the target. Target position coding integrates the target state information in the training frame into the feature representation, so that the model can use this coded information to enhance the feature representation of the target during prediction. The target bounding box coding calculates the distance from each position to the bounding box, generates densely distributed target box information, and encodes it through a multi-layer perceptron to make the feature dimensions consistent. At the same time, test information coding is also added to the test frame, so that the model can not only better capture the spatial and contextual information of the target when performing feature fusion, but also improve the accuracy and discrimination of the feature representation, thereby significantly improving the accuracy and robustness of target tracking.

[0071] The model predictor based on the Transformer model proposed in the present invention directly predicts the target model from the data through an end-to-end method at the same time, which can not only distinguish the features of the target from the background in advance, but also pay attention to the learning target feature information. In addition, the prediction model proposed in the present invention can not only use the feature information of the past training frames but also use the features of the current test frame, and also add the test features to the feature learning. The prediction model proposed in the present invention can use the information of the current test frame to build a more discriminative feature space for the current frame.

[0072] Third, the technical solution of the present invention fills the technical gap in the industry at home and abroad: the tracking framework of the new prediction model proposed in the present invention is based on the idea of ​​the DCF discriminant model, which integrates the target information and background information of the tracked target in the previous frame, and provides effective global reasoning when learning the model, but the DCF discriminant model has serious inductive bias on the prediction model. The present invention bridges the gap between the traditional DCF discriminant model-based method and the Transformer-based tracker. The target is located through a compact target model, and the weights obtained by the Transformer-based model predictor are used to enhance the learning ability of the target model. By introducing two new target state encodings, the Transformer can effectively utilize the target position and range information, and further expand the model predictor to generate bounding box regression weights. The contributions of the present invention include proposing a transformer-based model prediction module, extending the bounding box regression weight estimation, and developing new encoding methods and tracking processes, which bring new technical progress and solutions to the field of target detection and tracking.

[0073] Fourth, the technical solution of the present invention solves the technical problems of the prior art.

[0074] 1) Target tracking in complex backgrounds:

[0075] Existing problem: In complex backgrounds, traditional target tracking methods are easily disturbed by the environment, resulting in inaccurate target positioning.

[0076] Solution: By introducing the Transformer encoder and decoder, the global context information is used to perform global reasoning on the target location, which significantly improves the accuracy of target positioning.

[0077] 2) Multi-scale object processing:

[0078] Existing problems: Traditional methods have difficulty in handling targets of different scales, especially the detection and tracking effects of small targets are poor.

[0079] Solution: Encode the bounding box through a multi-layer perceptron (MLP) and combine it with a convolutional neural network (CNN) for bounding box regression, which can handle multi-scale targets, especially for small targets. It has better detection and tracking effects.

[0080] 3) Real-time and efficiency:

[0081] Existing problems: Traditional target tracking algorithms have bottlenecks in real-time and computational efficiency, making it difficult to meet actual application requirements.

[0082] Solution: By using ResNet50 for feature extraction and combining the efficient global information processing capabilities of Transformer, efficient real-time target tracking is achieved.

[0083] 4) Target classification accuracy:

[0084] Existing Problem: The accuracy of object classification is not high, especially when the object appearance varies greatly.

[0085] Solution: By combining the features of the Transformer decoder and encoder outputs, using a linear layer to generate classification weights, and obtaining the target classification score through similarity calculation, the accuracy of target classification is significantly improved.

[0086] Notable Technology

[0087] 1) Integrating deep learning and Transformer technology:

[0088] This invention combines deep learning with Transformer technology for the first time, and performs global context information reasoning through Transformer, which significantly improves the accuracy and robustness of target tracking.

[0089] 2) Anchor-free bounding box regression:

[0090] The anchor-free bounding box regression method is adopted. Through convolution operation and attention mechanism, convolution operation is performed directly on the feature map, which improves the efficiency and accuracy of bounding box regression.

[0091] 3) Multi-level feature fusion:

[0092] By multi-level fusion of target position encoding, bounding box information encoding and features of feature extraction network, the model can better capture the spatial and contextual information of the target and improve the performance of target tracking.

[0093] 4) Real-time enhancement:

[0094] By adopting the efficient ResNet50 feature extraction network and Transformer global information processing capabilities, this method has high real-time performance while maintaining high accuracy, meeting the needs of practical applications.

[0095] 5) Wide applicability:

[0096] This method is not only suitable for scenarios with high real-time requirements such as autonomous driving and drones, but can also be widely used in security monitoring, intelligent transportation, industrial inspection and other fields, and has broad application prospects.

[0097] In summary, the present invention effectively solves many problems in the prior art through innovative technical solutions, significantly improves the accuracy, real-time and robustness of target tracking, and has important technological progress and application value.

[0098] Fifth, the technical problems of the prior art solved by the parameters, algorithms, mathematical models, etc. of the technical solution of the present invention

[0099] 1. Parameter setting and feature extraction

[0100] Current issues:

[0101] Traditional target tracking algorithms are often affected by the complexity of the input image during feature extraction, resulting in insufficient feature expression. Especially when dealing with deep networks, the gradient vanishing problem is prone to occur.

[0102] Solution:

[0103] ResNet50 is used as the feature extraction network, and the parameter setting includes two basic blocks: ConvolutionalBlock and Identity Block. The specific parameter settings are as follows:

[0104] Convolutional Block: A 1×1 convolutional layer is used to reduce the number of channels, a 3×3 convolutional layer is used for feature learning, and finally a 1×1 convolutional layer is used to keep the dimension of the feature map consistent.

[0105] Identity Block: Contains three convolutional layers (1×1, 3×3, 1×1), which directly add the input to the output through skip connections, retaining the original input information, alleviating the gradient vanishing problem in deep networks, and enabling the network to learn effective feature representations more deeply and stably.

[0106] 2. Feature encoding and fusion

[0107] Current issues:

[0108] Traditional methods cannot fully integrate the target's state information and spatial features when processing target position encoding and bounding box information, resulting in insufficient robustness of the model.

[0109] Solution:

[0110] Through the feature encoding and fusion module, the parameters and algorithms are designed as follows:

[0111] Target position encoding: Set a learnable Token Embedding to represent the foreground, multiply it by the Gaussian graph centered at the target position, obtain the target encoding function ψ(y_i,R_fg)=y_i·R_fg, and superimpose it with the feature. The formula is v_i=x_i+ψ(y_i,R_fg).

[0112] Bounding box encoding: Calculate the normalized distance from each point on the training feature map to the target box, use a multi-layer perceptron (MLP) to encode the bounding box, and change the dimension from 4 to C so that the subsequent fusion feature dimensions are consistent. The formula is φ(d_i).

[0113] 3. Global Information Reasoning and Transformer Encoding

[0114] Current issues:

[0115] Traditional target tracking algorithms lack global context information reasoning and cannot accurately distinguish foreground and background in complex backgrounds.

[0116] Solution:

[0117] The Transformer encoder is used for global information reasoning. The specific parameters and algorithm design are as follows:

[0118] Multi-head self-attention module: Reshape the training and test features into a two-dimensional sequence, and perform global reasoning through the multi-head self-attention module, with the formula [t_1,…,t_m,t_test]=T_enc([v_1,…,v_m,v_test]).

[0119] Transformer decoder: receives the encoded features and outputs the target model weights, with the formula w=T_dec([t_1,…,t_m,t_test],R_fg).

[0120] 4. Bounding Box Regression and Object Classification

[0121] Current issues:

[0122] Traditional algorithms have low efficiency and accuracy in bounding box regression and target classification, especially for small targets and multi-scale targets.

[0123] Solution:

[0124] Through the anchor-free bounding box regression method and efficient target classification algorithm, the specific parameters and algorithm design are as follows:

[0125] Bounding box regression: Using convolution operations and attention mechanisms, the test features are weighted and the bounding box is regressed. The formula is Z_weighted = Attention(t_test*ω_bbreg)⊙t_test.

[0126] Target classification: Use the target classification weight and test features to calculate the similarity and obtain the target classification score. The formula is h(w,t_test)=w*t_test.

[0127] Significant technological advancements:

[0128] 1. Integrating Deep Learning and Transformer Technology

[0129] This invention combines deep learning with Transformer technology for the first time, and performs global context information reasoning through Transformer, which significantly improves the accuracy and robustness of target tracking.

[0130] 2. Anchor-free bounding box regression

[0131] The anchor-free bounding box regression method is adopted. Through convolution operation and attention mechanism, convolution operation is performed directly on the feature map, which improves the efficiency and accuracy of bounding box regression.

[0132] 3. Multi-level feature fusion

[0133] By multi-level fusion of target position encoding, bounding box information encoding and features of feature extraction network, the model can better capture the spatial and contextual information of the target and improve the performance of target tracking.

[0134] 4. Enhanced real-time performance

[0135] By adopting the efficient ResNet50 feature extraction network and Transformer global information processing capabilities, this method has high real-time performance while maintaining high accuracy, meeting the needs of practical applications.

[0136] 5. Wide applicability

[0137] This method is not only suitable for scenarios with high real-time requirements such as autonomous driving and drones, but can also be widely used in security monitoring, intelligent transportation, industrial detection and other fields, and has broad application prospects.

[0138] Through the above parameters, algorithms and mathematical models, the present invention realizes efficient and accurate target tracking and makes significant technical progress in complex background, multi-scale target processing, real-time performance and target classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0139] Figure 1 is a flow chart of a target tracking method of a prediction modeler provided by an embodiment of the present invention;

[0140] Figure 2 It is a flow chart of a tracking module of a target tracking method based on a prediction modeler combining position coding and target frame coding provided by an embodiment of the present invention;

[0141] Figure 3 It is a structural diagram of a target tracking system of a prediction modeler provided by an embodiment of the present invention;

[0142] Figure 4 This is a table of data set experiment results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0143] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0144] The following are two specific application examples based on the above target tracking method:

[0145] Example 1: Target tracking for autonomous vehicles

[0146] In autonomous vehicles, real-time and accurate tracking of vehicles and pedestrians ahead is an important task to ensure driving safety.

[0147] 1) Image acquisition: The cameras equipped on autonomous vehicles continuously capture video sequences of the road ahead.

[0148] 2) Feature extraction: Three frames of images are selected from the video sequence every second, two frames are used as training images, and one frame is used as a test image. They are input into a feature extraction network (such as ResNet50) for feature extraction.

[0149] 3) Feature encoding and fusion:

[0150] The training images are encoded with target position and bounding box information.

[0151] Test images added for test encoding.

[0152] 4) Feature union and Transformer encoding: The training and test features are input into the Transformer encoder to generate enhanced features through global reasoning.

[0153] 5) Predict target model weights: The Transformer decoder outputs the target model weights.

[0154] 6) Bounding box regression and target classification: Use weights to perform bounding box regression and target classification to identify the location and type of vehicles and pedestrians ahead.

[0155] 7) Decision-making and control: Based on the target tracking results, the autonomous driving system makes corresponding driving decisions, such as slowing down, turning or stopping.

[0156] Effect: This embodiment can track vehicles and pedestrians on the road ahead in real time, ensuring the safe driving of the autonomous vehicle in a complex road environment.

[0157] Example 2: Target tracking of drones

[0158] In UAV applications, real-time tracking of ground targets such as people, animals, or vehicles is critical for mission execution such as surveillance, search and rescue, or logistics.

[0159] 1) Image acquisition: The camera equipped by the drone continuously captures video sequences of ground targets.

[0160] 2) Feature extraction: Three frames of images are selected from the video sequence every second, two frames are used as training images, and one frame is used as a test image. They are input into a feature extraction network (such as ResNet50) for feature extraction.

[0161] 3) Feature encoding and fusion:

[0162] The training images are encoded with target position and bounding box information.

[0163] Test images added for test encoding.

[0164] 4) Feature union and Transformer encoding: The training and test features are input into the Transformer encoder to generate enhanced features through global reasoning.

[0165] 5) Predict target model weights: The Transformer decoder outputs the target model weights.

[0166] 6) Bounding box regression and target classification: Use weights to perform bounding box regression and target classification to identify the location and type of ground targets.

[0167] 7) Mission execution: Based on the target tracking results, the drone performs corresponding operations, such as tracking the target, adjusting the flight path, or executing the mission (such as delivering items, taking images or videos).

[0168] Effect: This embodiment can accurately track ground targets in real time, improving the efficiency and accuracy of UAV mission execution in applications such as monitoring, search and rescue, and logistics.

[0169] like Figure 1 As shown, the target tracking method of the prediction modeler provided by the embodiment of the present invention includes:

[0170] (1) Obtain three frames of images in a video sequence, two of which are used as training images and one as a test image, and input them into a feature extraction network for feature extraction;

[0171] (2) Add bounding box information encoding and insertion position information encoding to the Gaussian map centered on the target information in the training frame, add encoding to the test frame, and perform feature mixing with the depth image of the test frame;

[0172] (3) The features from the test and training branches are combined and input into the Transformer encoder to generate enhanced features by inferring the global information between the test frame and the training frame;

[0173] (4) The Transformer decoder predicts the target model weights through the output of the Transformer encoder;

[0174] (5) The predicted target model weights obtained through the prediction model are passed through a linear layer to obtain the bounding box regression weights and target classification weights.

[0175] (6) Receive the bounding box regression weight and target classification weight from the prediction module, as well as the test features extracted by the Transformer, perform bounding box regression on the test features, and implement target classification.

[0176] The target tracking method based on the prediction model of Transformer combined with position coding and target frame coding is characterized in that ResNet50 is used to extract feature information, and step (1) includes the following sub-steps:

[0177] (1.1) The entire target tracking framework of this method includes a test branch and a training branch.

[0178] (1.2) Obtain three different frames of images in the video sequence, two of which are used as training images and one as a test image.

[0179] (1.3) ResNet50 is used as the feature extraction module, where ResNet50 has two basic blocks: Convolutional Block and IdentityBlock.

[0180] The typical structure of Convolutional Block is to reduce the number of channels through a 1×1 convolution layer to reduce the computational complexity; then pass through a 3×3 convolution layer for learning and extracting features, and finally pass through a 1×1 convolution layer to keep the dimension of the feature map consistent.

[0181] The Identity Block consists of three convolutional layers, of which the first and third are 1×1 convolutional layers. The first convolutional layer is used to reduce or restore the number of channels, and the third is used to restore the number of channels of the feature map to keep the dimension consistent. The second layer is a 3x3 convolutional layer, which is used to learn to extract features. The input and output of the Identity Block have the same dimension, and the input is directly added to the output through a jump connection, which retains the information of the original input.

[0182] ResNet50 builds the entire network by stacking Convolutional Block and Identity Block, thereby alleviating the gradient vanishing problem in deep networks. This allows the network to be deeper. The residual connection allows the network to learn more effective feature representations and the training is more stable.

[0183] The target tracking method of the prediction model based on Transformer combined with position coding and target box coding is characterized by proposing a target positioning coding, embedding the position information coding in the training image, setting a learnable Token Embedding, and adding the coding of the target bounding box, so that the input of the model predictor is richer, and the information that can be used by the Transformer-based model predictor is richer. Step (2) includes the following sub-steps:

[0184] (2.1) Add a target position encoding to the training frame, which allows the model predictor to add the target state information in the training frame when predicting the target model. The specific operation is as follows: Set a learnable Token Embedding to represent the foreground so that the model predictor can incorporate the target state information from the training frame. Combine it with the Gaussian graph y centered at the target position i Point product is used as position information encoding. The specific target encoding function formula is as follows:

[0185] ψ(y i ,R fg )=y i ·R fg

[0186] Among them, ψ(y i ,R fg ) represents the target encoding function, and the features are obtained by encoding the target position. is a learnable embedding Token Embedding representation of the foreground, where C represents the number of channels, · represents the dot product, represents a Gaussian map centered at the location of the training object in the training image.

[0187] Then superimpose the target code with the original feature. The specific superposition method is as follows:

[0188] v i =x i +ψ(y i ,R fg )

[0189] The target code ψ(y i ,R fg ) and the features obtained by the feature extraction module of training image 1 and training image 2 and Add together, and the training features obtained are Contains the encoded target state information, where H and W are the height and width of the input training image, and C is the number of channels.

[0190] (2.2) For symmetry, a test code is added to identify the features corresponding to the test frame.

[0191] v test =x test +μ(R test )

[0192] Among them, x test is the test feature obtained by the feature extraction module of the test image, μ(R test ) is x test Repeating token R for each block test , because the specific target position is unknown in the test frame, TokenR test Superimposed on each patch position.

[0193] (2.3) Add another encoding to merge the information about the target bounding box, calculate the distance between each point on the training feature map and the four edges of the target box, and obtain a densely distributed target box, where the training frame represents the bounding box containing the target object in training frame i, and represents the position and size of the target object in training frame i, where and represents the coordinates of the center of the bounding box, and Represents the width and height of the bounding box, and the feature map x i Each position (j x ,j y ) coordinates are mapped to the coordinates in the image domain (k x ,k y ). The calculation formula is as follows:

[0194]

[0195] Where s is the stride of the feature extraction network. Calculate each remapped position to the left of the bounding box l i , the right side of the bounding box r i , the upper edge of the bounding box i , the bounding box bottom b i The normalized distance of is used to obtain the bounding box. The formula is as follows:

[0196]

[0197] Among them, H im =s·H,W im =s·W, corresponding to the spatial dimension of the image block and the depth feature x used to extract i The stride of the backbone network is d i =(l i ,t i ,r i ,b i ),and

[0198] (2.4) Use MLP multi-layer perceptron to encode the bounding box and change the dimension from 4 to C so that the subsequent fusion feature dimensions are consistent. After the dimension is increased, the training image features, test image features, and their target position encodings are superimposed together. The specific formula is as follows:

[0199]

[0200] Among them, ψ(y i ,R fg ) is the feature obtained by encoding the target position in the above step (2.1), x i is the training image frame feature obtained through the feature extraction network, is the target box d i Using Multilayer Perceptron By combining the three together, the model can better capture the spatial and contextual information of the target, while also obtaining the target state information of the training image frame to enhance the features.

[0201] The target tracking method based on a prediction modeler combining Transformer with position coding and target frame coding is characterized in that the Transformer encoder jointly processes the features of the training frame image and the test frame image, calculates the features to be used for the Transformer decoder, and simultaneously calculates the enhanced test frame features, which are used as the input of the target model in the subsequent positioning of the target, and its step (3) includes the following sub-steps:

[0202] (3.1) The training features containing multiple codes will be obtained Contains coded test features Reshape these superimposed features into And expand all m training features v along the first dimension i and test feature v test Connect them together. It turns it into a two-dimensional sequence and processes it through the Transformer encoder, where the sequence length is H·W and C is the feature dimension.

[0203] [t1,…,t m ,t test ]=T enc ([v1,…,v m ,v test ])

[0204] The Transformer encoder consists of a multi-head self-attention module, which enables it to perform global reasoning across the entire frame or even multiple training and test frames. After this encoder, the target state can be identified as foreground and background regions respectively, and the Transformer can distinguish between the two regions.

[0205] The target tracking method based on the prediction model of Transformer combined with position coding and target frame coding is characterized in that the Transformer decoder receives the Transformer encoder T i and T test , predict the target model weight, and step (4) includes the following sub-steps:

[0206] (4.1) The Transformer decoder receives the t of the Transformer encoder. i and t test , which is obtained by combining the characteristics of the entire training sample and the test sample. The details are as follows:

[0207] w=T dec ([t1,…,t m ,t test ],R fg )

[0208] Among them, input t1,…,t m ,t test It is obtained by joint reasoning of training samples and test samples, and uses the same learnable foreground embedding R fg The query is fed into the Transformer decoder, which enables the resulting weights to make the prediction model more discriminative.

[0209] The target tracking method based on the prediction model of Transformer combined with position coding and target frame coding is characterized in that the prediction model obtains the frame regression weight ω by passing the weight w output by the Transformer decoder through a linear layer bbreg and target classification weight ω cls , wherein step (5) comprises the following sub-steps:

[0210] (5.1) The weight w output by the Transformer decoder passes through a linear layer to obtain the weight ω for bounding box regression bbreg and target classification ω cls .

[0211] (5.2) Target weight ω cls The predicted target model is directly used for subsequent target classification, and the target model classifies the target by applying these weights to the test features.

[0212] (5.3) Weight ω for bounding box regression bbreg Conditioning the test features enables the network to regress the bounding box based on the target state of the current frame.

[0213] The target tracking method based on the prediction model of Transformer combined with position encoding and target box encoding is characterized in that the bounding box regression is performed through the attention map obtained by calculation, and then the border regression is performed, and the DCF discriminant model is used to obtain the target classification score, and the step (6) thereof includes the following sub-steps:

[0214] (6.1) In order to make the encoder output the test feature t test It is target perception. An attention map is obtained through self-attention calculation. The test features are weighted using the attention map. The weighted feature expression is as follows:

[0215] Z weighted =Attention(t test *ω bbreg )⊙t test ,

[0216] Where ⊙ represents element-by-element multiplication, The weighted features are then input into the convolutional neural network (CNN), which finally uses an exponential activation function to produce normalized bounding box predictions under the same ltrb representation. The bounding box of the target is regressed at each position in the feature map, and the boundary of the target relative to that position is represented by four distances (left, top, right, and bottom). By combining the feature extraction of the Transformer encoder and the anchor-free bounding box regression method, efficient bounding box regression is achieved by performing convolution operations directly on the feature map, while utilizing the global context information of the Transformer to improve detection accuracy.

[0217] (6.2) Use the Transformer decoder to output the weight ω generated by the linear layer cls And the test feature map output by the encoder, analogous to the DCF discriminant model, by learning a filter, and obtaining the target classification score h(w,t test ). The specific formula is as follows:

[0218] h(w,t test )=w*t test

[0219] Where w is the weight of the filter output by the pre-decoder, t test is the test feature after the training feature enhancement in the encoder, * is the convolution operation, which combines the output features of the Transformer decoder and encoder, generates classification weights using a linear layer, and obtains the target classification score through similarity calculation. This method combines the global context understanding ability of the Transformer with the efficiency of the convolution operation to achieve effective target classification.

[0220] Another object of the present invention is to provide a target tracking system of a prediction modeler for implementing the target tracking method of the prediction modeler, comprising:

[0221] A training image acquisition module is used to acquire three frames of images in a video sequence, two of which are used as training images and one as a test image, and input them into a feature extraction network for feature extraction;

[0222] A feature mixing module is used to add bounding box information encoding and insertion position information encoding to the Gaussian map centered on the target information in the training frame, add encoding to the test frame, and perform feature mixing with the test frame depth image;

[0223] The enhanced feature acquisition module is used to combine the features from the test and training branches and input them into the Transformer encoder to generate enhanced features by inferring the global information between the test frame and the training frame;

[0224] A target model weight prediction output module is used to predict the target model weight using the output of the Transformer encoder using the Transformer decoder;

[0225] The border regression weight and target classification weight acquisition module is used to obtain the border regression weight and target classification weight by passing the predicted target model weight obtained by the prediction model through a linear layer.

[0226] The target classification module is used to receive the bounding box regression weights and target classification weights from the prediction module, as well as the test features extracted by the Transformer, perform bounding box regression on the test features, and realize target classification.

[0227] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the target tracking method of the prediction modeler.

[0228] An application embodiment of the present invention provides an information data processing terminal, which includes a target tracking system of a prediction modeler.

[0229] 1. Specific application fields or related products of the present invention.

[0230] Application Example 1: Intelligent Security Video Surveillance System

[0231] In intelligent security video surveillance, in public places such as parks, squares, exhibition halls, etc., the security monitoring system uses target tracking technology to monitor people's behavior in real time. The system can identify and track potential threats or suspicious persons, for example, people who linger in the same place for too long, frequently change locations, or behave suspiciously; when the monitoring system detects abnormal behavior (such as fighting, gathering, etc.), it can immediately issue an alarm to notify security personnel to rush to the scene to deal with it quickly. The system can also record the entire process of the incident to provide reliable evidence for subsequent investigations.

[0232] 1) Training image acquisition module: Record multiple videos of public places, extract frames from the normal video dataset as training sets, and input these images into the pre-trained feature extraction network.

[0233] 2) Feature mixing module: The Gaussian map centered on the target information in the training frame is encoded with bounding box information and inserted with position information, and the test frame is also encoded and feature mixed with the test frame depth image.

[0234] 3) Enhanced feature acquisition module: The features from the test and training branches are combined and input into the Transformer encoder to generate enhanced features by inferring the global information between the test frame and the training frame.

[0235] 4) Prediction target output module: Input the enhanced features into the Transformer decoder to generate the predicted target model weights. Pass the predicted target model weights obtained by the prediction model through the linear layer to obtain the bounding box regression weights and target classification weights. Receive the bounding box regression weights and target classification weights from the prediction module, as well as the test features extracted by the Transformer, perform bounding box regression on the test features and implement target classification

[0236] Through the above modules, the system can monitor the video stream in real time, and identify and track normal target objects in the picture. At the same time, because of the real-time introduction of test frame feature information encoding, the system can detect and alarm abnormal behaviors such as abnormal wandering.

[0237] 2. Relevant evidence of the technical effects obtained by the embodiments of the present invention.

[0238] To illustrate the effect of the technical solution described in the present invention, please refer to Figure 4 We used three well-known datasets in the field of target tracking, LaSOT, NFS, and OTB, as benchmarks to verify that the proposed method of adding position information encoding and bounding box information encoding, as well as verifying that the information encoding of test features significantly improves the performance of the prediction modeler. Compared with the prediction modeler that does not use the idea proposed by this method, the effect is improved by 0.3% to 0.5% accuracy.

[0239] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. A person of ordinary skill in the art will appreciate that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVDROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0240] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A target tracking method for a prediction modeler, characterized in that: include: S1, obtain three frames of images in the video sequence, two of which are used as training images and one as a test image, and input them into the feature extraction network for feature extraction; S2, add bounding box information encoding and insertion position information encoding to the Gaussian map centered on the target information in the training frame, add encoding to the test frame as well, and perform feature mixing with the depth image of the test frame; S3, combines the features from the test and training branches and inputs them into the Transformer encoder, generating enhanced features by reasoning about the global information between the test frame and the training frame; S4, Transformer decoder predicts the target model weights through the output of Transformer encoder; S5, the predicted target model weight obtained by the prediction model is passed through a linear layer to obtain a bounding box regression weight and a target classification weight; S6 receives the bounding box regression weight and target classification weight from the prediction module, as well as the test features extracted by the Transformer, performs bounding box regression on the test features, and implements target classification.

2. The target tracking method of the prediction modeler according to claim 1, characterized in that: Step S1 includes the following sub-steps: (1.1) The entire target tracking framework process includes a test branch and a training branch; (1.2) Obtain three different frames of images in the video sequence, two of which are used as training images and one as a test image; (1.3) Use ResNet50 as the feature extraction module, where ResNet50 has two basic blocks: ConvolutionalBlock and IdentityBlock; The typical structure of Convolutional Block is to reduce the number of channels through a 1×1 convolution layer to reduce the computational complexity; then pass through a 3×3 convolution layer for learning and extracting features, and finally pass through a 1×1 convolution layer to keep the dimension of the feature map consistent; The Identity Block consists of three convolutional layers, of which the first and third are 1×1 convolutional layers. The first convolutional layer is used to reduce or restore the number of channels, and the third is used to restore the number of channels of the feature map to keep the dimension consistent. The second layer is a 3x3 convolutional layer, which is used to learn and extract features. The input and output of Identity Block have the same dimension, and the input is directly added to the output through skip connection, which preserves the information of the original input; ResNet50 builds the entire network by stacking Convolutional Block and Identity Block, thereby alleviating the gradient vanishing problem in deep networks; making the network deeper; residual connections enable the network to learn more effective feature representations and make training more stable.

3. The target tracking method of the prediction modeler according to claim 1, characterized in that: Step S2 includes the following sub-steps: (2.1) Add a target position encoding to the training frame, which allows the model predictor to add the target state information in the training frame when predicting the target model; the specific operation is as follows: set a learnable Token Embedding to represent the foreground so that the model predictor can incorporate the target state information from the training frame; compare it with the Gaussian graph y centered at the target position i Point multiplication is used as position information encoding; the specific target encoding function formula is as follows: ψ(y i ,R fg )=y i ·R fg Among them, ψ(y i , R fg ) represents the target encoding function, and the features are obtained by encoding the target position. is a learnable embedding Token Embedding representation of the foreground, where C represents the number of channels, · represents the dot product, represents a Gaussian map centered at the training target location in the training image; Then superimpose the target code with the original feature. The specific superposition method is as follows: v i =x i +ψ(y i ,R fg ) The target code ψ(y i , R fg ) and the features obtained by the feature extraction module of training image 1 and training image 2 and Add together, and the training features obtained are Contains the encoded target state information, where H and W are the height and width of the input training image, and C is the number of channels; (2.2) For symmetry, a test code is added to identify the features corresponding to the test frame; v test =x test +μ(R test ) Among them, x test is the test feature obtained by the feature extraction module of the test image, μ(R test ) is x test Repeating token R for each block test , because the specific target position is unknown in the test frame, Token R test Superimpose on each patch position; (2.3) Add another encoding to merge the information about the target bounding box, calculate the distance between each point on the training feature map and the four edges of the target box, and obtain a densely distributed target box, where the training frame represents the bounding box containing the target object in training frame i, and represents the position and size of the target object in training frame i, where and represents the coordinates of the center of the bounding box, and Represents the width and height of the bounding box, and the feature map x i Each position (j x , j y ) coordinates are mapped to the coordinates in the image domain (k x , k y ), the calculation formula is as follows: Where s is the stride of the feature extraction network; calculate each remapped position to the left of the bounding box l i , the right side of the bounding box r i , the upper edge of the bounding box i , the bounding box bottom b i The normalized distance of is used to obtain the bounding box. The formula is as follows: Among them, H im =s·H,W im =s·W, corresponding to the spatial dimension of the image block and the depth feature x used to extract i The stride of the backbone network is d i =(l i , t i , r i , b i ),and (2.4) Use MLP multi-layer perceptron to encode the bounding box and change the dimension from 4 to C so that the subsequent fusion feature dimensions are consistent; after the dimension is increased, the training image features, the test image features, and the target position encoding are superimposed together. The specific formula is as follows: Among them, ψ(y i , R fg ) is the feature obtained by encoding the target position in the above step (2.1), x i is the training image frame feature obtained through the feature extraction network, is the target box d i Using Multilayer Perceptron By combining the three together, the model can better capture the spatial and contextual information of the target, while also obtaining the target state information of the training image frame to enhance the features.

4. The target tracking method of the prediction modeler according to claim 1, characterized in that: Step S3 includes the following sub-steps: (3.1) will get training features containing multiple codes Contains coded test features Reshape these superimposed features into And expand all m training features v along the first dimension i and test feature v test Connect them together; turn them into a two-dimensional sequence and process them through the Transformer encoder, where the sequence length is H·W and C is the feature dimension; [t1,...,t m ,t test ]=T enc ([v1,...,v m ,v test ]) The Transformer encoder consists of a multi-head self-attention module, which enables it to perform global reasoning across the entire frame or even multiple training and test frames; after this encoder, the target state can be identified as foreground and background areas respectively, and the Transformer can distinguish between the two areas.

5. The target tracking method of the prediction modeler according to claim 1, characterized in that: Step S4 includes the following sub-steps: (4.1) The Transformer decoder receives the t of the Transformer encoder. i and t test , which is obtained by combining the characteristics of the entire training sample and the test sample, as follows: w=T dec ([t1,...,t m ,t test ], R fg ) Among them, input t1,...,t m , t test It is obtained by joint reasoning of training samples and test samples, and uses the same learnable foreground embedding R fg The query is fed into the Transformer decoder, which enables the resulting weights to make the prediction model more discriminative.

6. The target tracking method of the prediction modeler according to claim 1, characterized in that: Step S5 includes the following sub-steps: (5.1) The weight w output by the Transformer decoder passes through a linear layer to obtain the weight ω for bounding box regression bbreg and target classification ω cls ; (5.2) Weight ω cls The predicted target model is directly used for subsequent target classification. The target model classifies the target by applying these weights to the test features. (5.3) Weight ω for bounding box regression bbreg Conditioning the test features enables the network to regress the bounding box based on the target state of the current frame.

7. The target tracking method of the prediction modeler according to claim 1, characterized in that: Step S6 includes the following sub-steps: (6.1) In order to make the encoder output the test feature t test It is target perception. An attention map is obtained through self-attention calculation. The test features are weighted using the attention map. The weighted feature expression is as follows: Z weighted =Attention(t test *ω bbreg )⊙t test Where ⊙ represents element-by-element multiplication, The weighted features are then input into the convolutional neural network (CNN), which finally uses an exponential activation function to produce normalized bounding box predictions under the same ltrb representation; the bounding box of the target is regressed at each position in the feature map, and the boundary of the target relative to that position is represented by four distances; by combining the feature extraction of the Transformer encoder and the anchor-free bounding box regression method, efficient bounding box regression is achieved by performing convolution operations directly on the feature map, and the global context information of the Transformer is utilized to improve the detection accuracy; (6.2) Use the Transformer decoder to output the weight ω generated by the linear layer cls And the test feature map output by the encoder, analogous to the DCF discriminant model, by learning a filter, and obtaining the target classification score h(w, t test ), the specific formula is as follows: h(w,t test )=w*t test Where w is the weight of the filter output by the pre-decoder, t test is the test feature after the training feature enhancement in the encoder, * is the convolution operation, which combines the output features of the Transformer decoder and encoder, uses the linear layer to generate the classification weight, and obtains the target classification score through similarity calculation; this method combines the global context understanding ability of Transformer and the efficiency of convolution operation to achieve effective target classification.

8. A target tracking system for a prediction modeler that implements the target tracking method for a prediction modeler according to any one of claims 1 to 7, characterized in that: include: A training image acquisition module is used to acquire three frames of images in a video sequence, two of which are used as training images and one as a test image, and input them into a feature extraction network for feature extraction; A feature mixing module is used to add bounding box information encoding and insertion position information encoding to the Gaussian map centered on the target information in the training frame, add encoding to the test frame, and perform feature mixing with the test frame depth image; The enhanced feature acquisition module is used to combine the features from the test and training branches and input them into the Transformer encoder to generate enhanced features by inferring the global information between the test frame and the training frame; A target model weight prediction output module is used to predict the target model weight using the output of the Transformer encoder using the Transformer decoder; A module for obtaining a bounding box regression weight and a target classification weight, for obtaining a bounding box regression weight and a target classification weight by passing the predicted target model weight obtained by the prediction model through a linear layer; The target classification module is used to receive the bounding box regression weights and target classification weights from the prediction module, as well as the test features extracted by the Transformer, perform bounding box regression on the test features, and realize target classification.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the target tracking method of the prediction modeler as described in any one of claims 1 to 7.

10. An information data processing terminal, comprising the target tracking system of the prediction modeler according to claim 8.

Citation Information

Cited By

  • Wheel part detection method based on multi-scale dynamic fusion

    CN120071000A