Target tracking method under appearance change of improved transformation model prediction network
By improving the conversion model prediction network, using the backbone network and cascaded LSTM network to adapt to changes in target appearance, and predicting new positions through continuous frame target center point trajectory information, the problem of low tracking accuracy when the target appearance changes is solved, achieving a more stable better tracking performance and higher accuracy.
Patent Information
- Application Number
- CN202411461376.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-01-21
AI Technical Summary
The existing target tracking methods cannot be adjusted in time when the target appearance changes, resulting in low tracking accuracy and ineffective handling of interference from similar objects.
An improved conversion model prediction network is used to extract target depth features through the backbone network, and combined with an online weight update method to adapt to the target appearance state. Use the cascaded LSTM network to learn continuous tracking result information, optimize the model weight to adapt to the new appearance target, and use the target center point trajectory information in continuous frames to predict new positions, and perform weighting processing to alleviate similar object interference.
It improves the tracking accuracy when the target appearance changes, enhances the model's tracking stability for target appearance changes, and effectively alleviates the interference of similar objects on tracking.
Smart Images
Figure CN119338861A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of visual target tracking, and in particular to a target tracking method under appearance changes of an improved conversion model prediction network. Background Art
[0002] Target tracking is one of the research hotspots in the field of computer vision. Its method flow is: given a target frame in the initial frame of a video sequence, track the position and shape of the target in subsequent frames. During the tracking process, the target shape often changes to varying degrees, making it impossible for the tracking model to make timely adjustments. This is one of the problems that need to be urgently solved in the target tracking task.
[0003] During the tracking process, the target often undergoes appearance changes to varying degrees, making it impossible for the target tracking model to make timely adjustments. Existing target tracking work does not consider the state information of the target when its appearance changes, and cannot stably track targets with appearance changes. In addition, some methods focus on obtaining the feature representation of the target, and cannot handle the interference of similar objects when the target changes, resulting in low target tracking accuracy. Summary of the invention
[0004] The purpose of the present invention is to provide a method for tracking a target under appearance changes using an improved conversion model prediction network, aiming to improve the tracking accuracy when the target appearance changes.
[0005] To achieve the above object, the present invention provides a method for tracking a target under appearance changes using an improved conversion model prediction network, comprising the following steps:
[0006] Use the backbone network to extract the target's deep features and combine it with the online weight update method to adapt to the target's appearance state;
[0007] Collect target tracking result information based on the target appearance state, use a cascaded LSTM network to learn continuous result information and output the current target appearance state;
[0008] Improve the online learning sample utilization mechanism of the model. When the target undergoes severe appearance changes, use reliable target samples to optimize the model weights and adapt the model to the new appearance target.
[0009] The new position of the new appearance target is predicted using the trajectory information of the target center point in consecutive frames, and the target response score is weighted to allow the model to adapt to the new state of the target.
[0010] Among them, the ResNet50 backbone network is used to extract the target training and test feature information, and the target label information is encoded and integrated into the training features;
[0011] Processing the target training and test features, concatenating them, and jointly processing them using a transformer network to optimize target classification weights and bounding box regression weights;
[0012] Use the model weights to classify the target image and locate the target, and output the tracking result information;
[0013] Use subsequent reliable samples to continuously optimize the model.
[0014] The specific method of collecting target tracking result information based on the target appearance state, using the cascaded LSTM network to learn continuous result information and output the current target appearance state is:
[0015] Collect the tracking result information of the basic model, input the tracking result information of the first 20 frames into the cascade LSTM network, and initialize the discriminant model;
[0016] The subsequent tracking result information is integrated and input into the network to continuously judge the target appearance state.
[0017] The improved model online learning sample utilization mechanism uses reliable target samples to optimize model weights when the target undergoes severe appearance changes, so that the model can adapt to the new appearance target in the following specific ways:
[0018] When the discriminant network outputs a serious appearance change, the initial frame sample is discarded, and the model weight is optimized by combining reliable target samples and current target samples to adapt to the target appearance change;
[0019] The initial frame samples are reused after spacing several frames and stabilizing the tracking.
[0020] The specific method of using the trajectory information of the target center point of the continuous frames to predict the new position of the new appearance target and weighting the target response score to adapt the model to the new state of the target is as follows:
[0021] Initialize the prediction model using the target center point of the initial frame as the starting point;
[0022] Use the continuous target center point trajectory information to input the LSTM prediction network for learning;
[0023] Optimizing the prediction model using predicted target center points and reliable target center point information;
[0024] When the target undergoes a serious appearance change, the predicted center point is used as the target center position to re-crop the image and extract the target feature information;
[0025] The target response mapping scores are weighted based on distance;
[0026] After correcting the target response mapping, the target is accurately located through the bounding box regression weights to obtain the accurate target bounding box, and the current frame image sample information is updated in time to allow the model to adapt to the new state of the target.
[0027] The invention discloses a target tracking method under appearance change of an improved conversion model prediction network, which uses a backbone network to extract target depth features, and combines an online weight updating method to adapt to the target appearance state; collects target tracking result information based on the target appearance state, uses a cascaded LSTM network to learn continuous result information and outputs the current target appearance state, so as to provide a prerequisite for the model to cope with severe target appearance change; improves the model online learning sample utilization mechanism, and uses reliable target samples to optimize the model weights when the target undergoes severe appearance change, so as to adapt the model to the new appearance target; uses continuous frame target center point trajectory information to predict the new position of the new appearance target, and weights the target response score, so as to adapt the model to the new state of the target, and effectively alleviates the interference of similar objects on tracking when the target appearance changes; the method makes full use of the model online learning mechanism, and improves the tracking stability of the model to the target appearance change; collects continuous tracking result information and inputs it into the cascaded LSTM, so as to discriminate the current target appearance state, so as to provide a prerequisite for the model to cope with severe target appearance change. This method can effectively alleviate the problem that the tracking model cannot effectively cope with the problem of rapid and severe appearance changes of the target, and provide reliable criteria for subsequent solutions; it uses the continuous target center point position information to predict the current position of the target, and weights the target response score when the target appearance changes, effectively alleviating interference from similar objects and enhancing the tracking accuracy during the target appearance change process. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0029] Figure 1 This is a target tracking flow chart of the present invention.
[0030] Figure 2 This is the overall structure diagram of the network of the present invention.
[0031] Figure 3 This is the trajectory prediction effect diagram.
[0032] Figure 4 The following is a graph showing the tracking success rate results of different methods on the OTB100 dataset.
[0033] Figure 5This is the tracking accuracy result of different methods under the OTB100 dataset.
[0034] Figure 6 The following is the tracking success rate result of different methods under the LASOT dataset.
[0035] Figure 7 This is the tracking accuracy result of different methods under the LASOT dataset.
[0036] Figure 8 It is a flowchart of a method for tracking a target under appearance changes of an improved conversion model prediction network provided by the present invention.
[0037] Fig. 9 It is a flowchart of the specific method of using the backbone network to extract the target's deep features and combining it with the online weight update method to adapt to the target's appearance state.
[0038] Fig.10 It is a flowchart of a specific method for collecting target tracking result information based on the target appearance state, using a cascaded LSTM network to learn continuous result information and outputting the current target appearance state.
[0039] Fig.11 It is a flowchart of the specific method of improving the model's online learning sample utilization mechanism, optimizing the model weights by using reliable target samples when the target undergoes severe appearance changes, and allowing the model to adapt to the new appearance target.
[0040] Fig.12 It is a flowchart of a specific method of using the trajectory information of the target center point in consecutive frames to predict the new position of the new appearance target and weighting the target response score to allow the model to adapt to the new state of the target. DETAILED DESCRIPTION
[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0042] See also Figures 1 to 12 The present invention provides a method for tracking a target under appearance changes of an improved conversion model prediction network, comprising the following steps:
[0043] S1 uses the backbone network to extract the target's deep features and combines it with an online weight update method to adapt to the target's appearance state;
[0044] Specific method:
[0045] S11 uses the ResNet50 backbone network to extract target training and test feature information, and encodes and fuses the target label information into the training features;
[0046] S12 processes the target training and the test features, concatenates them, and uses a transformer network to perform joint processing to optimize the target classification weight and the bounding box regression weight;
[0047] S13 classifies the target image and locates the target using the model weights, and outputs tracking result information;
[0048] S14 uses subsequent reliable samples to continuously optimize the model.
[0049] In the embodiment of the present invention, the model is continuously optimized for subsequent tracking.
[0050] S2 collects target tracking result information based on the target appearance state, uses a cascaded LSTM network to learn continuous result information and outputs the current target appearance state;
[0051] Specific method:
[0052] S21 collects the tracking result information of the basic model, inputs the tracking result information of the first 20 frames into the cascade LSTM network, and initializes the discriminant model;
[0053] In the embodiment of the present invention, the target motion information is directly reflected by collecting the distance between the center points of the previous and next frames of the target, and combining the continuous frame boundary box b t =(x t ,y t ,w t ,h t ), target center point p t =(x 0t ,y 0t ) provides data support for the discriminant model, where (x, y) is the coordinate of the upper left corner of the bounding box, and (w, h) is the width and height of the target. The model weights can be used to obtain the response map R of the target in the image. t , the maximum value of its response mapping is the position of the current target in the image,
[0054] S t =max(R t )
[0055] In the formula, S t is the target confidence score of consecutive frames during target tracking, R t is the response map of the target in the image. Further, the response map R t As data input to the convolutional neural network to thoroughly mine the target information in the response map and obtain a continuous response vector
[0056]
[0057] In the formula, is the response vector output by the neural network, W R is the convolutional neural network weight. After obtaining enough sequence information, the bounding box b t , center point p t , confidence score S t , response vector Integrate into a sequence matrix,
[0058]
[0059] Where, t m is the time step from the history sequence to the current frame, The sequence matrix X t As input, initialize the discriminative model.
[0060] S22 integrates the subsequent tracking result information into the network and continuously judges the target appearance state.
[0061] In the embodiment of the present invention, the sequence information integrated by cascade LSTM learning is used, and a three-level cascade LSTM network is used, and the time step of each network is gradually reduced to extract the tracking sequence information and pay attention to the latest frame state information. Two fully connected layers are used to process the final output of LSTM to generate a ternary classification score, and finally judge the appearance state of the target.
[0062] S3 improves the online learning sample utilization mechanism of the model. When the target undergoes severe appearance changes, it uses reliable target samples to optimize the model weights and adapt the model to the new appearance target.
[0063] Specific method:
[0064] When the output target of the S31 discriminant network undergoes a serious appearance change, the initial frame sample is discarded, and the model weight is optimized by combining the reliable target sample and the current target sample to adapt to the target appearance change;
[0065] S32 reuses the initial frame samples after stabilizing the tracking by several frames.
[0066] S4 uses the trajectory information of the target center point in consecutive frames to predict the new position of the new appearance target and weights the target response score to allow the model to adapt to the new state of the target.
[0067] Specific method:
[0068] S41 initializes the prediction model using the target center point of the initial frame as a starting point;
[0069] S42 uses the continuous target center point trajectory information to input into the LSTM prediction network for learning;
[0070] In the embodiment of the present invention, the continuous target center point trajectory information is input into the LSTM prediction network for learning.
[0071]
[0072] In the formula, f is the forget gate, i is the input gate, o is the output gate, c is the storage unit, W and U are the weight matrices to be learned, and b is the bias vector to be learned.
[0073] S43 optimizes the prediction model using the predicted target center point and the reliable target center point information;
[0074] S44: when the target undergoes a serious appearance change, re-cropping the image using the predicted center point as the target center position to extract target feature information;
[0075] S45 weights the target response mapping scores based on distance;
[0076] In an embodiment of the present invention, the target response mapping score is weighted based on the distance.
[0077]
[0078] Where: b is the mapping weight parameter, ω d is the distance weighting parameter, p it The response map position coordinates for the image, p pt Map location coordinates for the response at the predicted center point.
[0079] After correcting the target response mapping, S46 accurately locates the target through the bounding box regression weight, obtains the accurate target bounding box, and promptly updates the current frame image sample information to allow the model to adapt to the new state of the target.
[0080] In order to better understand the present technical solution, the following embodiments are provided for further explanation:
[0081] Step 1: Construct learning samples and initialize the model. Use the backbone network to extract the target's deep features, use the initial frame samples to initialize the tracking model; extract the initial frame target center point position, use the initial position to initialize the prediction model; collect the tracking model result information, and after obtaining a sufficient amount of sequence information, put the bounding box b t , center point p t , confidence score S t , response vector Integrate into a sequence matrix,
[0082]
[0083] The cascaded LSTM learning integrates the sequence information and builds a stable target state discrimination model.
[0084] Step 2: During target tracking, normal tracking samples are stored for online learning. Once the target's appearance changes, the model will immediately discard the initial frame sample information and use the latest frame sample combined with historical reliable samples to re-update the model weights to adapt to the target's appearance changes. At the same time, to avoid model collapse, the initial frame information is reused after several frames and stable tracking.
[0085] Step 3: After the initial positioning of the target, the subsequent frames continuously collect the reliable center point coordinate information of the target and predict the target position information of the next frame to achieve the trajectory prediction of the target in the image. The tracking result of each frame will obtain the predicted target center point position p pt =(x pt ,y pt ), when the discriminant model determines that the current frame is in an unstable tracking state, the predicted center point is used as the target center position to re-crop the image, and the target response mapping score is weighted based on the distance.
[0086]
[0087] After correcting the target response mapping, the target is accurately located through the bounding box regression weights to obtain the accurate target bounding box, and the current frame image sample information is updated in time to allow the model to adapt to the new state of the target and continue to track the target.
[0088] Table 1 shows the average Euclidean distance between the center point of trajectory prediction and the center point of the real target in 6 video sequences in the public dataset OTB100. As shown in Table 1, the average Euclidean distance of the trajectory prediction method is less than 20 pixel values, and the average Euclidean distance of the Diving sequence is only 7.1 pixel values, which shows the effectiveness of the trajectory prediction method. Figure 3 The position of the trajectory prediction center point and the target real center point in the image, from Figure 3 It can be seen that the trajectory prediction curve is basically consistent with the real trajectory line, which further proves the effectiveness of the trajectory prediction method.
[0089] Table 1 Average Euclidean distance error (pixel points)
[0090]
[0091] Figure 4 and Figure 5 Comparison of tracking success rate and accuracy of different methods for deformable visual attributes in the OTB100 dataset, a total of 44 video sequences. Figure 4 and Figure 5 ,The success rate of the proposed method is better than other methods under different ,overlapping thresholds, and has higher tracking accuracy, which shows that the proposed method has better ,tracking stability.
[0092] Figure 6 and Figure 7 Comparison of tracking success rate and accuracy of different methods for deformable visual attributes in the LASOT dataset, a total of 142 video sequences. Figure 6 and Figure 7 ,The proposed method is obviously superior to other methods, which shows that the proposed method has better tracking effect compared with other methods.
[0093] What is disclosed above is only a preferred embodiment of the target tracking method under appearance changes of an improved conversion model prediction network of the present invention. Of course, this cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. A method for tracking a target under appearance changes using an improved conversion model prediction network, characterized in that: The following steps are involved: Use the backbone network to extract the target's deep features and combine it with the online weight update method to adapt to the target's appearance state; Collect target tracking result information based on the target appearance state, use a cascaded LSTM network to learn continuous result information and output the current target appearance state; Improve the online learning sample utilization mechanism of the model. When the target undergoes severe appearance changes, use reliable target samples to optimize the model weights and adapt the model to the new appearance target. The new position of the new appearance target is predicted using the trajectory information of the target center point in consecutive frames, and the target response score is weighted to allow the model to adapt to the new state of the target.
2. The method for tracking a target under appearance changes using an improved conversion model prediction network according to claim 1, characterized in that: The specific method of using the backbone network to extract the target depth features and combining the online weight update method to adapt to the target appearance state is as follows: Use the ResNet50 backbone network to extract target training and test feature information, and encode the target label information into the training features; Processing the target training and test features, concatenating them, and jointly processing them using a transformer network to optimize target classification weights and bounding box regression weights; Use the model weights to classify the target image and locate the target, and output the tracking result information; Use subsequent reliable samples to continuously optimize the model.
3. The method for tracking a target under appearance changes using an improved conversion model prediction network according to claim 1, wherein: The specific method of collecting target tracking result information based on the target appearance state, using the cascaded LSTM network to learn continuous result information and output the current target appearance state is: Collect the tracking result information of the basic model, input the tracking result information of the first 20 frames into the cascade LSTM network, and initialize the discriminant model; The subsequent tracking result information is integrated and input into the network to continuously judge the target appearance state.
4. The method for tracking a target under appearance changes using an improved conversion model prediction network according to claim 1, wherein: The improved model online learning sample utilization mechanism uses reliable target samples to optimize model weights when the target undergoes severe appearance changes, so that the model can adapt to the new appearance target in the following specific ways: When the discriminant network outputs a serious appearance change, the initial frame sample is discarded, and the model weight is optimized by combining reliable target samples and current target samples to adapt to the target appearance change; The initial frame samples are reused after spacing several frames and stabilizing the tracking.
5. The method for tracking a target under appearance changes using an improved conversion model prediction network according to claim 1, wherein: The specific method of using the trajectory information of the target center point of the continuous frames to predict the new position of the new appearance target and weighting the target response score to adapt the model to the new state of the target is: Initialize the prediction model using the target center point of the initial frame as the starting point; Use the continuous target center point trajectory information to input the LSTM prediction network for learning; Optimizing the prediction model using predicted target center points and reliable target center point information; When the target undergoes a serious appearance change, the predicted center point is used as the target center position to re-crop the image and extract the target feature information; The target response mapping scores are weighted based on distance; After correcting the target response mapping, the target is accurately located through the bounding box regression weights to obtain the accurate target bounding box, and the current frame image sample information is updated in time to allow the model to adapt to the new state of the target.