A single-target tracking method and system based on a credit distribution network
By introducing a credit allocation network and a guided focus loss function, combined with a pre-trained GoogleNet network, the problems of data update and occlusion in single-object tracking technology are solved, improving tracking accuracy and real-time performance, and enhancing the network's adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANGTZE DELTA REGION INST OF UNIV OF ELECTRONICS SCI & TECH OF CHINE (HUZHOU)
- Filing Date
- 2023-10-18
- Publication Date
- 2026-05-12
AI Technical Summary
Existing single-target tracking techniques based on convolutional neural networks suffer from problems such as untimely data updates, overfitting, high computational resource consumption, and insufficient handling of target occlusion, which limit the tracking performance.
A credit allocation network and a spatiotemporal memory network are introduced, combined with a guided focusing loss function. The tracking status is evaluated through the credit allocation network, the most reliable memory sample is selected, and features are extracted using a pre-trained GoogleNet network to generate a location information feature map to improve tracking accuracy and real-time performance.
It effectively prevents target appearance model contamination, improves tracking accuracy and stability, enhances network adaptability and real-time performance, and can better handle target occlusion and dynamic changes.
Smart Images
Figure CN117237416B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target tracking technology, and in particular to a single-target tracking method based on a credit allocation network. Background Technology
[0002] In recent years, the rapid development of deep learning technology has greatly advanced the task of visual object tracking. Visual object tracking involves estimating the position of a specific target in each frame of a video sequence, and it is increasingly being used in various fields such as human-computer interaction, security monitoring, and autonomous driving. However, due to complex background interference, developing trackers with strong adaptability and robustness remains challenging.
[0003] Currently, template matching-based tracking schemes are widely discussed. These methods typically generate a target template based on the target's position in the previous or initial frame for tracking in subsequent frames. However, when the target undergoes drastic changes during tracking, the target's shape in the frame to be tracked may be closer to the target's shape in certain historical frames than in the initial or previous frame. Therefore, consistently using the target from the initial or previous frame as a template degrades the adaptability of the matching model. To address this issue, some researchers have attempted to extract target information from historical frames. Most of them use different memory networks to obtain historical target information to update the tracking model. However, in real-world tracking scenarios, targets may encounter deformation, occlusion, or disappearance from the field of view, leading to tracking failure. In these cases, without a robust update evaluation mechanism, indiscriminate updating of memory samples contaminates the target's appearance model.
[0004] Existing technology: Single-object tracking technology based on convolutional neural networks. This technology uses convolutional neural networks to extract image features and trains the network to predict the target's position in subsequent frames.
[0005] Technical problems with existing technologies:
[0006] 1. Data update problem: Traditional single-object tracking technology based on convolutional neural networks usually requires a large amount of labeled data for training. However, in actual tracking, newly appearing or disappearing targets cannot be updated into the model in a timely manner, which limits the tracking effect.
[0007] 2. Overfitting problem: In consecutive video frames, the appearance of the target will undergo slight changes. If the model overfits to the training data, it will not be able to capture these slight changes well, resulting in a decrease in tracking accuracy.
[0008] 3. Real-time issues: Convolutional neural networks require significant computational resources, posing a challenge to real-time object tracking. This is especially true when processing high-resolution or high-frame-rate video, where computational latency becomes a problem.
[0009] 4. Target occlusion problem: In real-world video tracking scenarios, targets may be occluded by other objects. Traditional single-target tracking techniques based on convolutional neural networks are insufficient in handling this issue and cannot accurately track occluded targets.
[0010] The novel method described above effectively solves some problems in existing technologies, such as data update problems and overfitting problems, by introducing a credit allocation network and a spatiotemporal memory network, as well as a guided focus loss function, and improves the accuracy and real-time performance of target tracking. Summary of the Invention
[0011] To address the above shortcomings, this invention proposes a single-target tracking method based on a credit allocation network. By introducing a credit allocation network, the model contamination problem caused by unreliable memory samples during the update process is resolved. Simultaneously, a novel memory selection strategy can improve the quality of memory samples.
[0012] The technical solution of this invention is as follows: A single-target tracking method based on a credit allocation network, comprising the following steps:
[0013] Step 1: Generate a certain number of positive and negative samples based on the target location information given in the initial frame, initialize the credit allocation network using the guided focusing loss function, and put the initial frame into the memory pool as the first target memory sample.
[0014] Step 2: Use a pre-trained GoogleNet network to extract the depth features of all memory samples in the memory pool, and then concatenate them along the channel domain to obtain the memory features.
[0015] Step 3: Read the next frame image as the current frame to be tracked, and crop the current frame image based on the target position information in the previous frame image. Then input it into the GoogleNet network to obtain the current frame features.
[0016] Step 4: Input the memory features and current frame features into the spatiotemporal memory network, and use the memory frames to query the target's location information in the current frame to obtain the location information feature map.
[0017] Step 5: Use a single convolutional network to read the location information feature map of the current frame to generate classification, centrality, and regression response maps to predict the target's position in the current frame.
[0018] Step 6: Input the current frame image and the predicted target location information into the credit allocation network to obtain the credit score S of the current frame prediction result.
[0019] Step 7: Based on the credit score S, generate a certain number of positive and negative samples using the current frame and update the credit allocation network online using the guided focusing loss function.
[0020] Step 8: Update the memory samples in the memory pool based on the credit score of the prediction results for each historical frame.
[0021] Step 9: Repeat steps 2 through 7 until the video sequence has been traversed to complete target tracking.
[0022] Furthermore, in step one, the credit allocation network consists of three convolutional layers, two fully connected layers, and a final binary classification layer. The convolutional layers use parameters fixed from offline training, while the fully connected and binary classification layers are initialized with 500 positive samples and 2000 negative samples.
[0023] Furthermore, in step one, the guided focusing loss function can be described as:
[0024]
[0025] Where P y The output is the predicted value; y = 1 represents a positive sample; t is the number of iterations; λ is the initial focusing factor.
[0026] Furthermore, in step four, the location information feature map acquisition process is as follows:
[0027] S4.1 Calculate memory feature f m and current frame features f c The similarity between each pixel is obtained by calculating the similarity matrix Λ, and each element of the similarity matrix Λ is evaluated as follows:
[0028]
[0029] Where i is the index of each pixel in the memory feature map, and j is the index of each pixel in the current frame feature map. This represents the dot product of vectors. To prevent the value from being too large, we set a scaling factor. , where C is the dimension of the feature map.
[0030] S4.2 then normalizes the similarity matrix Λ using the softmax function, and adaptively reads the target information stored therein by multiplying the normalized similarity matrix with the memory features. Finally, the current frame features and the readout features are concatenated along the channel domain to obtain the location information feature map:
[0031] f l =concat[(f m )T×softmax(Λ), f c ]
[0032] in(·) T For matrix transpose, concat(·,·) represents concatenation along the channel dimension.
[0033] Furthermore, in step eight, the credit scores of all historical frame prediction results are divided into five intervals, and the frame with the highest score and greater than 0 in each interval is selected as a memory sample and placed into the memory pool. The sampling method for the memory samples can be formally described as follows:
[0034]
[0035] in, Let i be the sequence index of the memory sample in the historical frame, where i ∈ {1, 2, 3, 4, 5}.
[0036] Another objective of this invention is to provide a single-target tracking system based on a credit allocation network.
[0037] The system includes a sample labeling module for labeling objects in the image based on target location information in the initial frame; a credit allocation network module, consisting of three convolutional layers, two fully connected layers, and a binary classification layer, for evaluating the credit score of the target in the image; a loss function module for implementing a guided focusing loss function to achieve fast target localization; and a feature extraction module that uses a pre-trained GoogleNet network to perform deep feature extraction from the current frame and memory features.
[0038] The system further includes a similarity matrix generation module for generating a similarity matrix based on the dot product between the memory features and the current frame features for each pixel.
[0039] An information feature map module is used to generate a location information feature map by multiplying a similarity matrix with a memory feature.
[0040] A credit score segmentation module is used to segment the credit score based on the predicted results of historical frames.
[0041] The system further includes a memory pool module for storing prediction results for historical frames.
[0042] A sorting module for sorting memory samples based on their sequence index in the history frame.
[0043] An update module is used to update the memory samples in the memory pool, ensuring that they are always the most representative and relevant.
[0044] The system describes in detail the main components and functions of a single-target tracking system based on a credit allocation network, as well as its subsystems and modules.
[0045] The beneficial effects of this invention are:
[0046] First, this invention uses a credit allocation network to evaluate the tracking status, which provides a basis for updating the target memory samples and prevents background information from contaminating the target appearance model during the tracking process. Simultaneously, the memory selection strategy designed in this invention selects the most reliable samples within different historical intervals, effectively improving the model's adaptability and reliability.
[0047] Second, the method of the present invention provides an effective initialization method for target tracking, ensuring the accuracy and robustness of subsequent tracking.
[0048] This design enhances the network's functionality, enabling it to provide credit scores for targets more accurately, thereby improving tracking accuracy.
[0049] The guided focusing loss function of this invention is optimized for the specific needs of target tracking, providing more accurate positioning.
[0050] This invention uses a pre-trained deep network model to extract image features more accurately, thereby providing richer information for subsequent operations.
[0051] This invention can better maintain the continuity and consistency of target features, thereby improving tracking accuracy.
[0052] This invention effectively integrates past prediction information, enabling the system to adapt to dynamically changing scenarios, thereby improving tracking stability.
[0053] This invention ensures that the samples in the memory pool are always the most representative and relevant, further improving the accuracy and stability of tracking. Attached Figure Description
[0054] To more concisely and clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.
[0055] Figure 1 This is a flowchart of the single-target tracking method based on a credit allocation network according to the present invention.
[0056] Figure 2 This is a diagram illustrating the overall framework of the credit allocation network of the present invention.
[0057] Figure 3 This is a comparison chart of the tracking accuracy and tracking success rate of this invention with other algorithms on the OTB-100 dataset. Detailed Implementation
[0058] The technical solutions of the embodiments of the present invention will be described more intuitively below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0059] like Figure 1 As shown, firstly, a certain number of positive and negative samples are generated based on the target location information given in the first frame image to initialize the credit allocation network, and the first frame is placed into the target memory pool as the target memory frame. For the frame to be tracked, historical frames are input into the credit allocation network to obtain the tracking result score for each frame, and positive and negative samples are generated online based on the score to update the credit allocation network. The guided focusing loss function used in this update process can enable the network to learn difficult samples better and enhance the discrimination ability of the credit allocation network. Then, the memory frames in the target memory pool are updated based on the scores of all historical frames using the memory selection mechanism to ensure that the samples in the memory pool are always the most representative and relevant. Subsequently, the current frame and the memory frames are input into the feature extraction network to obtain the corresponding features. The features of both are input into the spatiotemporal memory network to query the target's location information in the current frame using the memory frames to obtain the location information feature map. Finally, the single convolutional network reads the location information feature map of the current frame to generate a response map to predict the target's position in the current frame and obtain the tracking result.
[0060] Specifically, the following steps are included:
[0061] Step 1: Generate a certain number of positive and negative samples based on the target location information given in the initial frame, initialize the credit allocation network using the guided focusing loss function, and put the initial frame into the memory pool as the first target memory sample.
[0062] The credit allocation network consists of three convolutional layers, two fully connected layers, and a final binary classification layer. The convolutional layers have fixed parameters obtained through offline training, while the fully connected and binary classification layers are initialized with 500 positive samples and 2000 negative samples.
[0063] The guided focus loss function can be described as follows:
[0064]
[0065] Where P y The output is the predicted value; y = 1 represents a positive sample; t is the number of iterations; λ is the initial focusing factor.
[0066] Step 2: Use a pre-trained GoogleNet network to extract the depth features of all memory samples in the memory pool, and then concatenate them along the channel domain to obtain the memory features.
[0067] Step 3: Read the next frame image as the current frame to be tracked, and crop the current frame image based on the target position information in the previous frame image. Then input it into the GoogleNet network to obtain the current frame features.
[0068] Step 4: Input the memory features and current frame features into the spatiotemporal memory network, and use the memory frames to query the target's location information in the current frame to obtain the location information feature map.
[0069] The process of obtaining the location information feature map is as follows:
[0070] S4.1 Obtains the similarity matrix Λ by calculating the similarity between each pixel of the memory feature fm and the current frame feature fc. Each element of the similarity matrix Λ is evaluated as follows:
[0071]
[0072] Where i is the index of each pixel in the memory feature map, j is the index of each pixel in the current frame feature map, and j represents the dot product of vectors. This represents the dot product of vectors; to prevent the value from being too large, we set a scaling factor. , where C is the dimension of the feature map.
[0073] S4.2 then normalizes the similarity matrix Λ using the softmax function, and adaptively reads the target information stored therein by multiplying the normalized similarity matrix with the memory features. Finally, the current frame features and the readout features are concatenated along the channel domain to obtain the location information feature map:
[0074] f l =concat[(f m )T×softmax(Λ), f c ]
[0075] in(·) T For matrix transpose, concat(·,·) represents concatenation along the channel dimension.
[0076] Step 5: Use a single convolutional network to read the location information feature map of the current frame to generate classification, centrality, and regression response maps to predict the target's position in the current frame.
[0077] Step 6: Input the current frame image and the predicted target location information into the credit allocation network to obtain the credit score S of the current frame prediction result.
[0078] Step 7: Based on the credit score S, generate a certain number of positive and negative samples using the current frame and update the credit allocation network online using the guided focusing loss function.
[0079] Step 8: Update the memory samples in the memory pool based on the credit score of the prediction results for each historical frame.
[0080] The method for updating memory samples is as follows: Divide the credit scores of all historical frame prediction results into 5 intervals, and select the frame with the highest score and greater than 0 in each interval as a memory sample and put it into the memory pool. The sampling method can be formally described as follows:
[0081]
[0082] in, Let i be the sequence index of the memory sample in the historical frame, where i ∈ {1, 2, 3, 4, 5}.
[0083] Step 9: Repeat steps 2 through 7 until the video sequence has been traversed to complete target tracking.
[0084] The method proposed in this paper is implemented based on PyTorch. The experiment was conducted on a PC with 16.0GB of memory, an Intel(R) Core(TM) i7-10700 (2.90GHz) CPU, and an NVIDIA GeForce GTX 1660 SUPER GPU. Figure 3 As shown, the single-target tracking method (Ours) based on credit allocation network provided in this invention is compared with tracking algorithms such as (STMTracker, SiamFC++, Ocean, ATOM, SiamFC, Staple) on the OTB-100 dataset. The results show that the algorithm provided in this invention has better performance in terms of success rate and accuracy.
[0085] like Figure 2 As shown, the credit allocation network consists of three convolutional layers, two fully connected layers, and a final binary classification layer. The convolutional layers use fixed parameters from offline training, while the fully connected and binary classification layers are initialized with 500 positive samples and 2000 negative samples. During tracking, the tracking result of each frame is input into the credit allocation network to obtain a tracking score for that frame. Based on the score, a certain number of positive and negative samples are collected in that frame, and the fully connected layers are updated online using a guided focusing loss function. This allows the network to learn more difficult samples, enhancing its discriminative performance. The output score of this network provides a basis for updating the memory pool, ensuring that the samples in the memory pool are always the most representative and relevant.
[0086] Example 1: Target tracking in surveillance video.
[0087] In surveillance video, the movement trajectory and behavioral patterns of targets can provide important information for security analysis. Using the proposed single-target tracking method based on a credit allocation network, targets can be tracked more accurately.
[0088] 1. The target position in the initial frame is determined by manual labeling or automatic detection, and positive and negative samples are generated for network initialization.
[0089] 2. Use pre-trained GoogleNet to extract the deep features of all memory samples in the memory pool, and then concatenate them along the channel domain to obtain the memory features.
[0090] 3. Crop the current frame image based on the target location information of the previous frame and input it into GoogleNet to obtain the current frame features.
[0091] 4. Input the memory features and current frame features into the spatiotemporal memory network to obtain the location information feature map.
[0092] 5. Use a single convolutional network to read the location information feature map of the current frame to generate classification, centrality and regression response maps to predict the target's position in the current frame.
[0093] 6. Process each frame in the surveillance video sequentially until the video sequence has been traversed to complete target tracking.
[0094] Example 2: Target tracking for autonomous vehicles.
[0095] In autonomous driving systems, accurate tracking of surrounding vehicles, pedestrians, and other targets is crucial. The proposed single-target tracking method based on a credit allocation network can be used to improve the accuracy and real-time performance of target tracking.
[0096] 1. The target position in the initial frame is automatically detected and determined by the vehicle's sensor system (such as radar, lidar, camera, etc.), and positive and negative samples are generated for network initialization.
[0097] 2. Use pre-trained GoogleNet to extract the deep features of all memory samples in the memory pool, and then concatenate them along the channel domain to obtain the memory features.
[0098] 3. Crop the current frame image based on the target location information of the previous frame and input it into GoogleNet to obtain the current frame features.
[0099] 4. Input the memory features and current frame features into the spatiotemporal memory network to obtain the location information feature map.
[0100] 5. Use a single convolutional network to read the location information feature map of the current frame to generate classification, centrality and regression response maps to predict the target's position in the current frame.
[0101] 6. The autonomous driving system makes decisions based on the predicted target location information, such as adjusting the vehicle's driving path.
[0102] 7. Process each frame sequentially until the video sequence has been traversed to complete target tracking.
[0103] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A single-target tracking method based on a credit allocation network, characterized in that the method first uses the position information of the target in the initial frame to label the objects in the image, collects sample sets that are similar to and dissimilar to the target, and then initializes the credit allocation network by guiding the focusing loss function; at the same time, the target in the initial frame is stored in the memory pool as the first target memory sample. The credit allocation network consists of three convolutional layers, two fully connected layers, and one binary classification layer; among them, The convolutional layers have fixed parameters obtained through offline training, while the fully connected layers and binary classification layers are dynamically adjusted based on the positive and negative samples collected during initialization. When performing target tracking, a guided focus loss function is used to achieve rapid target localization. This loss function integrates target information from the current frame and historical frames and is adjusted according to the set number of iterations and the initial focus factor to ensure tracking accuracy. Includes the following steps: Step 1: Generate a certain number of positive and negative samples based on the target location information given in the initial frame, initialize the credit allocation network using the guided focusing loss function, and put the initial frame into the memory pool as the first target memory sample. Step 2: Use a pre-trained GoogleNet network to extract the deep features of all memory samples in the memory pool, and then concatenate them along the channel domain to obtain the memory features; Step 3: Read the next frame image as the current frame to be tracked, and crop the current frame image according to the target position information in the previous frame image. Then input it into the GoogleNet network to obtain the current frame features. Step 4: Input the memory features and current frame features into the spatiotemporal memory network, and use the memory frames to query the target's location information in the current frame to obtain the location information feature map; Step 5: Use a single convolutional network to read the location information feature map of the current frame to generate classification, centrality, and regression response maps to predict the target's position in the current frame; Step 6: Input the current frame image and the predicted target location information into the credit allocation network to obtain the credit score S of the current frame prediction result; Step 7: Based on the credit score S, generate a certain number of positive and negative samples using the current frame and update the credit allocation network online using the guided focusing loss function; Step 8: Update the memory samples in the memory pool based on the credit score of the prediction results for each historical frame. Step 9: Repeat steps 2 through 7 until the video sequence has been traversed to complete target tracking; In step one, the credit allocation network consists of three convolutional layers, two fully connected layers, and a final binary classification layer; the convolutional layers are fixed using parameters trained offline, while the fully connected layers and the binary classification layer are initialized with 500 positive samples and 2000 negative samples. In step one, the guided focus loss function can be described as: Where P y This represents the predicted output; y = 1 represents a positive sample; t is the number of iterations; λ is the initial focusing factor. In step four, the location information feature map acquisition process is as follows: S4.1 Calculate memory feature f m and current frame features f c The similarity between each pixel is obtained by calculating the similarity matrix Λ, and each element of the similarity matrix Λ is evaluated as follows: Where i is the index of each pixel in the memory feature map, and j is the index of each pixel in the current frame feature map. This represents the dot product of vectors; to prevent the value from being too large, we set a scaling factor. Where C is the dimension of the feature map; S4.2 then uses the softmax function to normalize the similarity matrix Λ, and multiplies the normalized similarity matrix with the memory features to adaptively read the target information stored therein; finally, the current frame features and the readout features are concatenated along the channel domain to obtain the location information feature map: f l =concat[(f m )T×softmax(Λ),f c ] in(·) T For matrix transpose, concat(·,·) represents concatenation along the channel dimension; In step eight, the credit scores of all historical frame prediction results are divided into five intervals. The frame with the highest score and greater than 0 in each interval is selected as a memory sample and placed into the memory pool. The sampling method for the memory samples is as follows: in, Let i be the sequence index of the memory sample in the historical frame, where i ∈ {1, 2, 3, 4, 5}.
2. The method according to claim 1, characterized in that... In the process of acquiring the target location information feature map, the pre-trained GoogleNet network is first used to extract deep features from the current frame and the memory features. Then, a similarity matrix is generated by calculating the similarity between the two. After normalization, the matrix is multiplied with the memory features and concatenated with the current frame features to generate the location information feature map.
3. The method according to claim 2, characterized in that... The generation of the similarity matrix depends on the calculation of the dot product between the memory feature and the current frame feature for each pixel. To avoid the value being too large, a scaling factor is introduced to ensure that the value of the similarity matrix is within a suitable range, so as to optimize the generation of the location information feature map.
4. The method according to claim 1, characterized in that... After target tracking is completed, the method will segment the target based on the credit score of the prediction results of historical frames, and store the highest-scoring frame in each segment as a memory sample in the memory pool, thereby optimizing the contents of the memory pool in real time and ensuring its effective support for subsequent tracking.
5. The method according to claim 4, characterized in that... When selecting memory samples, they are first sorted based on their sequence index in the historical frames. Then, the sample with the highest credit score is selected based on the sorting results and stored in the memory pool, thus ensuring that the contents of the memory pool are always kept in the best state.
6. A single-target tracking system based on a credit allocation network applying the method of any one of claims 1-5, characterized in that the system comprises: A sample labeling module is used to label objects in an image based on target location information in the initial frame; A credit assignment network module, consisting of three convolutional layers, two fully connected layers and one binary classification layer, is used to evaluate the credit score of an object in an image. A loss function module is used to implement the guided focusing loss function to achieve rapid target localization; A feature extraction module that uses a pre-trained GoogleNet network to perform deep feature extraction on the current frame and memory features.
7. The single-target tracking system according to claim 6, characterized in that... The system further includes: A similarity matrix generation module is used to generate a similarity matrix based on the dot product between the memory features and the current frame features for each pixel; An information feature map module is used to generate a location information feature map by multiplying a similarity matrix with a memory feature; A credit score segmentation module is used to segment the credit score based on the predicted results of historical frames.
8. The single-target tracking system according to claim 7, characterized in that... The system further includes: A memory pool module is used to store the prediction results of historical frames; A sorting module for sorting memory samples according to their sequence index in the history frame; An update module is used to update the memory samples in the memory pool, ensuring that they are always the most representative and relevant.