A transformer-based segmented object tracking method and system
By segmenting the Siamese network and adding Transformer feature processing units, the shortcomings of existing target tracking methods in video logic organization and versatility are addressed, enabling efficient tracking of videos of different lengths.
Patent Information
- Application Number
- CN202311186115.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-14
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-09-14
AI Technical Summary
Existing target tracking methods cannot effectively organize the sequence of events in a video and have poor versatility for videos of varying lengths.
A segmented target tracking method based on Transformer is adopted. By dividing the Siamese network into short-time units and long-time units, and adding a Transformer feature processing unit after each unit, the network is trained by combining data preprocessing and a specific loss function to improve the video logic sorting ability.
It improves the adaptability and accuracy of the target tracking algorithm on videos of different lengths, enhances the understanding of the sequence logic of videos, and achieves more efficient target tracking.
Smart Images
Figure CN117197188B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of computer vision, deep learning technology and target tracking, and particularly relates to a segmented target tracking method and system based on Transformer. BACKGROUND
[0002] Target tracking is one of the popular research directions of computer vision, and especially with the rapid development of technologies such as unmanned driving, quite a few high-difficulty application scenarios are putting increasingly severe tests on the algorithms of this direction. Target tracking generally draws a target trajectory according to the real target frame of the first frame, and feature extraction of the target and even the background is a quite important step. Past algorithms generally rely on convolutional neural networks to realize feature extraction, but such structure is powerless to sort out the logic of video, which is not conducive to the algorithm to refer to historical data to calculate the target direction. On the other hand, too deep network level may lead to the decline of the discrimination ability of the algorithm, and the error rate increases when processing short videos. SUMMARY
[0003] The present application aims to provide a segmented target tracking method and system based on Transformer, to solve the technical problems that the existing target tracking method is powerless to sort out the logic, and is not good at generalizing to videos of different lengths.
[0004] To solve the above technical problems, the specific technical solutions of the present application are as follows:
[0005] A segmented target tracking method based on Transformer, comprising the following steps:
[0006] Step 1, data preprocessing step: rotating the training set data and changing its light and dark degree, and unifying the input size of the test set data;
[0007] Step 2, building a twin network and performing segmented training, comprising the following sub-steps:
[0008] Step 2.1, divide the twin network into short-time unit and long-time unit, and add a Transformer feature processing unit after each unit;
[0009] Step 2.2, set the network parameters, first train the short-time unit, and update the template of this unit at a high frequency;
[0010] Step 2.3, on the basis of step 2.2, simultaneously train the long-time unit and the short-time unit, and update the template of the long-time unit at a low frequency;
[0011] Step 3, input the image to be tracked, obtain the template and search image therefrom, and extract the features of the two, calculate the response map by the network and calculate the target position.
[0012] Further, the data preprocessing step in step 1 is specifically as follows:
[0013] For the training set, the image is maintained as is with a probability prob, and in other cases it is horizontally flipped, and the lightness of the image is linearly randomly changed within the scaleLow~scaleUp times of the original image attribute;
[0014] For the test set, the image is scaled and cropped in combination with the target position, so that the length and width of the template are both templateSize pixels, and the length and width of the search graph are both searchSize pixels.
[0015] Further, the network structure in step 2.1 is designed, specifically including the following steps:
[0016] Step 2.1.1, the first three convolution units derived from the AlexNet network are defined as short-time units, and the last two convolution units are defined as long-time units. The short-time units perform well in processing short videos, and the long-time units are suitable for processing long videos.
[0017] Step 2.1.2, construct a Transformer feature processing unit, hereinafter referred to as a processing unit, arrange it after the long-time unit and the short-time unit, supplement the video's logic of before and after to the features, and improve the accuracy of predicting the target region.
[0018] Further, in step 2.2, the network parameters are set, the short-time unit is trained first, and the template of this unit is updated at a high frequency, specifically including the following steps:
[0019] Step 2.2.1, the necessary settings of the network hyperparameters are as follows: first set the learning rate to lr, select the Adam optimizer, and train for epoch rounds.
[0020] Step 2.2.2, the loss function of the short-time unit is:
[0021]
[0022] Where N is the total number of samples, f and f' are the corresponding functions on both sides of the network, x i a is the i-th sample, x i p and x i n are the i-th positive sample and negative sample, α is a threshold value artificially specified, the + in the lower right corner of the square bracket represents taking the larger value between the value in the square bracket and 0, to ensure that the result is non-negative; the template update frequency during training is freqS.
[0023] Further, the long-term unit and the short-term unit are trained synchronously in step 2.3, and the template of the long-term unit is updated at a low frequency, specifically comprising the following steps:
[0024] The loss function of the long-term unit is:
[0025]
[0026] Wherein, D is the score map derived from the sample, u is a specific point of the score map, w represents the label of u, v is the score of u, and l represents the logistic regression operation; the long-term unit and the short-term unit will be trained together, the template update frequency of the short-term unit is fixed as freqS, the parameter of the long-term unit is freqL, and freqS<freqL.
[0027] Further, in step 3, the image to be tracked is input, and the response map is calculated by the network to calculate the target position, specifically comprising the following steps:
[0028] Step 301, the template and the search map obtained after processing the initial frame and other frames are taken as the input of the network;
[0029] Step 302, the features of the template and the search map are extracted by the network trained in claim 5 to obtain the processed template features and search map features;
[0030] Step 303, the Euclidean distance and convolution are calculated for the features output by the short-term unit and the long-term unit, respectively;
[0031] Step 304, the response map is obtained, the peak area in the response map is extracted and mapped to the tracking original image, so as to define the bounding box of the target, and one round of tracking is completed.
[0032] The application also discloses a segmented target tracking system based on the Transformer, which is characterized by comprising a data preprocessing unit, a segmented training unit, a feature extraction unit and a data post-processing unit.
[0033] The data preprocessing unit is used for performing the data preprocessing steps of rotation, brightness change, scaling and cropping on the training set and the image to be tracked.
[0034] The segmented training unit is used for improving the adaptability of the network to videos of different lengths, and is configured to perform the following steps:
[0035] Step A, the first three convolution units are taken as the short-term unit, and the last two convolution units are taken as the long-term unit;
[0036] Step B, the network hyperparameters and the loss function are set, the short-term unit is trained first, and then the whole network is trained;
[0037] The feature extraction unit is configured to perform the following steps:
[0038] Step A, the short-time unit is responsible for extracting shallow features, and the long-time unit is responsible for extracting deep features;
[0039] Step B, the long-short time unit is each followed by a Transformer feature processing unit, which is used to supplement the video sequence logic to the features;
[0040] The data post-processing unit is configured to perform the following steps:
[0041] Step A, calculate the Euclidean distance of the processed features of the short-time unit;
[0042] Step B, calculate the convolution of the processed features of the long-time unit;
[0043] Step C, superimpose the calculation results of step A and step B in this unit to obtain the response value of each point, find the peak value region, and map it to the target region.
[0044] The Transformer-based segmented target tracking method and system has the following advantages:
[0045] 1. The target tracking method and system are suitable for various application scenarios and can be applied in many data sets and engineering projects;
[0046] 2. The Transformer feature processing unit designed in the application enhances the ability of the algorithm to analyze the sequence logic in the video by using the Transformer, so that the past data can more effectively guide the tracking process;
[0047] 3. The application designs a segmented training mechanism, which divides the network into two units according to the depth, and adopts a targeted calculation method for the output features of the two units, so that the algorithm can be responsible for tracking videos of different lengths;
[0048] 4. The application adopts good preprocessing steps, training and tracking strategies, sets network hyperparameters, and defines flexible loss functions, which improve the training effect and efficiency of the network. BRIEF DESCRIPTION OF DRAWINGS
[0049] Figure 1 The figure is a schematic diagram of the Transformer-based segmented target tracking method of the application.
[0050] Figure 2 The figure is an embodiment effect diagram of the application. DETAILED DESCRIPTION
[0051] In order to better understand the purpose, structure and function of the present application, the present application of a segmented target tracking method and system based on Transformer will be further described in detail below in combination with the drawings.
[0052] The present application first proposes a segmented target tracking method based on Transformer, as shown in the reference Figure 1 The method comprises the following steps:
[0053] A segmented target tracking method based on Transformer comprises the following steps:
[0054] Step 1, data preprocessing step: rotate the training set data and change its light and dark degree, and unify the input size of the test set data;
[0055] For the training set, the original image is maintained with a probability prob, and the rest is horizontally flipped, and the light and dark degree of the image is linearly randomly changed in the scaleLow~scaleUp times of the original image attribute, and in specific embodiments, the above parameters can be set as: prob=0.7, scaleLow=0.8, scaleup=1.2;
[0056] For the test set, the image is scaled and cropped in combination with the target position, so that the length and width of the template are both templateSize pixels, and the length and width of the search graph are both searchSize pixels, and in specific embodiments, the above parameters can be set as: templateSize=127, searchSize=255.
[0057] Step 2, build a twin network and perform segmented training, including the following sub-steps:
[0058] Step 2.1, divide the twin network into short-time units and long-time units, and add a Transformer feature processing unit after the two units, specifically:
[0059] Step 201, the first three convolution units derived from the AlexNet network are defined as short-time units, and the last two convolution units are defined as long-time units, the short-time units perform well in processing short videos, and the long-time units are suitable for processing long videos;
[0060] Step 202, construct a Transformer feature processing unit, arrange it after the long-time unit and the short-time unit, supplement the video sequence logic to the features, and improve the accuracy of the predicted target area.
[0061] Step 2.2, set the network parameters, first train the short-time unit, and update the template of this unit at a high frequency, specifically:
[0062] Step 203, the necessary settings of network hyperparameters are as follows: first, set the learning rate to lr, and select the Adam optimizer, train for epoch rounds, in specific embodiments, the above parameters can be set as: lr = 10 -2 , epoch = 50;
[0063] Step 204, the loss function of the short-term unit is:
[0064]
[0065] where N is the total number of samples, f and f' are the corresponding functions on both sides of the network, x i a is the i-th sample, x i p and x i n are the i-th positive and negative samples, and a is a threshold value artificially set, the + in the lower right corner of the square bracket represents taking the larger value between the value in the square bracket and 0, to ensure that the result is non-negative; the template update frequency during training is freqS, in specific embodiments, the above parameters can be set as: freqS = 0.2.
[0066] Step 2.3, on the basis of step 2.2, the long-term unit and the short-term unit are trained synchronously, and the template of the long-term unit is updated at a low frequency, specifically:
[0067] The loss function of the long-term unit is:
[0068]
[0069] where D is a score map derived from the sample, u is a specific point of the score map, w represents the label of u, v is the score of u, and l represents the logistic regression operation; the long-term unit and the short-term unit will participate in training together, the template update frequency of the short-term unit is fixed as freqS, and the parameter of the long-term unit is freqL, and freqS < freqL, in specific embodiments, the above parameters can be set as: freqS = 0.2, freqL = 0.0005.
[0070] Step 3, step 3, input the image to be tracked, obtain the template and the search image therefrom, and extract the features of the two, calculate the response map by the network and calculate the target position, specifically including the following sub-steps:
[0071] Step 3.1, process the initial frame and other frames to obtain the template and the search image, as the input of the network;
[0072] Step 3.2, extract the features of the template and the search image by the trained network to obtain the processed template features and search image features;
[0073] Step 3.3, for the features output by the short-time unit and the long-time unit, respectively, calculate the Euclidean distance and the convolution;
[0074] Step 3.4, derive the response map, extract the peak area in the response map and map it to the tracking original map, so as to delimit the bounding box of the target, and complete a round of tracking.
[0075] The application further discloses a segmented target tracking system based on the Transformer, which is characterized by comprising a data preprocessing unit, a segmented training unit, a feature extraction unit and a data post-processing unit.
[0076] The data preprocessing unit is used for performing data preprocessing steps of rotation, brightness change, scaling and cropping on the training set and the to-be-tracked image.
[0077] The segmented training unit is configured to perform the following steps:
[0078] Step A, the first three convolution units are used as short-time units, and the last two convolution units are used as long-time units;
[0079] Step B, network hyperparameters and loss functions are set, and the short-time unit is trained first, and then the whole network is trained;
[0080] The feature extraction unit is configured to perform the following steps:
[0081] Step A, the short-time unit is responsible for extracting shallow features, and the long-time unit is responsible for extracting deep features;
[0082] Step B, the long-time unit and the short-time unit are each followed by a Transformer feature processing unit, which is used to supplement the video logic to the features;
[0083] The data post-processing unit is configured to perform the following steps:
[0084] Step A, the processed features of the short-time unit are used to calculate the Euclidean distance;
[0085] Step B, the processed features of the long-time unit are used to calculate the convolution;
[0086] Step C, the calculation results of step A and step B in the unit are superimposed to obtain the response value of each point, and the peak area is found out, and the target area is mapped.
[0087] The application adopts an end-to-end overall framework to realize target tracking, and after an image is input into a video, corresponding tracking results can be obtained, Figure 2is a set of embodiment effect pictures, wherein the green frame is the tracking result of the present application, the red frame is the real range of the target, both are close to each other, and the effect is satisfactory.
[0088] Those skilled in the art can understand that each block in the structural diagram and / or block diagram and / or flow diagram and the combination of the blocks in the structural diagram and / or block diagram and / or flow diagram can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a professional computer or a processor of other programmable data processing method to generate a machine, so that the instructions executed by the computer or the processor of other programmable data processing method create a method specified in the block or blocks of the structural diagram and / or block diagram and / or flow diagram for implementing the structural diagram and / or block diagram and / or flow diagram.
[0089] It can be understood that the present application is described by some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, modifications can be made to these features and embodiments to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.
Claims
1. A segment-based object tracking method based on Transformer, characterized in that, The method comprises the following steps: Step 1, data preprocessing step: rotating the training set data and changing its brightness, and unifying the input size of the test set data; Step 2, building a twin network and performing segmented training, comprising the following sub-steps: Step 2.1, designing a network structure, dividing the twin network into a short-time unit and a long-time unit, and adding a Transformer feature processing unit after the two units; Step 2.2, setting network parameters, first training the short-time unit, and updating the template of the unit at a high frequency; Step 2.3, on the basis of step 2.2, synchronously training the long-time unit and the short-time unit, and updating the template of the long-time unit at a low frequency; Step 3, inputting the image to be tracked, obtaining the template and the search image therefrom, and extracting the features of the two, calculating the response map by the network and calculating the target position; The data preprocessing step in step 1 is specifically as follows: For the training set, the image is maintained as original with a probability prob, and in other cases, the image is horizontally flipped, and the brightness of the image is linearly randomly changed within the scaleLow~scaleUp times of the original image attribute; For the test set, the image is scaled and cropped in combination with the target position, so that the length and width of the template are both templateSize pixels, and the length and width of the search image are both searchSize pixels; The network structure in step 2.1 is specifically designed as follows: Step 2.1.1, the first three convolution units derived from the AlexNet network are defined as the short-time unit, and the last two convolution units are defined as the long-time unit; Step 2.1.2, constructing a Transformer feature processing unit, arranging it after the long-time unit and the short-time unit, supplementing the video sequence logic to the features, and improving the accuracy of predicting the target region; The network parameters in step 2.2 include hyperparameters and loss functions, and step 2.2 specifically comprises the following steps: Step 2.2.1, the network hyperparameters are set as follows: first set the learning rate to lr, and select the Adam optimizer, and train for epoch rounds; The loss function of the short-time unit in step 2.2 is: where N is the total number of samples, f and f' are the corresponding functions on both sides of the network, x i a is the i-th sample, x i p is the i-th positive sample, x i n is the i-th negative sample, a is a threshold value artificially set, the + in the lower right corner of the square bracket represents taking the larger value between the value in the square bracket and 0, ensuring that the result is non-negative; the template update frequency during training is freqS.
2. The Transformer-based fragmented target tracking method according to claim 1, wherein, The synchronous training of the long-time unit and the short-time unit in step 2.3 is specifically as follows: The loss function of the long-time unit is: Where D is a score map derived from a sample, u is a specific point of the score map, w represents the label of u, v is the score of u, and l represents a logistic regression operation; the long-time unit and the short-time unit will be trained together, the template update frequency of the short-time unit is fixed as freqS, the parameter of the long-time unit is freqL, and freqS<freqL.
3. The Transformer-based fragmented target tracking method according to claim 2, wherein, Step 3 inputs the image to be tracked, calculates the response map by the network, and calculates the target position, comprising the following steps: Step 301, obtaining the template and the search image after processing the initial frame and other frames, as the input of the network; Step 302, extracting the features of the template and the search image by the network trained in step 2.3 to obtain the processed template features and search image features; Step 303, calculating the Euclidean distance and convolution for the features output by the short-time unit and the long-time unit, respectively; Step 304, obtain the response map, extract the peak region in the response map and map it to the tracking original map, so as to delimit the boundary box of the target and complete one round of tracking.
4. The Transformer-based fragmented object tracking system according to any one of claims 1-3, wherein, The data preprocessing unit, the segmented training unit, the feature extraction unit and the data post-processing unit are included. The data preprocessing unit is configured to perform rotation, brightness change, scaling and cropping data preprocessing steps on the training set and the image to be tracked. The segmented training unit is configured to improve the adaptability of the network to videos of different lengths and perform the following steps: Step A, the first three convolution units are short-time units, and the last two convolution units are long-time units; Step B, set the network hyperparameters and loss function, train the short-time units first, and then train the whole network; The feature extraction unit is configured to extract multi-level features of the image and perform the following steps: Step A, the short-time unit is responsible for extracting shallow features, and the long-time unit is responsible for extracting deep features; Step B, the long-time unit and the short-time unit are each followed by a Transformer feature processing unit for supplementing the sequence logic of the video to the features; The data post-processing unit is configured to perform the following steps: Step A, calculate the Euclidean distance of the processed features of the short-time unit; Step B, calculate the convolution of the processed features of the long-time unit; Step C, superimpose the calculation results of step A and step B to obtain the response value of each point, find the peak region, and map it to the target region.