Target tracking method, system, terminal and storage medium

Abstract

The application provides a target tracking method, system, terminal and storage medium, the method comprises: obtaining the template feature of a to-be-tracked video and the search feature of each frame image, performing cross-correlation processing on the template feature and the search feature to obtain cross-correlation features, performing feature point classification, center point analysis and feature deviation analysis on the cross-correlation features to obtain a classification feature map, a center degree score map and a deviation coordinate map; determining a target frame center point according to the classification feature map and the center degree score map, and generating a tracking result according to the target frame center point and the deviation coordinate map. The classification feature map can represent the classification result of the foreground and the background in the cross-correlation features, the center degree score map can represent the probability that each pixel point in the cross-correlation features belongs to the best center, and the deviation coordinate map can represent the deviation between each pixel point in the cross-correlation features and the target frame, thereby improving the accuracy of the tracking result generated based on the target frame center point and the deviation coordinate map of each frame image.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a target tracking method, system, terminal, and storage medium. Background Technology

[0002] Visual object tracking is a hot topic in computer vision research, and the technology has made significant progress over the past two to three decades. With the development of artificial intelligence, autonomous driving technology has rapidly become a focus, and visual object tracking technology has irreplaceable applications. Visual object tracking technology estimates the position and shape of objects in an image sequence, uses the obtained motion information to analyze the object's behavior, and then performs more advanced tasks. Research on applications in autonomous driving is often limited by the limited computing resources of onboard systems. Although object tracking techniques based on discriminant correlation filters can be applied in real-time with limited resources, these methods have complex optimization strategies and are not robust enough in real-world environments. Deep learning-based object tracking technology can strike a good balance between efficiency and accuracy, making it a good choice for applying visual object tracking technology to autonomous driving.

[0003] In recent years, the industry has proposed many deep learning-based target tracking technologies for autonomous driving, especially tracking methods based on Siamese network structures. However, in the face of uncertainties such as resolution, occlusion, illumination and object movement in real environments, the accuracy of existing target tracking methods is relatively low. Summary of the Invention

[0004] The purpose of this invention is to provide a target tracking method, system, terminal, and storage medium, aiming to solve the problem of low accuracy in existing target tracking methods.

[0005] The present invention is implemented as follows: a target tracking method, the method comprising:

[0006] Acquire the video to be tracked, and obtain the template features of the video to be tracked and the search features of each frame of the video to be tracked;

[0007] The template features and the search features are cross-correlated to obtain cross-correlated features. Feature point classification, centroid analysis and feature deviation analysis are then performed on the cross-correlated features to obtain a classification feature map, a centroid score map and a deviation coordinate map.

[0008] Based on the classification feature map and the centrality score map, the center point of the target box in each frame image is determined, and the tracking result of the video to be tracked is generated based on the center point of the target box in each frame image and the deviation coordinate map.

[0009] Furthermore, obtaining the template features of the video to be tracked and the search features of each frame in the video to be tracked includes:

[0010] The network is trained and the initial parameters are iteratively updated to adjust the deviations, thus obtaining the feature extraction network.

[0011] The feature extraction network is used to extract features from each frame of the image to obtain the template features and the search sub-features. The template features and the search sub-features each include a first preset feature, a second preset feature and a third preset feature.

[0012] The search feature is obtained by concatenating the template feature and the first preset feature, the second preset feature and the third preset feature in the search sub-feature.

[0013] Furthermore, the step of concatenating the template features and the first, second, and third preset features from the search sub-features to obtain the search features includes:

[0014] For the first preset feature, the features of the first preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the first concatenated feature;

[0015] For the second preset feature, the features of the second preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the second concatenated feature;

[0016] For the third preset feature, the features of the third preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the third concatenated feature;

[0017] The first splicing feature, the second splicing feature, and the third splicing feature are spliced ​​together to obtain a fourth splicing feature, and the fourth splicing feature is convolved to obtain a convolutional feature.

[0018] The convolutional features are subjected to cross-attention processing, and the cross-attention processed convolutional features are combined with the first preset features to obtain the search features.

[0019] Furthermore, the step of performing feature point classification, centroid analysis, and feature deviation analysis on the cross-correlation features to obtain a classification feature map, a centrality score map, and a deviation coordinate map includes:

[0020] Each pixel in the cross-correlation feature is classified into foreground and background to obtain the classification feature map. The binary classification is used to classify each pixel into foreground and background. The classification feature map includes the classification results of foreground and background for each pixel.

[0021] Calculate the score of each pixel in the cross-correlation feature as the optimal center to obtain the centrality score map;

[0022] The deviations between each pixel in the cross-correlation feature and the left, top, right, and bottom edges of the target box are calculated to obtain the deviation coordinate map.

[0023] Furthermore, determining the center point of the target bounding box for each frame image based on the classification feature map and the centrality score map includes:

[0024] A cosine window penalty is applied to the centrality score map to obtain a penalized score map, and the foreground value of each pixel is extracted from the classification feature map.

[0025] If the foreground value of any pixel is less than the background value, then the pixel is set to 0;

[0026] Calculate the product between the foreground value of each pixel and the penalty score map, and determine the pixel with the largest product value as the center point of the target box.

[0027] Furthermore, generating the tracking result of the video to be tracked based on the center point of the target bounding box in each frame and the deviation coordinate map includes:

[0028] Based on the position of the center point of the target box, obtain the border deviation value between the center point of the target box and the border in the deviation coordinate diagram;

[0029] Based on the bounding box deviation value and the position of the target box center point, the predicted bounding box of each frame image is determined respectively;

[0030] The deviation between the predicted bounding box of each frame image and the target bounding box in the previous frame image is calculated to obtain the bounding box deviation value. The product between the bounding box deviation value and the update ratio value of each frame image is calculated to obtain the update value. The bounding box deviation value is used to characterize the difference between the results of the previous and next frames.

[0031] The updated values ​​of each frame image are superimposed with the target bounding box in the previous frame image to obtain the tracking results of each frame image, and the tracking results of the video to be tracked are generated based on the tracking results of each frame image.

[0032] Furthermore, before performing cross-correlation processing on the template features and the search features to obtain cross-correlation features, the method further includes:

[0033] The tracking result of the previous frame image is obtained, and the feature is cropped based on the tracking result of the previous frame image to obtain the regression feature;

[0034] The template features and regression features of the previous frame image are obtained and concatenated to obtain the fifth concatenated feature, and the fifth concatenated feature is then subjected to convolution processing.

[0035] The fifth stitched feature after convolution processing is enhanced by an attention mechanism, and the enhanced fifth stitched feature is superimposed with the template feature of the initial frame image in the video to be tracked to obtain the superimposed feature;

[0036] The template features of the current frame image are updated based on the superimposed features.

[0037] Another objective of this invention is to provide a target tracking system, the system comprising:

[0038] The feature acquisition module is used to acquire the video to be tracked, and to acquire the template features of the video to be tracked and the search features of each frame of the video to be tracked;

[0039] The cross-correlation module is used to perform cross-correlation processing on the template features and the search features to obtain cross-correlation features, and to perform feature point classification, centroid analysis and feature deviation analysis on the cross-correlation features to obtain a classification feature map, a centroid score map and a deviation coordinate map.

[0040] The tracking result generation module is used to determine the center point of the target box in each frame image based on the classification feature map and the centrality score map, and to generate the tracking result of the video to be tracked based on the center point of the target box in each frame image and the deviation coordinate map.

[0041] Another objective of this invention is to provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0042] Another objective of this invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0043] In this embodiment of the invention, cross-correlation processing is performed on template features and search features to obtain cross-correlation features, thereby improving the correlation between the features of the template features and search features. By performing feature point classification, center point analysis, and feature deviation analysis on the cross-correlation features respectively, a classification feature map, a centrality score map, and a deviation coordinate map are obtained. The classification feature map can effectively represent the classification results of foreground and background in the cross-correlation features. The centrality score map can effectively represent the probability that each pixel in the cross-correlation features belongs to the optimal center, thereby improving the accuracy of the target box center point determined in each frame of the image based on the classification feature map and the centrality score map. The deviation coordinate map can effectively represent the deviation between each pixel in the cross-correlation features and the target box, thereby improving the accuracy of the tracking results generated based on the target box center point and the deviation coordinate map of each frame of the image. Attached Figure Description

[0044] Figure 1 This is a flowchart of the target tracking method provided in the first embodiment of the present invention;

[0045] Figure 2 This is a flowchart of the target tracking method provided in the second embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the target tracking system provided in the third embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the target tracking system provided in the fourth embodiment of the present invention;

[0048] Figure 5 This is a flowchart illustrating the implementation of the feature fusion module provided in the fourth embodiment of the present invention;

[0049] Figure 6 This is an implementation flowchart of the tracking regression module provided in the fourth embodiment of the present invention;

[0050] Figure 7 This is an implementation flowchart of the template update module provided in the fourth embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the structure of the terminal device provided in the fifth embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0053] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0054] Example 1

[0055] Please see Figure 1 This is a flowchart of a target tracking method provided in the first embodiment of the present invention. This target tracking method can be applied to any terminal device or system, and includes the following steps:

[0056] Step S10: Obtain the video to be tracked, and obtain the template features of the video to be tracked and the search features of each frame of the video to be tracked;

[0057] In this process, by obtaining the template patch and the search patch for each frame in the video to be tracked, the position of the search patch for the next frame is determined by the result of the previous frame. The template patch and the search patch for each frame in the video to be tracked are respectively input into a feature extraction network for parameter setting. Based on the parameter setting, feature extraction is performed to obtain the template feature and the search feature. The feature extraction network in this step can be set according to the requirements. In this embodiment, the ResNet50 network is used.

[0058] Step S20: Perform cross-correlation processing on the template features and the search features to obtain cross-correlation features, and perform feature point classification, centroid analysis and feature deviation analysis on the cross-correlation features to obtain classification feature map, centroid score map and deviation coordinate map;

[0059] Specifically, by cross-correlation processing of template features and search features to obtain cross-correlated features, the correlation between the template features and search features is improved. In this step, feature point classification, centroid analysis, and feature deviation analysis are performed on the cross-correlated features to obtain a classification feature map, a centrality score map, and a deviation coordinate map, including:

[0060] Each pixel in the cross-correlation feature is classified into two categories to obtain the classification feature map;

[0061] The binary classification is used to classify the foreground and background of each pixel. The classification feature map includes the classification results of the foreground and background of each pixel. That is, the classification feature map can effectively represent the classification results of the foreground and background in the cross-correlation features.

[0062] Calculate the score of each pixel in the cross-correlation feature as the optimal center to obtain the centrality score map;

[0063] Among them, the centrality score map can effectively represent the probability that each pixel in the cross-correlation feature belongs to the best center, thereby improving the accuracy of the target box center points of each frame image determined based on the classification feature map and the centrality score map.

[0064] The deviations between each pixel in the cross-correlation feature and the left, top, right, and bottom edges of the target box are calculated to obtain the deviation coordinate map.

[0065] Among them, the deviation coordinate map can effectively characterize the deviation between each pixel in the cross-correlation feature and the target box;

[0066] Specifically, in this step, the cross-correlation features obtained from the cross-correlation operation are input into three sets of convolutions. Convolution group A regresses to obtain a W×H×2 feature map, which performs a binary classification of foreground and background for each point on the feature map, with each point containing two values ​​representing the classification result of foreground and background. Convolution group B regresses to obtain a W×H×1 feature map, which calculates the score of each point on the feature map belonging to the optimal center, and is used together with the classification feature map to determine the position of the optimal center target point. Convolution group C regresses to obtain a W×H×4 feature map, which represents the deviation from the target box to the left, top, right, and bottom at the current point.

[0067] Furthermore, before performing cross-correlation processing on the template features and the search features to obtain cross-correlation features, this step further includes:

[0068] The tracking result of the previous frame image is obtained, and the feature is cropped based on the tracking result of the previous frame image to obtain the regression feature;

[0069] The template features and regression features of the previous frame image are obtained and concatenated to obtain the fifth concatenated feature, and the fifth concatenated feature is then subjected to convolution processing.

[0070] The fifth stitched feature after convolution processing is enhanced by an attention mechanism, and the enhanced fifth stitched feature is superimposed with the template feature of the initial frame image in the video to be tracked to obtain the superimposed feature;

[0071] The template features of the current frame image are updated based on the superimposed features;

[0072] In this embodiment, an adaptive update step is included for template features to avoid the shortcomings of poor flexibility and inaccurate updates in early template feature update strategies. The adaptive update step includes three different feature inputs: template features from the initial frame image, updated template features from the previous frame, and features generated by cropping the tracking results from the previous frame. The previous frame template features and previous frame regression features are concatenated, and after a convolutional layer, the useful feature information is further enhanced using an ECA attention mechanism. The initial template features are then added to the ECA output through skip connections to obtain superimposed features. This ensures that the updated template features contain accurate and reliable target feature information, while also including the differences in features brought about by target motion.

[0073] Step S30: Determine the center point of the target box in each frame image based on the classification feature map and the centrality score map, and generate the tracking result of the video to be tracked based on the center point of the target box in each frame image and the deviation coordinate map.

[0074] In this step, determining the center point of the target bounding box for each frame image based on the classification feature map and the centrality score map includes:

[0075] A cosine window penalty is applied to the centrality score map to obtain a penalized score map, and the foreground value of each pixel is extracted from the classification feature map.

[0076] If the foreground value of any pixel is less than the background value, then the pixel is set to 0;

[0077] Calculate the product between the foreground value of each pixel and the penalty score map, and determine the pixel with the largest product value as the center point of the target box;

[0078] First, a cosine window penalty is applied to the center score map to obtain a penalty score map, thereby avoiding excessive displacement of the target box. Values ​​belonging to the foreground are extracted from the classification feature map, and points with foreground values ​​less than background values ​​are set to 0. The point with the largest value is obtained by multiplying it with the penalty score map, and this point is the selected center point of the target box.

[0079] Further, in this step, generating the tracking result of the video to be tracked based on the center point of the target bounding box in each frame image and the deviation coordinate map includes:

[0080] Based on the position of the center point of the target box, obtain the border deviation value between the center point of the target box and the border in the deviation coordinate diagram;

[0081] Based on the bounding box deviation value and the position of the target box center point, the predicted bounding box of each frame image is determined respectively;

[0082] The deviation between the predicted bounding box of each frame and the target bounding box in the previous frame is calculated to obtain the bounding box deviation value. The product between the bounding box deviation value and the update ratio value of each frame is calculated to obtain the update value. The bounding box deviation value is used to characterize the difference between the results of the previous and next frames.

[0083] The updated values ​​of each frame image are superimposed with the target bounding box in the previous frame image to obtain the tracking results of each frame image, and the tracking results of the video to be tracked are generated based on the tracking results of each frame image.

[0084] Specifically, based on the position of the target box center point, the distance between the target box center point and the bounding box is obtained from the deviation coordinate feature map. The maximum values ​​of the left, top, right, and bottom values ​​are limited to prevent the obtained tracking box from exceeding the search area. The box is then mapped back to the size of the original search map to obtain the predicted box for each frame.

[0085] To make the tracking results smoother, the position of the predicted bounding box in each frame is not directly used as the final result. Instead, the deviation between the predicted bounding box and the target bounding box in the previous frame is calculated, multiplied by the update ratio, and then superimposed with the target bounding box in the previous frame to obtain the final tracking result. The update ratio = predicted bounding box aspect ratio × classification foreground value × 0.5.

[0086] In this embodiment, cross-correlation processing is performed on template features and search features to obtain cross-correlation features, which improves the correlation between the features of template features and search features. By performing feature point classification, center point analysis, and feature deviation analysis on the cross-correlation features, a classification feature map, a centrality score map, and a deviation coordinate map are obtained. The classification feature map can effectively represent the classification results of foreground and background in the cross-correlation features. The centrality score map can effectively represent the probability that each pixel in the cross-correlation features belongs to the optimal center, thereby improving the accuracy of the target box center point determined based on the classification feature map and the centrality score map of each frame image. The deviation coordinate map can effectively represent the deviation between each pixel in the cross-correlation features and the target box, thereby improving the accuracy of the tracking results generated based on the target box center point and deviation coordinate map of each frame image.

[0087] Example 2

[0088] Please see Figure 2 This is a flowchart of a target tracking method provided in the second embodiment of the present invention. This embodiment further refines step S10 in the first embodiment, including the following steps:

[0089] Step S11: The initial parameters are iteratively updated continuously based on network training, and the deviation is adjusted by feedback to obtain the feature extraction network;

[0090] The feature extraction network can be configured according to requirements; in this step, the ResNet50 network is used.

[0091] Step S12: Extract features from each frame of the image according to the feature extraction network to obtain the template features and search sub-features;

[0092] The template features and search sub-features both include a first preset feature, a second preset feature, and a third preset feature, which can be set according to requirements.

[0093] Step S13: Concatenate the template features and the first preset feature, the second preset feature and the third preset feature in the search sub-features to obtain the search features;

[0094] The concatenation function can be used to concatenate the first preset feature, the second preset feature, and the third preset feature. Optionally, in this step, concatenating the template feature and the first preset feature, the second preset feature, and the third preset feature in the search sub-features to obtain the search feature includes:

[0095] For the first preset feature, features of the first preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the first concatenated feature. For the second preset feature, features of the second preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the second concatenated feature.

[0096] For the third preset feature, the features of the third preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the third concatenated feature;

[0097] The first preset percentage channel, the second preset percentage channel, and the third preset percentage channel can all be set according to requirements. In this step, the first preset percentage channel, the second preset percentage channel, and the third preset percentage channel are set to 50%, 30%, and 20%, respectively.

[0098] The first splicing feature, the second splicing feature, and the third splicing feature are spliced ​​together to obtain a fourth splicing feature, and the fourth splicing feature is convolved to obtain a convolutional feature.

[0099] Specifically, the fourth splicing feature is obtained by adding the values ​​at the same positions among the first splicing feature, the second splicing feature, and the third splicing feature, and then averaging them.

[0100] The convolutional features are subjected to cross-attention processing, and the cross-attention processed convolutional features are combined with the first preset features to obtain the search features;

[0101] Specifically, the convolutional features are input into the cross-attention module for processing. The convolutional features processed by cross-attention are then combined with the first preset features to form a residual structure, thereby obtaining the search feature.

[0102] In this embodiment, the initial parameters are iteratively updated through network training, and the deviation is adjusted by feedback to obtain the feature extraction network. Based on the feature extraction network, the extraction of template features and search sub-features is guaranteed. By concatenating the first preset feature, the second preset feature and the third preset feature in the template features and search sub-features, the search feature can be effectively obtained.

[0103] Example 3

[0104] Please see Figure 3 This is a schematic diagram of the target tracking system 100 provided in the third embodiment of the present invention, including: a feature acquisition module 10, a cross-correlation module 11, and a tracking result generation module 12, wherein:

[0105] The feature acquisition module 10 is used to acquire the video to be tracked, and to acquire the template features of the video to be tracked and the search features of each frame of the video to be tracked.

[0106] The cross-correlation module 11 is used to perform cross-correlation processing on the template features and the search features to obtain cross-correlation features, and to perform feature point classification, centroid analysis and feature deviation analysis on the cross-correlation features to obtain a classification feature map, a centroid score map and a deviation coordinate map.

[0107] The tracking result generation module 12 is used to determine the center point of the target box of each frame image based on the classification feature map and the centrality score map, and to generate the tracking result of the video to be tracked based on the center point of the target box of each frame image and the deviation coordinate map.

[0108] In this embodiment, cross-correlation processing of template features and search features is performed to obtain cross-correlation features, which improves the correlation between the features of template features and search features. By performing feature point classification, center point analysis, and feature deviation analysis on the cross-correlation features, a classification feature map, a centrality score map, and a deviation coordinate map are obtained. The classification feature map can effectively represent the classification results of foreground and background in the cross-correlation features. The centrality score map can effectively represent the probability that each pixel in the cross-correlation features belongs to the optimal center, thereby improving the accuracy of the target box center point determined in each frame of the image based on the classification feature map and the centrality score map. The deviation coordinate map can effectively represent the deviation between each pixel in the cross-correlation features and the target box, thereby improving the accuracy of the tracking results generated based on the target box center point and deviation coordinate map of each frame of the image.

[0109] Example 4

[0110] Please see Figure 4This is a schematic diagram of the target tracking system provided in the fourth embodiment of the present invention, including: a feature fusion module 13, a tracking regression module 14, and a template update module 15, wherein:

[0111] The target tracking system in this embodiment uses a ResNet50 network. It performs feature fusion on different three-layer features to obtain features corresponding to the template patch and the search patch. The two are then cross-correlated, and the cross-correlated features are input into the tracking regression module 14 to obtain a classification feature map, a centrality score map, and a deviation coordinate map. The position with the highest score is calculated based on the classification feature map and the centrality score map as the center of the target box. The tracking result is obtained by combining the deviation coordinate map. In the first frame, the features extracted by the initial module patch are directly cross-correlated. Subsequent frames are updated using the initial template patch features, the previous frame template features, and the previous frame regression features. The new template features are obtained through the template update module 15, and then cross-correlated operations are performed.

[0112] Feature fusion module 13, such as Figure 5 As shown, the feature fusion module fuses three different features output by the feature extraction network. First, it selects the first 50%, 30%, and 20% channels of each feature and concatenates them. Then, it performs a convolution operation after passing through the cross attention module (CCA). Finally, it adds the feature to the first feature to form a residual structure, thus obtaining the final fused feature. Compared with directly concatenating the three extracted features and then reducing the dimensionality, this saves computation and does not reduce the tracking accuracy.

[0113] Tracking regression module 14, such as Figure 6 As shown, the tracking regression module 14 mainly inputs the cross-correlation features obtained from the cross-correlation operation into three sets of convolutions. Convolution group A regresses to obtain a W×H×2 feature map, which performs a binary classification of foreground and background for each point on the feature map. Each point contains two values ​​representing the classification result of foreground and background. Convolution group B regresses to obtain a W×H×1 feature map, which calculates the score of each point on the feature map belonging to the optimal center. This score is used together with the classification feature map to determine the position of the optimal center target point. Convolution group C regresses to obtain a W×H×4 feature map, which represents the deviation of the current point from the left, top, right, and bottom of the target box. Finally, the target position is obtained by combining the feature map with the selected optimal center point, and the tracking result is obtained.

[0114] Template update module 15, such as Figure 7As shown, the template update module 15 is crucial for learning adaptive template feature updates, avoiding the shortcomings of poor flexibility and inaccurate updates in early template feature update strategies. The template update module 15 mainly includes three different feature inputs: the template features determined in the initial frame, the template features updated in the previous frame, and features generated by cropping the tracking results from the previous frame. The previous frame template features and the previous frame regression features are concatenated, and after a convolutional layer, the useful feature information is further enhanced using the ECA attention mechanism. The initial template features are then added to the ECA output through skip connections to obtain new template features. This ensures that the updated template features contain accurate and reliable target feature information, while also including the differences in features brought about by target motion.

[0115] In this embodiment, supervision is performed by calculating the foreground / background classification loss, centrality score loss, tracking box IOU loss, and template update loss, as detailed below:

[0116] Calculate the cross-entropy loss function on the classification feature map, where i represents a point in the feature map, p i y represents the probability that a point is predicted as the foreground. i The value represents the label at that point, with 1 for the foreground and 0 for the background. This loss is used to supervise the output classification feature map, limiting the target center point to the foreground.

[0117] Calculate the loss function for vertex deviation from the center in the centrality score map, where N represents the number of points in the feature map that belong to the foreground, i represents one of those points, and d... i C represents the distance from the point to the center of the target. i This represents the value of the point on the centrality score map. Using this loss can effectively limit the target center point from deviating significantly, thus improving the accuracy of tracking.

[0118] L reg =-ln(L IoU (A reg A gt )) Calculate the IOU loss between the regressed target box and the ground truth target box, where L IOU A represents the IOU calculation function. reg A gt These are the regressed target bounding box and the ground truth target bounding box, respectively. This loss function is used to supervise the tracking results of the model's regression.

[0119] Calculate the distance loss between the updated template features and the real template features, where φ represents the template feature update function. These are the initial template features, the previous frame template features, and the previous frame regression features, respectively. This represents accurate template features. During training, the smaller this loss value, the more accurate the template update.

[0120] Loss=αL cls +βL cen +γL reg +δL update The overall loss is calculated, where α, β, γ, and δ are the weight coefficients of each loss term. The network is then trained using gradient descent with this loss value under supervision.

[0121] For target tracking box calculation, in this embodiment, a cosine window penalty is first applied to the centrality score map to obtain a new centrality score map, thereby avoiding excessive target box displacement. Values ​​belonging to the foreground are extracted from the classification feature map. Points with foreground values ​​less than background values ​​are set to 0. The value of the foreground value is multiplied by the new centrality score map to obtain the point with the largest value; this point is the optimal target center point. Based on the point's position, the distance from the center point to the bounding box is obtained from the deviation coordinate feature map. Maximum values ​​are limited for the left, top, right, and bottom values ​​to prevent the obtained tracking box from exceeding the search area. The value is then mapped back to the original size of the search map to obtain the tracking result. To make the tracking result smoother, the tracking result is not directly used as the final result. The deviation from the target box in the previous frame is calculated, multiplied by the update ratio, and then superimposed on the target box in the previous frame to obtain the final tracking result. The update ratio = predicted box aspect ratio × classification foreground value × 0.5.

[0122] This embodiment employs a Siamese network structure to construct a target tracking method applicable to in-vehicle systems. A lightweight model structure is designed based on the Siamese network, adopting an anchor-free strategy to eliminate the complex computational burden of pre-set anchors. Even in in-vehicle systems with limited computing resources, it can still output accurate tracking results in real time. Facing complex changes in the environment and the target, it can adaptively learn and update template features, preserving both the original target features and incorporating relative feature changes in the real-time scene, greatly avoiding tracking failures caused by excessive scene changes or target deformation. The tracking results obtained through this target tracking system accurately locate the target bounding box, demonstrating high accuracy and stability, and achieving a speed of 32fps, making it suitable for further applications in autonomous driving.

[0123] Example 5

[0124] Figure 8 This is a structural block diagram of a terminal device 2 provided in the fifth embodiment of this application. For example... Figure 8As shown, the terminal device 2 in this embodiment includes a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for a target tracking method. When the processor 20 executes the computer program 22, it implements the steps in the various embodiments of the target tracking methods described above.

[0125] For example, the computer program 22 may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 20 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 22 in the terminal device 2. The terminal device may include, but is not limited to, the processor 20 and the memory 21.

[0126] The processor 20 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0127] The memory 21 can be an internal storage unit of the terminal device 2, such as a hard drive or memory of the terminal device 2. The memory 21 can also be an external storage device of the terminal device 2, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 2. Furthermore, the memory 21 can include both internal and external storage units of the terminal device 2. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0128] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If an integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer-readable storage medium can be non-volatile or volatile. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of a computer-readable storage medium may be appropriately added to or subtracted from the contents as required by the legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable storage medium may not include electrical carrier signals and telecommunication signals.

[0130] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A target tracking method, characterized in that, The method includes: Acquire the video to be tracked, and obtain the template features of the video to be tracked and the search features of each frame of the video to be tracked; The template features and the search features are cross-correlated to obtain cross-correlated features. Feature point classification, centroid analysis and feature deviation analysis are then performed on the cross-correlated features to obtain a classification feature map, a centroid score map and a deviation coordinate map. Based on the classification feature map and the centrality score map, the center point of the target box in each frame image is determined, and the tracking result of the video to be tracked is generated based on the center point of the target box in each frame image and the deviation coordinate map. The process of obtaining the template features of the video to be tracked and the search features of each frame in the video to be tracked includes: The network is trained and the initial parameters are iteratively updated to adjust the deviations, thus obtaining the feature extraction network. The feature extraction network is used to extract features from each frame of the image to obtain the template features and the search sub-features. The template features and the search sub-features each include a first preset feature, a second preset feature and a third preset feature. The search feature is obtained by concatenating the template feature and the first preset feature, the second preset feature and the third preset feature in the search sub-feature.

2. The target tracking method as described in claim 1, characterized in that, The step of concatenating the template features and the first, second, and third preset features from the search sub-features to obtain the search features includes: For the first preset feature, the features of the first preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the first concatenated feature; For the second preset feature, the features of the second preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the second concatenated feature; For the third preset feature, the features of the third preset percentage channel in the template feature and the search sub-feature are extracted and concatenated to obtain the third concatenated feature; The first splicing feature, the second splicing feature, and the third splicing feature are spliced ​​together to obtain a fourth splicing feature, and the fourth splicing feature is convolved to obtain a convolutional feature. The convolutional features are subjected to cross-attention processing, and the cross-attention processed convolutional features are combined with the first preset features to obtain the search features.

3. The target tracking method as described in claim 1, characterized in that, The process of performing feature point classification, centroid analysis, and feature deviation analysis on the cross-correlation features to obtain a classification feature map, a centrality score map, and a deviation coordinate map includes: Each pixel in the cross-correlation feature is classified into foreground and background to obtain the classification feature map. The binary classification is used to classify each pixel into foreground and background. The classification feature map includes the classification results of foreground and background for each pixel. Calculate the score of each pixel in the cross-correlation feature as the optimal center to obtain the centrality score map; The deviations between each pixel in the cross-correlation feature and the left, top, right, and bottom edges of the target box are calculated to obtain the deviation coordinate map.

4. The target tracking method as described in claim 1, characterized in that, The step of determining the center point of the target bounding box in each frame image based on the classification feature map and the centrality score map includes: A cosine window penalty is applied to the centrality score map to obtain a penalized score map, and the foreground value of each pixel is extracted from the classification feature map. If the foreground value of any pixel is less than the background value, then the pixel is set to 0; Calculate the product between the foreground value of each pixel and the penalty score map, and determine the pixel with the largest product value as the center point of the target box.

5. The target tracking method as described in claim 1, characterized in that, The step of generating the tracking result of the video to be tracked based on the center point of the target bounding box in each frame image and the deviation coordinate map includes: Based on the position of the center point of the target box, obtain the border deviation value between the center point of the target box and the border in the deviation coordinate diagram; Based on the bounding box deviation value and the position of the target box center point, the predicted bounding box of each frame image is determined respectively; The deviation between the predicted bounding box of each frame image and the target bounding box in the previous frame image is calculated to obtain the bounding box deviation value. The product between the bounding box deviation value and the update ratio value of each frame image is calculated to obtain the update value. The bounding box deviation value is used to characterize the difference between the results of the previous and next frames. The updated values ​​of each frame image are superimposed with the target bounding box in the previous frame image to obtain the tracking results of each frame image, and the tracking results of the video to be tracked are generated based on the tracking results of each frame image.

6. The target tracking method as described in any one of claims 1 to 5, characterized in that, Before performing cross-correlation processing on the template features and the search features to obtain cross-correlation features, the method further includes: The tracking result of the previous frame image is obtained, and the feature is cropped based on the tracking result of the previous frame image to obtain the regression feature; The template features and regression features of the previous frame image are obtained and concatenated to obtain the fifth concatenated feature, and the fifth concatenated feature is then subjected to convolution processing. The fifth stitched feature after convolution processing is enhanced by an attention mechanism, and the enhanced fifth stitched feature is superimposed with the template feature of the initial frame image in the video to be tracked to obtain the superimposed feature; The template features of the current frame image are updated based on the superimposed features.

7. A target tracking system, characterized in that, The system includes: The feature acquisition module is used to acquire the video to be tracked, and to acquire the template features of the video to be tracked and the search features of each frame of the video to be tracked; The cross-correlation module is used to perform cross-correlation processing on the template features and the search features to obtain cross-correlation features, and to perform feature point classification, centroid analysis and feature deviation analysis on the cross-correlation features to obtain a classification feature map, a centroid score map and a deviation coordinate map. The tracking result generation module is used to determine the center point of the target box in each frame image based on the classification feature map and the centrality score map, and to generate the tracking result of the video to be tracked based on the center point of the target box in each frame image and the deviation coordinate map. The feature acquisition module is also used to iteratively update the initialized parameters based on network training, provide feedback to adjust deviations, and obtain the feature extraction network. The feature extraction network is used to extract features from each frame of the image to obtain the template features and the search sub-features. The template features and the search sub-features each include a first preset feature, a second preset feature and a third preset feature. The search feature is obtained by concatenating the template feature and the first preset feature, the second preset feature and the third preset feature in the search sub-feature.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images