Multi-scale object tracking method and system based on scale and location separation

Abstract

The application discloses a multiscale target tracking method based on scale and position separation, comprising the following steps: inputting a template image and a current frame search image into a multiscale network respectively for feature extraction, and obtaining multiscale template features and target features correspondingly; inputting the template features and the target features into a position estimation detection head respectively for processing, and performing convolution on two output results to obtain a preliminary position estimation result; inputting the target features into a scale estimation detection head for processing to obtain a scale estimation result; clustering the preliminary position estimation result to obtain a final position estimation result; splicing the final position estimation result and the scale estimation result to obtain a prediction result; when judging that the number of processed search image frames reaches a preset number, updating the template features, and performing prediction on a next frame search image. The method improves the stability and robustness of the whole tracking process, and has high tracking accuracy.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, specifically relating to a multi-scale target tracking method and system based on scale and position separation. Background Technology

[0002] Single-object tracking is an important area in computer vision, with significant demand in both research and practical applications. In typical practical applications, single-object tracking tasks are long-duration, lasting from tens of seconds to several minutes or more, requiring stable tracking of the target during this period. These tasks also face numerous challenges. In recent years, a series of algorithms based on Siamese networks have been widely adopted in single-object tracking, such as SiamFC, SiamRPN, and SiamRPN++. These algorithms are fundamentally based on Siamese network architectures, using networks like AlexNet and ResNet as the backbone. However, due to model and algorithmic framework limitations, they typically employ only simple linear template update schemes.

[0003] However, existing Siamese network-based algorithms still have the following problems: On the one hand, since the scale of common target types (such as people, cars, and airplanes) typically targeted in tracking tasks changes with the distance between the target and the observer, and deformation problems such as target rotation and flipping also occur, the simple linear template update method used by Siamese network-based algorithms cannot well cope with the problems of target scale changes and target deformation. When the target scale changes significantly, the tracking results will quickly become inaccurate. At the same time, during the change, the selection of the tracking box is not stable enough, resulting in problems such as tracking box drift, inaccurate selection, and even tracking failure. On the other hand, the correlation between the scale estimation and position estimation of the algorithm is high, making it difficult to use the template update method. If the results of neighboring frames are used as the updated template and given high weight in subsequent iterations, the tracking results will not perfectly overlap with the target, leading to rapid accumulation of errors and quick tracking inaccuracy. If it is given low weight in subsequent iterations, it will not be very effective in correcting the inaccuracy of the tracking box caused by the deformation of the target.

[0004] In summary, existing target tracking algorithms suffer from low accuracy in tracking boxes when the target scale changes drastically, making the entire tracking process unstable and lacking robustness. Furthermore, when the target deforms, the cumulative error from template updates affects the tracking performance and the accuracy of target category recognition. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a multi-scale target tracking method and system based on scale and position separation. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, this invention proposes a multi-scale target tracking method based on scale and position separation, comprising:

[0007] Step 1: Obtain the template image and search image of the target to be tracked;

[0008] Step 2: Input the template image into the trained multi-scale network for feature extraction to obtain multi-scale template features, which are then used as the initial template features;

[0009] Step 3: Input the current frame search image into the trained multi-scale network for feature extraction to obtain multi-scale target features;

[0010] Step 4: Input the template features and target features into the trained position estimation detection head for processing, and convolve the two outputs to obtain the preliminary position estimation result; at the same time, input the target features into the trained scale estimation detection head for processing to obtain the scale estimation result.

[0011] Step 5: Cluster the preliminary location estimation results, filter out duplicate results and results with confidence scores below a set threshold, and obtain the final location estimation results;

[0012] Step 6: Combine the final position estimation result and scale estimation result according to scale to obtain the prediction result of the current frame search image;

[0013] Step 7: Determine whether the number of search image frames processed has reached the preset number. If not, return to step 3 and predict the next search image frame; otherwise, proceed to step 8.

[0014] Step 8: Update the template features based on the prediction results of the current frame search image, and clear the number of processed search image frames; then return to step 3 to perform prediction for the next frame search image, thereby achieving target tracking.

[0015] Secondly, this invention proposes a multi-scale target tracking system based on scale and position separation, used to implement the multi-scale target tracking method based on scale and position separation proposed in the first aspect of this invention, including:

[0016] The data acquisition module is used to acquire template images and search images of the target to be tracked;

[0017] The feature extraction module includes a multi-scale network, which is used to extract features from the template image to obtain multi-scale template features; the multi-scale network is also used to extract features from the current frame search image to obtain multi-scale target features.

[0018] The feature processing module includes a position estimation detection head and a scale estimation detection head. The position estimation detection head processes the template features and target features, and convolves the two outputs to obtain a preliminary position estimation result. The scale estimation detection head processes the target features to obtain a scale estimation result.

[0019] The clustering module is used to cluster the preliminary location estimation results, filter out duplicate results and results with confidence levels below a set threshold, and obtain the final location estimation results.

[0020] The stitching module is used to stitch the final position estimation result and scale estimation result together according to scale to obtain the prediction result of the search image in the current frame;

[0021] The update module is used to update the template features based on the prediction results of the current search image when the number of currently processed search image frames reaches a preset number, while simultaneously clearing the number of processed search image frames.

[0022] The beneficial effects of this invention are:

[0023] 1. The multi-scale target tracking method based on scale and position separation provided by this invention, on the one hand, uses a multi-scale network for feature extraction to obtain multiple scale features of the target, so that the tracking accuracy can be high even when the target scale changes greatly during tracking. This overcomes the problem of low tracking box accuracy of existing tracking algorithms when the target scale changes greatly, and improves the stability and robustness of the entire tracking process. On the other hand, it separates scale estimation and position estimation, and uses template updates during the tracking process to reduce the cumulative error of the most recently updated template, thereby improving the tracking accuracy.

[0024] 2. The multi-scale target tracking method based on scale and position separation provided by this invention also trains the scale estimation detection head using a single target dataset, enabling the tracking algorithm to track specific targets more accurately in practical applications.

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a multi-scale target tracking method based on scale and position separation provided in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram illustrating feature extraction of a template image using a multi-scale network, provided in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram illustrating feature extraction of a search image using a multi-scale network, as provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram illustrating the principle of position estimation based on template features and target features provided in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram illustrating the principle of scale estimation based on target features provided in an embodiment of the present invention;

[0031] Figure 6 This is another flowchart illustrating the multi-scale target tracking method based on scale and position separation provided in this embodiment of the invention;

[0032] Figure 7 This is a structural block diagram of a multi-scale target tracking system based on scale and position separation provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1

[0035] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-scale target tracking method based on scale and position separation provided in this embodiment of the invention. The multi-scale target tracking method based on scale and position separation provided in this embodiment specifically includes the following steps:

[0036] Step 1: Obtain the template image and search image of the target to be tracked.

[0037] Generally, the first image obtained that includes the target to be tracked can be used as the base tracking template. Alternatively, a target image can be selected from a pre-built image library as the base tracking template. Meanwhile, subsequent frames are used as search images (or tracking images).

[0038] Step 2: Input the template image into the trained multi-scale network for feature extraction to obtain multi-scale template features, which are then used as the initial template features.

[0039] First, network construction and training are required.

[0040] It is understandable that when the constructed multi-scale network has too many output scales, it will lead to a decrease in algorithm speed, while too few output scales cannot cope with the multi-scale changes of the target. Therefore, this embodiment preferably constructs a network structure with three different output scales.

[0041] Optionally, as an implementation method, this embodiment uses the DarkNet53 network architecture as the basic architecture of the multi-scale network. After processing by the DarkNet53 network, features at three scales can be obtained.

[0042] Specifically, this embodiment uses object detection datasets and object tracking datasets to train the multi-scale network (i.e., the DarkNet53 network). It can be understood that while training this network, the position estimation detection head and scale estimation detection head to be used subsequently can also be trained simultaneously. The position estimation detection head is trained using the object detection dataset, and the scale estimation detection head is trained using the object detection dataset.

[0043] Optionally, the target tracking dataset includes got10k or TrackingNet; the detection dataset includes coco or VOC2017. Furthermore, in the specific implementation of this solution, those skilled in the art can also use other target detection and target tracking datasets to train the DarkNet53 network.

[0044] It should be noted that this embodiment uses a single-target dataset for training the scale estimation detection head; for example, a person, vehicle, or aircraft. This allows the tracking algorithm to track specific targets more accurately in practical applications. For specific training methods of the scale estimation detection head, please refer to existing related technologies.

[0045] Then, the trained network is used for feature extraction.

[0046] This embodiment includes a size determination of the template image before inputting it into the trained scaling network for feature extraction. If the template image size is larger than a preset size, it is compressed to the preset size; or...

[0047] When the size of the template image is smaller than the preset size, the template image is modulated to the preset size by filling a certain range around the target of the template image with gray pixels.

[0048] Specifically, the template image input to the first multi-scale network is generally the entire target, excluding the surrounding environment. When the entire target is larger than 416×416, it is compressed to 416×416. If the template image size is smaller than 416×416, the area around the target that exceeds the range needs to be filled with gray pixels (128, 128, 128). Finally, the actual input is the adjusted 416x416 RGB image.

[0049] Please see Figure 2 , Figure 2 This is a schematic diagram of feature extraction of template images using a multi-scale network provided in an embodiment of the present invention; wherein, the Backbone is the Darknet53 network, and the output feature matrices of three scales have dimensions of 13x13x1024, 26x26x512, and 52x52x256, respectively targeting targets of large, medium and small scales.

[0050] Thus, we have obtained template features at three scales, which are also known as initial template features.

[0051] Step 3: Input the current frame search image into the trained multi-scale network for feature extraction to obtain multi-scale target features.

[0052] Accordingly, before inputting the search image into the trained multi-scale network for feature extraction, the following steps are also included:

[0053] The size of the current frame search image is adjusted to a preset range relative to the target in the previous frame search image, and the excess part is filled with gray pixels. Finally, the current frame search image is modulated to the preset size through compression.

[0054] Specifically, the search image input to the multi-scale network needs to be adjusted relative to the previous frame image. Typically, it is an image within a 1024×720 range around the target in the previous frame image. Areas outside the range are filled with gray pixels (128,128,128). Then, the entire image is compressed into a 416x416 RGB image.

[0055] Please see Figure 3 , Figure 3 This is a schematic diagram of feature extraction of a search image using a multi-scale network provided in an embodiment of the present invention; wherein the Backbone is the Darknet53 network, and the output feature matrices of three scales have dimensions of 13x13x1024, 26x26x512, and 52x52x256, respectively targeting targets of large, medium and small scales.

[0056] Thus, the target features at three scales were obtained.

[0057] This embodiment is based on the DarkNet53 skeleton and outputs target features at three scales, which enables high tracking accuracy even when the target scale changes significantly during tracking. This overcomes the common problem that when the distance between the target and the observer changes drastically, the target size changes with the distance, causing the tracking accuracy to decrease, the entire tracking process to become unstable, and the robustness to decrease. Examples include situations where a drone is flying at high speed and approaches people or vehicles on the ground from the sky.

[0058] Step 4: Input the template features and target features into the trained position estimation detection head for processing, and convolve the two outputs to obtain the preliminary position estimation result; at the same time, input the target features into the trained scale estimation detection head for processing to obtain the scale estimation result.

[0059] In this embodiment, the position estimation detection head mainly uses cross-correlation convolution, which can perform convolution on target features at three scales, making it more robust to changes in target scale.

[0060] For details, please see Figure 4 , Figure 4 This is a schematic diagram illustrating the principle of position estimation based on template features and target features provided in this embodiment of the invention. First, template features at three scales and target features at three scales are input into the trained position estimation detection head to obtain the corresponding outputs. Then, the two parts of the output are convolved to finally obtain the target position confidence scores at three scales: 13×13, 26×26, and 52×52, which are the position estimation results.

[0061] It should be noted that the location estimation result obtained here contains a large number of detection boxes, so it can only be used as a preliminary location estimation result. The final location estimation result can only be obtained by filtering it through the operation in step 5.

[0062] Correspondingly, the target features at the three scales are input into the trained scale estimation detection head to obtain the scale estimation results.

[0063] In this embodiment, the scale estimation detection head uses the same method as the YOLOv3 detection algorithm for predicting target boxes in the tracking algorithm. The basic structure is the same, but the original output parameters are modified. Compared with similar SiamRPN series algorithms, it has higher accuracy in predicting target boxes.

[0064] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the principle of scale estimation based on target features provided in an embodiment of the present invention. The target features at three scales are input into the trained scale estimation detection head to obtain target scales of 13×13, 26×26, and 52×52, which are the scale estimation results.

[0065] It should be noted that the output parameters of the scale estimation detection head have been modified in this embodiment, while retaining parameters such as bx, by, bw, bh, and obj. The first four items represent the coordinates of the top left corner of the object frame, as well as its width and height, and the last item represents the confidence level that an object exists within the frame.

[0066] As can be seen, in terms of algorithm structure, the target scale estimation is independent of the position estimation. The scale estimation is to estimate the target preset during training, so the template update method can be used. The template update method is used in the tracking process. At the same time, the robustness to target scale changes is improved by predicting multiple scales of the target.

[0067] Step 5: Cluster the preliminary location estimation results, filter out duplicate results and results with confidence scores below a set threshold, and obtain the final location estimation results;

[0068] Optionally, as an implementation method, this embodiment uses the kmeans clustering algorithm to cluster the preliminary location estimation results, and performs unified processing on the results of the same target at different scales, and then obtains the confidence and scale estimation of the corresponding location target (location estimation) as the final location estimation result.

[0069] Step 6: Combine the final position estimation result and scale estimation result according to scale to obtain the prediction result of the current frame search image;

[0070] For specific splicing algorithms, please refer to existing related technologies; this embodiment will not provide a detailed description here.

[0071] This completes the prediction and tracking of the search image in the current frame.

[0072] Step 7: Determine whether the number of search image frames processed has reached the preset number. If not, return to step 3 and predict the next search image frame; otherwise, proceed to step 8.

[0073] Step 8: Update the template features based on the prediction results of the current frame search image, and clear the number of processed search image frames; then return to step 3 to perform prediction for the next frame search image, thereby achieving target tracking.

[0074] Specifically, in this embodiment, the prediction result of the current frame search image is weighted and superimposed with the initial template features to update the template features, which can reduce the error caused by the change of the target.

[0075] In this embodiment, template features are not updated within a certain number of frames. After the number of frames is exceeded, due to the separation of scale estimation and position estimation, the prediction result of the current frame can be appropriately enlarged proportionally, and pixels around the target (i.e., the surrounding environment) can be selected to assist in localization. At the same time, the initial frame is not discarded. The new features are added to the initial template features, each with a certain weight. Then, the result and the search area features are sent to the position estimation head for prediction and tracking of the search image in the next frame.

[0076] Deformation issues such as target rotation, scaling, flipping, changes in lighting, and occlusion can be mitigated through template updates, improving tracking accuracy in these situations. However, typical template update mechanisms often lead to rapid error accumulation. In the algorithm structure designed in this embodiment, target scale estimation is independent of position estimation; scale estimation is performed on a pre-set target from training. Therefore, a template update method can be used. Using template updates during tracking reduces the accumulated error of the most recently updated template, improving tracking accuracy in these cases. Furthermore, predicting the target at multiple scales enhances robustness to target scale changes.

[0077] The following is combined with Figure 6 The flowchart of the multi-scale target tracking method based on scale and position separation provided by this invention will be described again.

[0078] First, the first frame image (picture) of the target to be tracked is input into the Darknet53 network as a template image to obtain multi-scale template features; at the same time, the i-th frame image of the target to be tracked is input into the Darknet53 network as a search image to obtain multi-scale target features.

[0079] Then, the detection head is used to perform position estimation and scale estimation on the multi-scale template features and multi-scale target features respectively. The position estimation results are clustered and filtered, and then concatenated with the scale estimation results to finally obtain the target information.

[0080] Finally, determine whether a template update is needed. The criterion is whether the number of search image frames processed since the last template image update has reached a preset number, such as 10 frames. If not, no template update is needed. If so, the template features need to be updated before processing the (i+1)th frame image, until all images have been processed to achieve target tracking.

[0081] The multi-scale target tracking method based on scale and position separation provided by this invention, on the one hand, uses a multi-scale network for feature extraction to obtain multiple scale features of the target, so that the tracking accuracy can be high even when the target scale changes greatly during tracking. This overcomes the problem of low tracking box accuracy of existing tracking algorithms when the target scale changes greatly, and improves the stability and robustness of the entire tracking process. On the other hand, separating scale estimation and position estimation can reduce the cumulative error of the most recently updated template, thereby improving the tracking accuracy.

[0082] Example 2

[0083] Based on Embodiment 1 above, and using the same inventive concept, this embodiment provides a multi-scale target tracking system based on scale and position separation. Please refer to... Figure 7 , Figure 7 This is a structural block diagram of a multi-scale target tracking system based on scale and position separation provided in an embodiment of the present invention, including:

[0084] The data acquisition module is used to acquire template images and search images of the target to be tracked;

[0085] The feature extraction module includes a multi-scale network, which is used to extract features from the template image to obtain multi-scale template features; the multi-scale network is also used to extract features from the current frame search image to obtain multi-scale target features.

[0086] The feature processing module includes a position estimation detection head and a scale estimation detection head. The position estimation detection head processes the template features and target features, and convolves the two outputs to obtain a preliminary position estimation result. The scale estimation detection head processes the target features to obtain a scale estimation result.

[0087] The clustering module is used to cluster the preliminary location estimation results, filter out duplicate results and results with confidence levels below a set threshold, and obtain the final location estimation results.

[0088] The stitching module is used to stitch the final position estimation result and scale estimation result together according to scale to obtain the prediction result of the search image in the current frame;

[0089] The update module is used to update the template features based on the prediction results of the current search image when the number of currently processed search image frames reaches a preset number, while simultaneously clearing the number of processed search image frames.

[0090] The multi-scale target tracking system based on scale and position separation provided in this embodiment can be used to implement the multi-scale target tracking method based on scale and position separation provided in Embodiment 1 above. For detailed implementation process, please refer to Embodiment 1 above. This embodiment will not repeat the description here.

[0091] Therefore, the system can also achieve high tracking accuracy when the target size changes significantly during tracking, thus improving the stability, robustness, and accuracy of the entire tracking process.

[0092] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multi-scale target tracking method based on scale and position separation, characterized in that, include: Step 1: Obtain the template image and search image of the target to be tracked; Step 2: Input the template image into the trained multi-scale network for feature extraction to obtain multi-scale template features, which are then used as the initial template features; Step 3: Input the current frame search image into the trained multi-scale network for feature extraction to obtain multi-scale target features; Step 4: Input the template features and the target features into the trained position estimation detection head for processing, and convolve the two outputs to obtain a preliminary position estimation result; at the same time, input the target features into the trained scale estimation detection head for processing to obtain a scale estimation result; Step 5: Cluster the preliminary location estimation results, filter out duplicate results and results with confidence levels below a set threshold, and obtain the final location estimation results; Step 6: Combine the final position estimation result and the scale estimation result according to scale to obtain the prediction result of the current frame search image; Step 7: Determine whether the number of search image frames processed has reached the preset number. If not, return to step 3 and predict the next search image frame. Otherwise, proceed to step 8; Step 8: Update the template features based on the prediction results of the current frame search image, and clear the number of processed search image frames; then return to step 3 to predict the next frame search image, thereby achieving target tracking.

2. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, The multi-scale network includes three different output scales.

3. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, The multi-scale network adopts the DarkNet53 network architecture.

4. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, In step 2, before inputting the template image into the multi-scale network for feature extraction, the following steps are also included: The template image is size-determined: if the template image size is larger than a preset size, it is compressed to the preset size; or, When the size of the template image is smaller than the preset size, the template image is modulated to the preset size by filling a certain range around the target of the template image with gray pixels.

5. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, Step 3, before inputting the current frame search image into the multi-scale network for feature extraction, also includes: The size of the current frame search image is adjusted to a preset range relative to the target in the previous frame search image, and the excess part is filled with gray pixels. Finally, the current frame search image is modulated to the preset size through compression.

6. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, In step 4, the position estimation detection head is trained using a target tracking dataset, which includes got10k or TrackingNet; The scale estimation detection head is trained using a single-target detection dataset, which may include coco or VOC2017.

7. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, The output parameters of the scale estimation detection head include the coordinates of the top left corner of the object box, the width of the object box, the height of the object box, and the confidence level that an object exists within the box.

8. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, In step 5, the k-means clustering algorithm is used to cluster the preliminary location estimation results.

9. The multi-scale target tracking method based on scale and position separation according to claim 1, characterized in that, In step 8, the template features are updated based on the prediction results of the search image in the current frame, specifically including: The prediction results of the current frame search image are weighted and superimposed with the initial template features to update the template features.

10. A multi-scale target tracking system based on scale and position separation, used to implement the multi-scale target tracking method based on scale and position separation as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire template images and search images of the target to be tracked; The feature extraction module includes a multi-scale network, which is used to extract features from the template image to obtain multi-scale template features; the multi-scale network is also used to extract features from the current frame search image to obtain multi-scale target features. The feature processing module includes a position estimation detection head and a scale estimation detection head. The position estimation detection head is used to process the template features and the target features, and convolve the two output results to obtain a preliminary position estimation result. The scale estimation detection head is used to process the target features to obtain scale estimation results; The clustering module is used to cluster the preliminary location estimation results, filter out duplicate results and results with confidence levels below a set threshold, and obtain the final location estimation results. The stitching module is used to stitch the final position estimation result and the scale estimation result together according to the scale to obtain the prediction result of the current frame search image; The update module is used to update the template features based on the prediction results of the current frame search image when the number of currently processed search image frames reaches a preset number, and at the same time clear the number of processed search image frames.

Images