An adaptive tracking method and system based on super-resolution features
By using an adaptive tracking method based on super-resolution features, super-resolution template images are generated using KCF and SRGAN. Through centroid calculation and multi-scale adjustment, the instability problem of small target tracking in complex scenes is solved, and stable tracking of small targets in core areas is achieved.
Patent Information
- Application Number
- CN202410113116.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-26
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-26
AI Technical Summary
Small target tracking is difficult to perform accurately in complex real-world scenarios, especially under conditions of varying lighting, target occlusion, and scale changes. Existing technologies struggle to effectively distinguish small targets from the background, leading to unstable tracking.
An adaptive tracking method based on super-resolution features is adopted. The KCF target tracking framework is used, combined with the SRGAN method of deep learning to generate super-resolution template images, and the target center is adjusted by centroid calculation and multi-scale adaptation to achieve stable tracking of small targets in the core area.
It improves the stability of small target tracking, enabling accurate tracking of core parts in complex scenarios and reducing tracking instability caused by scale changes.
Smart Images

Figure CN117911457B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target tracking technology, and more specifically, to an adaptive tracking method and system based on super-resolution features. Background Technology
[0002] The core components of medium to large-sized targets are often small targets, and small target tracking has long been a challenge in target tracking. In real-world scenarios, due to the abundance of small targets, small target tracking has broad application prospects, playing a crucial role in many fields such as autonomous driving, drone tracking, smart healthcare, defect tracking, and aerial image analysis. In recent years, the rapid development of deep learning technology has injected new vitality into small target tracking.
[0003] However, compared to targets of regular size, small targets typically lack sufficient appearance information, cover a small image area, and have fewer effective features for target tracking. Furthermore, commonly used trackers are insensitive to small targets, making it difficult to distinguish them from background targets. Therefore, small target tracking remains highly challenging. In addition, real-world scenes are complex, often involving drastic changes in lighting, target occlusion, target pose variations, and target scale changes. These factors have a more pronounced impact on the features of small targets, further increasing the difficulty of small target tracking.
[0004] In view of this, the present invention provides an adaptive tracking method and system based on super-resolution features, which can still accurately track the target even when there is a lack of core features, pose changes, and scale changes, thereby improving the stability of tracking. Summary of the Invention
[0005] The present invention aims to provide an adaptive tracking method based on super-resolution features, comprising: determining an initial bounding box for a region of interest in a video to be detected; selecting an initial template image based on the initial bounding box and generating an initial super-resolution template image; constructing initial cyclic samples and initial labels for the initial super-resolution template image, and training model parameters of an initial detection model based on the initial cyclic samples and initial labels; extracting the first video frame of the video to be detected, and obtaining a first template image based on the first video frame and the initial bounding box; generating a first super-resolution template image of the first template image, and adjusting the center pose of the first super-resolution template image by centroid calculation to obtain a first adjusted image; calculating the FHOG features of the first adjusted image to obtain a first feature; performing multi-scale transformation on the first feature, and inputting the images obtained after multi-scale transformation into the initial detection model, wherein the initial detection model outputs multiple first response maps; determining the first bounding box of the region of interest in the first video frame based on the first response map with the highest confidence and the size of its scale transformation; tracking the second video frame of the video to be detected based on the first bounding box, and repeating this process until the tracking of the last video frame of the video to be detected is completed.
[0006] Furthermore, the initial target bounding box is obtained through target detection model.
[0007] Furthermore, a super-resolution template image is generated by inputting the template image into SRGAN.
[0008] Furthermore, adjusting the center pose of the first super-resolution template image through centroid calculation includes: calculating the sum of gray values of all pixels in the first super-resolution template image to obtain a sum of gray values; calculating the sum of the products of each pixel and its corresponding horizontal axis coordinate and the sum of the products of each pixel and its corresponding vertical axis coordinate in the first super-resolution template image; and calculating the centroid coordinates of the first super-resolution template image based on the sum of gray values, the sum of the horizontal axis products, and the sum of the vertical axis products.
[0009] Furthermore, the expression for calculating the sum of the grayscale values is:
[0010]
[0011] in, Represents the sum of grayscale values; i represents the horizontal axis variable; j represents the vertical axis variable; This represents the grayscale value of the pixel with the horizontal axis at i and the vertical axis at j.
[0012] Furthermore, the expressions for calculating the sum of the products along the horizontal axis and the sum of the products along the vertical axis are as follows:
[0013]
[0014] in, This represents the sum of products on the horizontal axis; i represents the horizontal axis variable; j represents the vertical axis variable; This represents the grayscale value of the pixel with the horizontal axis at i and the vertical axis at j; This represents the sum of the products along the vertical axis.
[0015] Furthermore, the expression for calculating the centroid coordinates is as follows:
[0016]
[0017] in, This represents the sum of products along the horizontal axis; This represents the sum of the products along the vertical axis; Represents the sum of grayscale values; The x-axis coordinates representing the centroid; The vertical axis coordinates represent the centroid.
[0018] Furthermore, the multi-scale includes three scales: 1x scale, 1.05x scale, and 0.95x scale.
[0019] Furthermore, the first target bounding box of the region of interest in the first frame of video is determined to be the size according to the corresponding scale transformation, and the first response map with the highest confidence is restored to the original size to obtain the first target bounding box.
[0020] This invention also provides an adaptive tracking system based on super-resolution features, including an initial bounding box determination module, an initialization module, and an update and detection module. The initial bounding box determination module determines initial bounding boxes for regions of interest in the video to be detected. The initialization module selects an initial template image based on the initial bounding boxes and generates an initial super-resolution template image; constructs initial cyclic samples and initial labels for the initial super-resolution template image, and trains model parameters for an initial detection model based on the initial cyclic samples and initial labels. The update and detection module extracts the first video frame of the video to be detected and obtains a first template image based on the first video frame and the initial bounding boxes; generates... The first template image is a first super-resolution template image, and the center pose of the first super-resolution template image is adjusted by centroid calculation to obtain a first adjusted image; the FHOG features of the first adjusted image are calculated to obtain a first feature; the first feature is subjected to multi-scale transformation, and the images obtained after multi-scale transformation are respectively input into the initial detection model, and the initial detection model outputs multiple first response maps; based on the first response map with the highest confidence and the size of its scale transformation, the first target box of the region of interest in the first frame of video is determined; the second frame of the video to be detected is tracked based on the first target box, and this process is repeated until the tracking of the last frame of the video to be detected is completed.
[0021] The technical solutions of the embodiments of the present invention have at least the following advantages and beneficial effects:
[0022] The present invention provides an adaptive tracking method and system based on super-resolution features, which can be widely applied to the tracking of small targets in the core area. By using the KCF target tracking framework and incorporating the deep learning SRGAN method, centroid calculation and multi-scale adaptation methods to the specific characteristics of the core area, stable tracking of small targets in the core area can be achieved.
[0023] Due to changes in lens focal length and target movement, target bounding box tracking becomes unstable. This invention effectively avoids this instability by incorporating an adaptive scale method. Specifically, three scales (original scale, smaller scale, and larger scale) are automatically selected during KCF update detection. Attached Figure Description
[0024] Figure 1 An exemplary flowchart of an adaptive tracking method based on super-resolution features provided by the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0026] Figure 1 An exemplary flowchart of an adaptive tracking method based on super-resolution features provided by the present invention.
[0027] like Figure 1 As shown, the flowchart of the adaptive tracking method based on super-resolution features includes the following steps:
[0028] Step 110: Determine the initial bounding box of the region of interest in the video to be detected.
[0029] The video to be detected refers to the video in which target tracking needs to be performed. The initial bounding box refers to the rectangular box containing the target to be detected, drawn before target detection. The initial bounding box can be obtained by manual selection or by detection by the target detection model.
[0030] Step 120: Select an initial template image based on the initial target bounding box and generate an initial super-resolution template image.
[0031] A template image refers to an image used for object detection. The template image can be selected from the video to be detected. For example, a frame image of the video to be detected can be used as the template image. The initial template image can be the frame image of frame 0 of the video to be detected. The first template image can be the frame image of the first frame of the video to be detected. And so on, the second template image, the (t+1)th template image, where t+1 is the last frame of the video to be detected. Similarly, the initial template image can also be the frame image of frame 1, and the first template image can be the frame image of frame 2. A super-resolution template image refers to an image with super-resolution obtained by performing super-resolution modeling on the template image. Super-resolution template images can be generated by inputting the template image into an SRGAN.
[0032] Step 130: Construct initial cyclic samples and initial labels for the initial super-resolution template image, and train the model parameters of the initial detection model based on the initial cyclic samples and initial labels.
[0033] The initial loop sample refers to the loop sample established using the initial super-resolution template image. The initial label refers to the label of the initial loop sample, used to identify the first bounding box of the tracked target in each initial loop sample. The initial detection model refers to the model parameter of the first detection model, which is trained using the first loop sample and the first label of the first super-resolution template image. The first detection model is then used to detect the second bounding box of the tracked target in the second frame image. Finally, the t-th detection model is used to detect the (t+1)-th bounding box of the tracked target in the (t+1)-th frame image.
[0034] Step 140: Extract the first video frame of the video to be detected, and obtain the first template image based on the first video frame and the initial target box.
[0035] The first video frame can refer to the first image in the video to be detected. The portion of the first video frame enclosed by the initial target bounding box can be used as the first template image.
[0036] Step 150: Generate a first super-resolution template image of the first template image, and adjust the center pose of the first super-resolution template image by centroid calculation to obtain a first adjusted image.
[0037] The first adjusted image refers to the image obtained by adjusting the center pose of the first super-resolution template image based on the centroid. The centroid can refer to the first-order centroid. First-order centroid calculation is necessary because when a medium-to-large target containing the core area undergoes a significant pose change, the core area is easily occluded by other parts of the target. To further improve the stability and accuracy of core area tracking, adjusting the target center pose through centroid calculation is essential. The goal of centroid calculation is to fine-tune the coordinates of the target center in a timely manner, avoiding the rapid accumulation of errors due to small errors over longer tracking times, which could ultimately lead to tracking instability or even loss of tracking or mistracking. The steps of first-order centroid calculation are as follows:
[0038] Calculate the sum of gray values of all pixels in the first super-resolution template image to obtain the gray value sum; wherein, the expression for calculating the gray value sum is:
[0039]
[0040] in, Represents the sum of grayscale values; i represents the horizontal axis variable; j represents the vertical axis variable; This represents the grayscale value of the pixel with the horizontal axis at i and the vertical axis at j.
[0041] Calculate the sum of the products of each pixel and its corresponding horizontal axis coordinate in the first super-resolution template image, and the sum of the products of each pixel and its corresponding vertical axis coordinate; wherein the expressions for calculating the sum of the horizontal axis products and the sum of the vertical axis products are:
[0042]
[0043] in, This represents the sum of products on the horizontal axis; i represents the horizontal axis variable; j represents the vertical axis variable; This represents the grayscale value of the pixel with the horizontal axis at i and the vertical axis at j; This represents the sum of the products along the vertical axis.
[0044] Based on the sum of the gray values, the sum of the products along the horizontal axis, and the sum of the products along the vertical axis, the centroid coordinates of the first super-resolution template image are calculated. The expression for calculating the centroid coordinates is:
[0045]
[0046] in, This represents the sum of products along the horizontal axis; This represents the sum of the products along the vertical axis; Represents the sum of grayscale values; The x-axis coordinates representing the centroid; The vertical axis coordinates represent the centroid.
[0047] Step 160: Calculate the FHOG features of the first adjusted image to obtain the first feature.
[0048] The first feature refers to the feature obtained by performing FHOG feature extraction on the first adjusted image.
[0049] Step 170: Perform multi-scale transformation on the first feature, and input the images obtained after multi-scale transformation into the initial detection model respectively. The initial detection model outputs multiple first response images.
[0050] The multi-scale method includes three scales: a 1x scale, a 1.05x scale, and a 0.95x scale. For example, the first feature can be scaled by 1.05x and 0.95x respectively, and then the first feature, the 1.05x scaled first feature, and the 0.95x scaled first feature can be input into the initial detection model, and the detection model outputs its corresponding first response map. Of course, the multi-scale method can also be other scales.
[0051] Step 180: Determine the first target bounding box of the region of interest in the first frame of video based on the first response map with the highest confidence and the size of its scale transformation.
[0052] In other embodiments, the first feature can be directly input into the initial detection model, which performs multi-scale target detection and outputs the final first response map. The first target box refers to the rectangular box that tracks the location of the target in the first frame image. The first target box in the first frame video where the region of interest is determined can be restored to its original size based on the size of the corresponding scale transformation, thus obtaining the first target box.
[0053] Step 190: Track the second video frame of the video to be detected based on the first target bounding box, and repeat this process until the last video frame of the video to be detected is tracked. Specifically, target tracking is performed on the (t+1)th frame, including: constructing the t-th cyclic sample and the t-th label based on the t-th super-resolution template image, and training the t-th detection model; obtaining the (t+1)-th template image based on the (t+1)-th video frame and the t-th target bounding box, and generating the (t+1)-th super-resolution template image; performing centroid adjustment and feature extraction on the (t+1)-th super-resolution template image to obtain the (t+1)-th feature; performing multi-scale transformation on the (t+1)-th feature and inputting it into the t-th detection model to obtain the (t+1)-th response map with the highest confidence, and obtaining the (t+1)-th target bounding box in the (t+1)-th video frame.
[0054] This invention utilizes the KCF (Kernel Correlation Filter) target tracking framework. Based on the characteristics of the core target area, it employs the SRGAN (Super-Resolution Generative Adversarial Network) method to generate a super-resolution image of the core area, adjusts the target center through centroid calculation, and incorporates adaptive scaling to adapt to tracking instability caused by scale changes. This solves the instability in tracking small targets in the core area caused by feature lack, pose changes, and scale changes.
[0055] The present invention also provides an adaptive tracking system based on super-resolution features, including an initial target box determination module, an initialization module, and an update and detection module.
[0056] The initial bounding box determination module is used to determine the initial bounding boxes of regions of interest in the video to be detected.
[0057] The initialization module is used to select an initial template image based on the initial target bounding box and generate an initial super-resolution template image; construct initial cyclic samples and initial labels of the initial super-resolution template image, and train the model parameters of the initial detection model based on the initial cyclic samples and initial labels.
[0058] The update and detection module is used to extract the first video frame of the video to be detected, and obtain a first template image based on the first video frame and the initial target box; generate a first super-resolution template image of the first template image, and adjust the center pose of the first super-resolution template image by centroid calculation to obtain a first adjusted image; calculate the FHOG features of the first adjusted image to obtain a first feature; perform multi-scale transformation on the first feature, and input the images obtained after multi-scale transformation into the initial detection model respectively, and the initial detection model outputs multiple first response maps; determine the first target box of the region of interest in the first video frame based on the first response map with the highest confidence and the size of its scale transformation; track the second video frame of the video to be detected based on the first target box, and repeat this process until the tracking of the last video frame of the video to be detected is completed.
[0059] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A super-resolution feature-based adaptive tracking method, characterized by, include: Determine the initial bounding box for the region of interest in the video to be detected; An initial template image is selected based on the initial target bounding box, and an initial super-resolution template image is generated; Construct initial cyclic samples and initial labels for the initial super-resolution template image, and train the model parameters of the initial detection model based on the initial cyclic samples and initial labels; Extract the first video frame of the video to be detected, and obtain the first template image based on the first video frame and the initial target box; A first super-resolution template image is generated from the first template image, and the center pose of the first super-resolution template image is adjusted by centroid calculation to obtain a first adjusted image; The step of adjusting the center pose of the first super-resolution template image through centroid calculation includes: Calculate the sum of the gray values of all pixels in the first super-resolution template image to obtain the gray value sum: ; wherein, represents a gray value; i represents a horizontal axis variable; j represents a vertical axis variable; represents a gray value of a pixel point with a horizontal axis i and a vertical axis j; Calculate the sum of the products of each pixel and its corresponding x-axis coordinate in the first super-resolution template image, and the sum of the products of each pixel and its corresponding y-axis coordinate: ; in, This represents the sum of products on the horizontal axis; i represents the horizontal axis variable; j represents the vertical axis variable; This represents the grayscale value of the pixel with the horizontal axis at i and the vertical axis at j; This represents the sum of the products along the vertical axis; Based on the sum of the gray values, the sum of the products along the horizontal axis, and the sum of the products along the vertical axis, the centroid coordinates of the first super-resolution template image are calculated: ; in, This represents the sum of products along the horizontal axis; This represents the sum of the products along the vertical axis; Represents the sum of grayscale values; The x-axis coordinates representing the centroid; The vertical axis coordinates representing the centroid; Calculate the FHOG features of the first adjusted image to obtain the first feature; The first feature is subjected to multi-scale transformation, and the images obtained after multi-scale transformation are respectively input into the initial detection model. The initial detection model outputs multiple first response images. Based on the first response map with the highest confidence and the size of its scale transformation, the first target box of the region of interest in the first frame of video is determined; Based on the first target bounding box, the second video frame of the video to be detected is tracked, and this process is repeated until the last video frame of the video to be detected is tracked.
2. The adaptive tracking method based on super-resolution features according to claim 1, characterized in that, The initial target bounding box is obtained by the target detection model.
3. The adaptive tracking method based on super-resolution features according to claim 1, characterized in that, Super-resolution template images are generated by inputting template images into SRGAN.
4. The adaptive tracking method based on super-resolution features according to claim 1, characterized in that, The multi-scale includes three scales: 1x scale, 1.05x scale, and 0.95x scale.
5. The adaptive tracking method based on super-resolution features according to claim 1, characterized in that, The first target bounding box of the region of interest in the first frame of video is determined by the size of the corresponding scale transformation, and the first response map with the highest confidence is restored to the original size to obtain the first target bounding box.
6. An adaptive tracking system based on super-resolution features, characterized in that, The adaptive tracking method based on super-resolution features as described in any one of claims 1 to 5 includes an initial target bounding box determination module, an initialization module, and an update and detection module; The initial bounding box determination module is used to determine the initial bounding box of the region of interest in the video to be detected; The initialization module is used to select an initial template image based on the initial target box and generate an initial super-resolution template image; construct initial cyclic samples and initial labels of the initial super-resolution template image, and train the model parameters of the initial detection model based on the initial cyclic samples and initial labels; The update and detection module is used to extract the first video frame of the video to be detected, and obtain the first template image based on the first video frame and the initial target box; A first super-resolution template image is generated from the first template image, and the center pose of the first super-resolution template image is adjusted by centroid calculation to obtain a first adjusted image; the FHOG features of the first adjusted image are calculated to obtain a first feature; the first feature is subjected to multi-scale transformation, and the images obtained after multi-scale transformation are respectively input into the initial detection model, and the initial detection model outputs multiple first response maps; based on the first response map with the highest confidence and the size of its scale transformation, the first target box of the region of interest in the first frame of video is determined; the second frame of the video to be detected is tracked based on the first target box, and this process is repeated until the tracking of the last frame of the video to be detected is completed.
Citation Information
Patent Citations
A system and a method for tracking mobile objects using cameras and tag devices
CA2934102A1
Systems and methods for customization of reviews
CA3098792A1
