Single target tracking system, method, electronic device and storage medium in complex scenarios

Abstract

The present invention discloses a single target tracking system, method, electronic device and storage medium in complex scenes. The tracking system includes: a preprocessing module, which is used to execute: processing the initial target frame and subsequent video frames according to the input illegal template frame to obtain a template area and a search area; passing the target template area into a re-identification network to obtain the initial features of the target; a tracking initial screening module, which is used to execute: obtaining the template area and the search area of ​​the preprocessing module, passing the two into a deep learning single target tracking algorithm, and then screening out the top 10 candidate tracking frames with high confidence through the NMS algorithm, and then passing the top 10 candidate tracking frames into a feature comparison module; the feature comparison module is used to execute: comparing the cosine similarities of the top 10 candidate tracking frames with the target initial features, and selecting the best target tracking frame according to the similarity; a threshold linkage module, which is used to execute: adjusting the size of the search area of ​​the candidate template in the preprocessing module according to the confidence of the output candidate frame.

Description

Technical Field

[0001] The present invention relates to a single target tracking system, method, electronic equipment and storage medium in complex scenarios, and belongs to the technical field of video surveillance and security. Background Art

[0002] Target tracking is a crucial component of computer vision research, with significant application demand in areas such as surveillance and security, unmanned driving, and precision guidance. These applications can be categorized into civilian and military fields, each with its own distinct characteristics. In the civilian field, video surveillance systems must operate with high stability and long-term performance due to the uncertainty surrounding the target's appearance and duration. In the military, high-speed maneuvering targets can exceed Mach 5, requiring tracking systems to maintain real-time performance and accuracy in complex battlefield environments. Precisely because of these challenges, and the difficulty of manually identifying and marking targets to be tracked to meet the requirements of practical tracking systems, the development of alternative target tracking algorithms is crucial.

[0003] In recent years, single-target tracking algorithms based on deep learning twin network series have made great progress. However, in actual scenarios, the interference encountered by the target is more extremely complex, which greatly reduces the tracking performance.

[0004] Patent 1: "A Robust Single Target Tracking Method Based on Deep Learning," CN201910882990.9. Key design elements include setting a threshold to determine whether to initiate a template update, using confidence levels to update the template, and timely updating features based on target changes, thereby avoiding tracking errors caused by template updates. Patent 2: "A Single Target Tracking Method Based on Multiple Features," CN110807794A. This design utilizes a correlation filter tracking method to perform correlation operations on convolutional features and differential image features, then fuses the resulting response maps. The fusion results are then used as the basis for dynamic target coordinate correction to track the target. Summary of the Invention

[0005] The shortcomings of the existing technology are as follows: Patent 1: "A robust single target tracking method based on deep learning", the main disadvantage of its design is that the tracking score setting threshold is used to update the target template, but once there are multiple targets with the same attributes nearby, it is very likely to track other targets while still having a high confidence level, resulting in errors in the subsequent template update, and the target is completely lost; Patent 2: "A single target tracking method based on multiple features", the main disadvantage of its design is that it still uses the traditional target tracking algorithm. When the target undergoes large deformation, lighting changes, or occlusion, the target will still be lost, resulting in subsequent false alarms and other situations.

[0006] The purpose of the present invention is to overcome the technical defects of the existing technology, solve the above-mentioned technical problems, and propose a single target tracking system, method, electronic device and storage medium in complex scenarios. The system and method utilize the latest single target tracking algorithm based on deep learning and the re-identification network, and innovatively combine the two together, which greatly improves the performance of stable tracking of the target under extreme conditions such as interference, lighting, and drastic deformation. At the same time, threshold feedback is used to adjust the search area. When the target disappears temporarily and then appears again, the system and method can still achieve the ability to recognize and stably track.

[0007] The present invention specifically adopts the following technical solution: a single target tracking system in complex scenarios, comprising:

[0008] The preprocessing module is used to perform the following operations: processing the initial target frame and subsequent video frames according to the input violation template bounding box to obtain the template area and search area; passing the target template area to the re-identification network to obtain the initial features of the target;

[0009] The tracking screening module is used to perform the following operations: obtain the template area and search area from the preprocessing module, pass them to the deep learning single target tracking algorithm, and then use the NMS algorithm to filter out the top 10 candidate tracking frames with high confidence. The top 10 candidate tracking frames are then passed to the feature comparison module.

[0010] The feature comparison module is used to perform the following operations: compare the cosine similarity between the first 10 candidate tracking frames and the target initial features, and select the best target tracking frame based on the similarity;

[0011] The threshold linkage module is used to perform: adjusting the sizes of the candidate template area and the search area in the pre-processing module according to the confidence of the output candidate bounding box.

[0012] The present invention also proposes a single target tracking method in complex scenarios, comprising the following steps:

[0013] The preprocessing step specifically includes: processing the initial target frame and subsequent video frames according to the input violation template bounding box to obtain the template area and search area; passing the target template area to the re-identification network to obtain the initial features of the target;

[0014] The initial tracking screening step specifically includes: obtaining the template area and search area from the preprocessing step, passing them to the deep learning single target tracking algorithm, and then filtering out the top 10 candidate tracking frames with high confidence through the NMS algorithm, and then passing the top 10 candidate tracking frames to the feature comparison step;

[0015] The feature comparison step specifically includes: comparing the cosine similarity of the first 10 candidate tracking frames with the target initial features, and selecting the best target tracking frame based on the similarity;

[0016] The threshold linkage step specifically includes: adjusting the sizes of the candidate template area and the search area in the preprocessing step according to the confidence of the output candidate bounding box.

[0017] As a preferred embodiment, the pre-processing step specifically includes:

[0018] Step SS11: Use the target detection algorithm to detect the video stream, or manually select the current frame of the video stream init The tracking target in the image is obtained by taking the bounding box B of the tracking target. init =(x,y,w,h), and crop the template area to get the target O_crop;

[0019] Step SS12: Get the RGB three channels (R mean ,G mean ,B mean ), the average of the RGB three channels is: RGB mean =(R mean +G mean +B mean ) / 3;

[0020] Step SS13: Calculate the size of the template area and the search area according to formula (1) and formula (2); init As the center point, a square with a length and width of z_sz is cropped from the original image, and then the square size is adjusted to (127, 127) through the interpolation algorithm to obtain the template image Z_crop. Similarly, based on the center of the border transmitted by the tracking frame of the previous frame, a square with a length and width of x_sz is cropped from the current frame, and then the square is adjusted to (271, 271) through the interpolation algorithm to obtain the search area X_crop. Once the cropping exceeds the image boundary, the mean RGB value in step SS12 is used. mean Fill the pixels to ensure that the cropped area is inside the image;

[0021]

[0022] x_sz=z_sz*271 / 127 (2)

[0023] Among them, x is the horizontal coordinate of the center point of the initial frame, y is the vertical coordinate of the center point of the initial frame, w is the width of the initial frame, and h is the height of the initial frame.

[0024] As a preferred embodiment, the tracking and initial screening step specifically includes:

[0025] Step SS21: Collect the data sets required for single target tracking, including COCO, GOT10K, VOT2020, LASOT, TrackingNet, VID, DET, YOUTUBEBB, and UAV123, a total of nine data sets for neural network training;

[0026] Step SS22: training a single target tracking model based on deep learning;

[0027] Step SS23: The template area and the search area are respectively passed into the single target tracking model in step SS22 to obtain a series of candidate bounding boxes and corresponding confidences. Then, the NMS algorithm is used to sort them from large to small according to the confidences, and the top 10 bounding boxes are selected to obtain the candidate set of tracking bounding boxes {(B1, Score tracking1 ),(B2,Score tracking2 )...(B 10 ,Score tracking10 )}; where B represents the coordinates of the border, Score tracking Represents the corresponding confidence score;

[0028] Step SS24: Crop the 10 bounding boxes in step SS23 corresponding to the video frame in the original image to obtain candidate crop areas.

[0029] As a preferred embodiment, the threshold of the NMS algorithm in step SS23 is 0.4.

[0030] As a preferred embodiment, the feature comparison step specifically includes:

[0031] Step SS31: Collect a pedestrian re-identification dataset for training to obtain a deep learning metric learning network. The deep learning metric learning network learns the cosine distance between objects and then finds the cluster to which they belong according to the nearest neighbor distance;

[0032] Step SS32: resize the O_crop obtained in step SS11 of the preprocessing step and the 10 candidate regions obtained in step SS24 of the tracking and initial screening step to a size of 64 (width) * 128 (height);

[0033] Step SS33: The region cropped in step SS32 is passed to the deep learning metric learning network in step SS31 to obtain the respective 128-dimensional feature vectors Feature init and Feature candidate ={Feature candidate1 ,Feature candidate2 ...,Feature candidate10}, and then calculate their respective cosine similarity scores, Score reid ={Score reid1 ,Score reid2 ...,Score reid10};

[0034] Step SS34: According to formula (3), the tracking score is combined with the score of the candidate frame to obtain the final tracking score Score final ;

[0035] Score final =w*Score tracking +(1-w)*Score reid (3)

[0036] Step SS35: Score final Sort from large to small and output the bounding box corresponding to the highest confidence.

[0037] As a preferred embodiment, w in step SS34 is selected as 0.4.

[0038] As a preferred embodiment, the threshold linkage step specifically includes:

[0039] Step SS41: Determine Score final Is it greater than the threshold Score threshold If it is greater than, then do not enter this module and exit; if it is less than the threshold, then enter step SS42;

[0040] Step SS42: Resize the search area according to formula (4), and then perform crop processing in the next frame;

[0041] Re-search area:

[0042] x_sz=1.5*z_sz*271 / 127 (4).

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the program.

[0044] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method are implemented.

[0045] The beneficial effects achieved by the present invention are: 1: High-precision, stable and real-time tracking can be achieved for any framed target with a size greater than 10 pixels. A better-performing tracking model is obtained by using a richer training data set, and tracking can be completed even if the target is deformed by distance or affected by lighting; 2: When the target passes through complex interference of the same attribute category, the patent can still achieve stable tracking without losing the tracked target; 3: The mutual fusion of feature re-comparison and deep learning tracking algorithm greatly reduces the probability of tracking failure, and the reliability of tracking is further increased by re-feature comparison of the tracking candidate frame; 4: When the target is lost in the picture for a period of time and reappears, the method can still achieve accurate identification and re-tracking of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the principle topology of the single target tracking system based on deep learning in complex scenarios of the present invention;

[0047] Figure 2 This is a flowchart of the single target tracking method based on deep learning in complex scenarios of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0049] Example 1: The present invention proposes a single target tracking system in complex scenarios, comprising:

[0050] The preprocessing module is used to perform the following operations: processing the initial target frame and subsequent video frames according to the input violation template bounding box to obtain the template area and search area; passing the target template area to the re-identification network to obtain the initial features of the target;

[0051] The tracking initial screening module is used to perform the following operations: obtain the template area and search area of ​​the preprocessing module. The tracking module uses the latest single-target tracking algorithm based on deep learning, SiamCar, and uses the NMS algorithm to screen out the top 10 candidate tracking frames with high confidence. These top 10 candidate tracking frames are then passed to the feature comparison module.

[0052] The feature comparison module is used to perform the following operations: compare the cosine similarity between the first 10 candidate tracking frames and the target initial features, and select the best target tracking frame based on the similarity;

[0053] The threshold linkage module is used to perform: adjusting the size of the search area of ​​the candidate template in the pre-processing module according to the confidence of the output candidate bounding box; so as to achieve the ability to still recognize and track the target after it disappears and reappears.

[0054] The single target tracking system in complex scenarios of the present invention can greatly improve the tracking accuracy and robustness of the target in a variety of complex situations, consumes less time, and can fully achieve real-time effects.

[0055] Example 2: The present invention also proposes a single target tracking method in a complex scene, comprising the following steps:

[0056] The preprocessing step specifically includes: processing the initial target frame and subsequent video frames according to the input violation template bounding box to obtain the template area and search area; passing the target template area to the re-identification network to obtain the initial features of the target;

[0057] The initial tracking screening step specifically includes: obtaining the template area and search area of ​​the preprocessing step, screening out the top 10 candidate tracking frames with high confidence using the NMS algorithm, and then passing the top 10 candidate tracking frames into the feature comparison step;

[0058] The feature comparison step specifically includes: comparing the cosine similarity of the first 10 candidate tracking frames with the target initial features, and selecting the best target tracking frame based on the similarity;

[0059] The threshold linkage step specifically includes: adjusting the sizes of the candidate template area and the search area in the preprocessing step according to the confidence of the output candidate bounding box.

[0060] Preferably, the pretreatment step specifically includes:

[0061] Step SS11: Use target detection algorithms (both traditional detection algorithms and deep learning algorithms) to detect the video stream, or manually select the current frame of the video stream. init The tracking target in the image is obtained by taking the bounding box B of the tracking target. init =(x,y,w,h), and crop the template area to get the target O_crop;

[0062] Step SS12: Get the RGB three channels (R mean ,G mean ,B mean ), the average of the three RGB channels is: RGB mean =(R mean +G mean +B mean ) / 3;

[0063] Step SS13: Calculate the size of the template area and the search area according to formula (1) and formula (2); initAs the center point, a square with a length and width of z_sz is cropped from the original image, and then the square size is adjusted to (127, 127) through the interpolation algorithm to obtain the template image Z_crop. Similarly, based on the center of the border transmitted by the tracking frame of the previous frame, a square with a length and width of x_sz is cropped from the current frame, and then the square is adjusted to (271, 271) through the interpolation algorithm to obtain the search area X_crop. Once the cropping exceeds the image boundary, the mean RGB value in step SS12 is used. mean Fill the pixels to ensure that the cropped area is inside the image;

[0064]

[0065] x_sz=z_sz*271 / 127 (2)

[0066] Among them, x is the horizontal coordinate of the center point of the initial frame, y is the vertical coordinate of the center point of the initial frame, w is the width of the initial frame, and h is the height of the initial frame.

[0067] Preferably, the follow-up initial screening step specifically includes:

[0068] Step SS21: Collect the data sets required for single target tracking, including COCO, GOT10K, VOT2020, LASOT, TrackingNet, VID, DET, YOUTUBEBB, and UAV123, a total of nine data sets for neural network training; this data set contains richer data, further increasing the robustness of the target;

[0069] Step SS22: According to the single target tracking algorithm siamcar, a single target tracking model based on deep learning is trained;

[0070] Step SS23: The template area and the search area are respectively passed into the single target tracking model in step SS22 to obtain a series of candidate bounding boxes and corresponding confidences. Then, the NMS algorithm is used to sort them from large to small according to the confidences, and the top 10 bounding boxes are selected to obtain the candidate set of tracking bounding boxes {(B1, Score tracking1 ),(B2,Score tracking2 )...(B 10 ,Score tracking10 )}; where B represents the coordinates of the border, Score tracking Represents the corresponding confidence score;

[0071] Step SS24: Crop the 10 borders in step SS23 corresponding to the video frame to obtain candidate crop areas.

[0072] Preferably, the threshold of the NMS algorithm in step SS23 is 0.4. Experiments show that this value has a good effect on categories such as pedestrians.

[0073] Preferably, the feature comparison step specifically includes:

[0074] Step SS31: According to Table 1, a pedestrian re-identification dataset is collected for training to obtain a deep learning metric learning network. The deep learning metric learning network learns the cosine distance between objects and then finds the cluster to which it belongs based on the nearest neighbor distance;

[0075] Step SS32: resize the O_crop obtained in step SS11 of the preprocessing step and the 10 candidate regions obtained in step SS24 of the tracking and initial screening step to a size of 64 (width) * 128 (height);

[0076] Step SS33: The region cropped in step SS32 is passed to the deep learning metric learning network in step SS31 to obtain the respective 128-dimensional feature vectors Feature init and Feature candidate ={Feature candidate1 ,Feature candidate2 ...,Feature candidate10}, and then calculate their respective cosine similarity scores, Score reid ={Score reid1 ,Score reid2 ...,Score reid10};

[0077] Step SS34: According to formula (3), the tracking score is combined with the score of the candidate frame to obtain the final tracking score Score final ;

[0078] Score final =w*Score tracking +(1-w)*Score reid (3)

[0079] Step SS35: Score final Sort from large to small and output the bounding box corresponding to the highest confidence.

[0080] Table 1: Deep cosine metric learning network architecture based on deep learning.

[0081]

[0082]

[0083] Preferably, the value w in step SS34 is selected as 0.4. Experimental verification shows that when the value w is 0.4, the experimental effect is the best.

[0084] Preferably, the threshold linkage step specifically includes: setting a threshold according to the size of the output confidence, and when the score is less than the threshold, starting the linkage mechanism to react to the pre-processing module to expand the search area to further find the temporarily disappeared target;

[0085] Step SS41: Determine Score final Is it greater than the threshold Score threshold If it is greater than, then do not enter this module and exit; if it is less than the threshold, then enter step SS42;

[0086] Step SS42: Resize the search area according to formula (4), and then perform crop processing in the next frame;

[0087] Re-search area:

[0088] x_sz=1.5*z_sz*271 / 127 (4).

[0089] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the program.

[0090] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method are implemented.

[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. Single target tracking method in complex scenes, characterized by: The steps include: The preprocessing step specifically includes: processing the initial target frame and subsequent video frames according to the input violation template bounding box to obtain the template area and the search area; passing the target template area to the re-identification network to obtain the initial features of the target; the preprocessing step specifically includes: Step SS11: Use the target detection algorithm to detect the video stream, or manually select the current frame of the video stream init The tracking target in the image is obtained by taking the bounding box B of the tracking target. init =(x, y, w, h), and crop the template area to obtain the target O_crop, where x is the horizontal coordinate of the center point of the initial frame, y is the vertical coordinate of the center of the initial frame, w is the width of the initial frame, and h is the height of the initial frame; Step SS12: Get the RGB three channels (R mean ,G mean ,B mean ), the average of the three RGB channels is: RGB mean =(R mean +G mean +B mean ) / 3; Step SS13: Calculate the size of the template area and the search area according to formula (1) and formula (2); init As the center point, a square with a length and width of z_sz is cropped from the original image, and then the square size is adjusted to (127, 127) through the interpolation algorithm to obtain the template image Z_crop. Similarly, according to the center of the border transmitted by the tracking frame of the previous frame, a square with a length and width of x_sz is cropped from the current frame, and then the square size is adjusted to (271, 271) through the interpolation algorithm to obtain the search area X_crop. Once the cropping exceeds the image boundary, the mean RGB value in step SS12 is used. mean Fill the pixels to ensure that the cropped area is inside the image; x_sz = z_sz * 271 / 127 (2) Where x is the horizontal coordinate of the center point of the initial frame, y is the vertical coordinate of the center point of the initial frame, w is the width of the initial frame, and h is the height of the initial frame; The initial tracking screening step specifically includes: obtaining the template area and search area from the preprocessing step, passing them to the deep learning single target tracking algorithm, and then filtering out the top 10 candidate tracking frames with high confidence through the NMS algorithm, and then passing the top 10 candidate tracking frames to the feature comparison step; The feature comparison step specifically includes: comparing the cosine similarity of the first 10 candidate tracking frames with the target initial features, and selecting the best target tracking frame based on the similarity; The threshold linkage step specifically includes: adjusting the size of the search area of ​​the candidate template in the preprocessing step according to the confidence of the output candidate bounding box.

2. The single target tracking method in complex scenes according to claim 1, characterized in that: The tracking and initial screening steps specifically include: Step SS21: Collect the data sets required for single target tracking, including COCO, GOT10K, VOT2020, LASOT, TrackingNet, VID, DET, YOUTUBEBB, and UAV123, a total of nine data sets for neural network training; Step SS22: According to the single target tracking algorithm siamcar, a single target tracking model based on deep learning is trained; Step SS23: The template area and the search area are respectively passed into the single target tracking model in step SS22 to obtain a series of candidate bounding boxes and corresponding confidences. Then, the NMS algorithm is used to sort them from large to small according to the confidences, and the top 10 bounding boxes are selected to obtain the candidate set of tracking bounding boxes {(B1, Score tracking1 ),(B2,Score tracking2 )...(B 10 ,Score tracking10 )}; where B represents the coordinates of the border, Score tracking Represents the corresponding confidence score; Step SS24: Crop the 10 bounding boxes in step SS23 corresponding to the video frame in the original image to obtain candidate crop areas.

3. The single target tracking method in complex scenes according to claim 2, characterized in that: The threshold of the NMS algorithm in step SS23 is 0.

4.

4. The single target tracking method in complex scenes according to claim 1, characterized in that: The feature comparison step specifically includes: Step SS31: Collect a pedestrian re-identification dataset for training to obtain a deep learning metric learning network. The deep learning metric learning network learns the cosine distance between objects and then finds the cluster to which they belong according to the nearest neighbor distance; Step SS32: resize the O_crop obtained in step SS11 of the preprocessing step and the 10 candidate regions obtained in step SS24 of the tracking and initial screening step to a size of 64 (width) * 128 (height); Step SS33: The region cropped in step SS32 is passed to the deep learning metric learning network in step SS31 to obtain the respective 128-dimensional feature vectors Feature init and Feature candidate ={Feature candidate1 ,Feature candidate2 ...,Feature candidate10 }, and then calculate their respective cosine similarity scores, Score reid ={Score reid1 ,Score reid2 ...,Score reid10 }; Step SS34: According to formula (3), the tracking score is combined with the score of the candidate bounding box to obtain the final tracking score Score final ; Score final =w*Score tracking +(1-w)*Score reid (3) Step SS35: Score final Sort from large to small and output the bounding box corresponding to the highest confidence.

5. The single target tracking method in complex scenes according to claim 4, characterized in that: In step SS34, w is selected as 0.

4.

6. The single target tracking method in complex scenes according to claim 4, characterized in that: The threshold linkage step specifically includes: Step SS41: Determine Score final Is it greater than the threshold Score threshold If it is greater than, do not enter step SS42 and exit; if it is less than the threshold, enter step SS42; Step SS42: Resize the search area according to formula (4), and then perform crop processing in the next frame; The re-search area is: x_sz=1.5*z_sz*271 / 127(4).

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A storage medium having a computer program stored thereon, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images