Target tracking method, device, computer equipment and storage medium

Abstract

The present application relates to a target tracking method, device, computer equipment and storage medium. The method includes: matching the detection result of the current image frame with the tracking result corresponding to the current image frame to obtain the matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame; when the matching result corresponding to the current image frame meets the matching success condition, the tracker is updated according to the detection result of the current image frame; when the matching result corresponding to the current image frame does not meet the matching success condition, if the detection result of the current image frame meets the detection accuracy condition, a new tracker is created for the target; if the detection result of the current image frame does not meet the detection accuracy condition, the detection result of the current image frame is discarded. The use of this method can improve the target tracking effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target tracking method, device, computer equipment and storage medium. Background Art

[0002] With the development of image processing technology, target tracking technology has emerged, which can automatically identify and track the position and movement of specific targets in multiple frames of images. It is widely used in video surveillance, autonomous driving, augmented reality and other fields.

[0003] In traditional technology, the target detected in the image is used to update the tracker. However, in complex real-world scenes, affected by factors such as occlusion and light changes, the target detected by the target detection technology may not be accurate, which affects the update of the tracker and leads to poor tracking results. Summary of the invention

[0004] Based on this, it is necessary to provide a target tracking method, device, computer equipment, computer-readable storage medium and computer program product with better tracking effect in response to the above technical problems.

[0005] In a first aspect, the present application provides a target tracking method, comprising:

[0006] Matching the detection result of the current image frame with the tracking result corresponding to the current image frame to obtain the matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame;

[0007] When the matching result corresponding to the current image frame satisfies a matching success condition, updating the tracker according to the detection result of the current image frame;

[0008] In the case where the matching result corresponding to the current image frame does not satisfy the matching success condition, if the detection result of the current image frame satisfies the detection accuracy condition, creating a new tracker for the target;

[0009] If the detection result of the current image frame does not meet the detection accuracy condition, the detection result of the current image frame is discarded.

[0010] In a second aspect, the present application also provides a target tracking device, comprising:

[0011] A matching module, used to match the detection result of the current image frame with the tracking result corresponding to the current image frame to obtain the matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame;

[0012] An updating module, configured to update the tracker according to the detection result of the current image frame when the matching result corresponding to the current image frame satisfies a matching success condition;

[0013] A creation module is used to create a new tracker for the target if the matching result corresponding to the current image frame does not meet the matching success condition, if the detection result of the current image frame meets the detection accuracy condition; if the detection result of the current image frame does not meet the detection accuracy condition, discard the detection result of the current image frame.

[0014] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method when executing the computer program.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps in the above method when executed by a processor.

[0016] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps in the above method when executed by a processor.

[0017] The above-mentioned target tracking method, device, computer equipment, storage medium and computer program product match the detection result of the current image frame with the tracking result corresponding to the current image frame to obtain the matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame. Since target detection may be affected by various factors in complex scenes, the detection result may not be able to meet the detection accuracy condition. Only using the detection result that meets the detection accuracy condition to update the tracker cannot fully utilize the detection result of target detection, and there is a problem of target missed detection. This problem of target missed detection will further cause the problem of tracking loss. When the matching result corresponding to the current image frame meets the matching success condition, the tracker is updated according to the detection result of the current image frame, and it is determined whether to retain the detection result of the current image frame in combination with the tracking result corresponding to the current image frame, which can effectively avoid the problem of target missed detection and improve the tracking effect. In the case where the matching result corresponding to the current image frame does not meet the matching success condition, if the detection result of the current image frame meets the detection accuracy condition, it means that the detection result is accurate and the tracking result that does not match it is inaccurate, and the prediction of the tracker is inaccurate. At this time, in order to ensure the accuracy of tracking, a new tracker is created for the target. If the detection result of the current image frame does not meet the detection accuracy condition, it means that the detection result is not accurate enough, and the tracking result that does not match it may not be inaccurate. At this time, discarding the detection result of the current image frame and not using the detection result of the current image frame to process the tracker can effectively improve the target tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a target tracking method provided in an embodiment of the present application;

[0020] Figure 2 A simplified flowchart of a target tracking method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a process for updating image features corresponding to a tracker provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a first direction and a second direction provided in an embodiment of the present application;

[0023] Figure 5 A structural block diagram of a target tracking device provided in an embodiment of the present application;

[0024] Figure 6 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0025] Figure 7 An internal structure diagram of another computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] In some embodiments, Figure 1 As shown, a target tracking method is provided, and the method is applied to a computer device as an example for description, including the following steps 102 to 108. Among them:

[0028] Step 102, matching the detection result of the current image frame with the tracking result corresponding to the current image frame to obtain the matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame.

[0029] The tracker is used to track a target in a sequence of images. The sequence of images includes multiple image frames. The tracker can predict the tracking result corresponding to the current image frame based on the detection result of the previous image frame. The time sequence corresponding to the previous image frame is earlier than the time sequence corresponding to the current image frame.

[0030] Exemplarily, the computer device may perform target detection on the current image frame to obtain a detection result of the current image frame. The target in the current image frame is predicted by a tracker to obtain a tracking result corresponding to the current image frame. The detection result may include a detection frame corresponding to the target. The tracking result may include a tracking frame corresponding to the target. The computer device may match the detection frame and the tracking frame to obtain a matching result corresponding to the current image frame.

[0031] In some embodiments, the computer device may determine a matching result based on a difference between a detection result of the current image frame and a corresponding tracking result.

[0032] In some embodiments, the computer device may perform similarity analysis and matching on the detection result of the current image frame and the corresponding tracking result to obtain a matching result. It is understood that the matching result may represent the similarity between the target represented by the tracking result and the target represented by the detection result.

[0033] In some embodiments, the computer device may determine the matching result based on an intersection-over-union ratio between the detection result of the current image frame and the corresponding tracking result.

[0034] In some embodiments, the computer device may determine a first matching result based on the difference between the detection result of the current image frame and the corresponding tracking result. The computer device may determine a second matching result based on the difference between the detection result of the previous image frame and the detection result of the current image frame. The second matching result and the first matching result are weightedly fused to obtain a matching result corresponding to the current image frame. It can be understood that by matching the detection result of the previous image frame with the detection result of the current image frame, it is possible to determine whether the target in the current image frame is the same target as the target tracked by the tracker, and by matching the detection result with the corresponding tracking result, it is possible to determine whether the tracking result is accurate.

[0035] In some embodiments, the computer device may include at least one of a terminal or a server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0036] Step 104: When the matching result corresponding to the current image frame satisfies the matching success condition, the tracker is updated according to the detection result of the current image frame.

[0037] The matching success condition refers to a condition for determining whether the detection result of the current image frame and the tracking result corresponding to the current image frame match successfully.

[0038] Exemplarily, the computer device may determine that the matching result corresponding to the current image frame meets the matching success condition when the matching result corresponding to the current image frame reaches the matching threshold. At this time, the detection result of the current image frame is used to update the tracker.

[0039] In some embodiments, when the similarity represented by the matching result is not less than the matching threshold, the matching result satisfies the matching success condition. The matching threshold may be preset or determined according to the tracking situation of the target. It is understandable that the matching result between the detection result of each image frame and the corresponding tracking result may not necessarily meet the matching success condition, that is, if the tracker loses the target, then the matching threshold should be smaller when the target is tracked again or when the target is just being tracked. For example, from the beginning of tracking the target or reassociating the target after tracking is lost, the matching threshold gradually increases, and after continuously tracking the target for a preset number of frames, the matching threshold is fixed.

[0040] In some embodiments, when the similarity represented by the matching result is within a preset matching range, the matching result satisfies the matching success condition.

[0041] In some embodiments, the computer device may update the Kalman filter in the tracker according to the detection result of the current image frame.

[0042] Step 106 , when the matching result corresponding to the current image frame does not satisfy the matching success condition, if the detection result of the current image frame satisfies the detection accuracy condition, a new tracker is created for the target.

[0043] The detection accuracy condition refers to the condition used to determine whether the detection result is accurate.

[0044] Exemplarily, the computer device may determine that the matching result corresponding to the current image frame does not meet the matching success condition when the matching result corresponding to the current image frame does not reach the matching threshold. At this time, if the detection result of the current image frame meets the detection accuracy condition, it means that the detection result is accurate, but the tracking result is inaccurate. A new tracker is created for the target represented by the detection result of the current image frame, and the detection result of the current image frame is no longer used to update the tracker.

[0045] In some embodiments, the computer device may determine a high confidence threshold. When the confidence of the detection result of the current image frame reaches the high confidence threshold, the detection result of the current image frame satisfies the detection accuracy condition. It is understood that the computer device will output the detection result and the confidence of the detection result for the target detection of the current image frame.

[0046] In some embodiments, the computer device may determine a high confidence range. When the confidence of the detection result of the current image frame is within the high confidence range, the detection result of the current image frame satisfies the detection accuracy condition.

[0047] In some embodiments, when the confidence of the detection result of the current image frame is not less than the high confidence threshold, the detection result of the current image frame satisfies the detection accuracy condition.

[0048] Step 108: If the detection result of the current image frame does not meet the detection accuracy condition, the detection result of the current image frame is discarded.

[0049] It can be understood that if the detection result of the current image frame does not meet the detection accuracy condition, it means that the detection result is inaccurate. Even if the tracking result does not match the detection result, it does not mean that the tracking result is inaccurate. At this time, the detection result of the current image frame is directly discarded.

[0050] In some embodiments, when the confidence of the detection result of the current image frame does not reach the high confidence threshold, the detection result of the current image frame does not meet the detection accuracy condition.

[0051] In the above target tracking method, the detection result of the current image frame and the tracking result corresponding to the current image frame are matched to obtain the matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame. Since target detection may be affected by various factors in complex scenes, the detection result may not meet the detection accuracy condition. Only using the detection result that meets the detection accuracy condition to update the tracker cannot fully utilize the detection result of target detection, and there is a problem of target missed detection. This problem of target missed detection will further cause the problem of tracking loss. When the matching result corresponding to the current image frame meets the matching success condition, the tracker is updated according to the detection result of the current image frame, and whether to retain the detection result of the current image frame is determined in combination with the tracking result corresponding to the current image frame, which can effectively avoid the problem of target missed detection and improve the tracking effect. When the matching result corresponding to the current image frame does not meet the matching success condition, if the detection result of the current image frame meets the detection accuracy condition, it means that the detection result is accurate and the tracking result that does not match it is inaccurate, and the prediction of the tracker is inaccurate. At this time, in order to ensure the accuracy of tracking, a new tracker is created for the target. If the detection result of the current image frame does not meet the detection accuracy condition, it means that the detection result is not accurate enough, and the tracking result that does not match it may not be inaccurate. At this time, discarding the detection result of the current image frame and not using the detection result of the current image frame to process the tracker can effectively improve the target tracking effect.

[0052] In some embodiments, the detection result of the current image frame and the tracking result corresponding to the current image frame are matched to obtain the matching result corresponding to the current image frame, including: when the detection result of the current image frame meets the detection distortion condition, the detection result of the current image frame is discarded; when the detection result of the current image frame does not meet the detection distortion condition, the detection result of the current image frame and the tracking result corresponding to the current image frame are matched to obtain the matching result corresponding to the current image frame.

[0053] The detection distortion condition refers to a condition used to determine whether a detection result is distorted.

[0054] Exemplarily, the computer device may determine whether the confidence of the detection result of the current image frame is less than a low confidence threshold. When the confidence of the detection result is less than the low confidence threshold, the detection result of the current image frame satisfies the detection distortion condition, and the detection result of the current image frame is no longer matched with the corresponding tracking result, but the detection result of the current image frame is directly discarded. When the confidence of the detection result is not less than the low confidence threshold, the detection result of the current image frame does not meet the detection distortion condition. At this time, regardless of whether the detection result meets the detection accuracy condition, the detection result of the current image frame is matched with the corresponding tracking result to obtain a corresponding matching result.

[0055] It is understandable that due to the complex situations that arise during the tracking process, the target may be obscured, and the confidence of the corresponding detection results is relatively low. However, it does not necessarily mean that the target is not detected. Therefore, in order to avoid adverse effects caused by missed detections and distorted detection results, a detection distortion condition is set to filter out distorted detection results and match the undistorted detection results with the tracking results.

[0056] In some embodiments, the low confidence threshold is less than the high confidence threshold.

[0057] In some embodiments, the computer device can determine whether the confidence of the detection result of the current image frame is within a low confidence range. When the confidence of the detection result is within the low confidence range, the detection result of the current image frame satisfies the detection distortion condition. When the confidence of the detection result is not within the low confidence range, the detection result of the current image frame does not satisfy the detection distortion condition.

[0058] In some embodiments, Figure 2 As shown, a simple flow chart of the target tracking method is provided. The current image frame is input into the target detection model, and the target detection model outputs the detection frame of the current image frame. The detection frame is subjected to double threshold processing. When the confidence of the detection frame is greater than the high confidence threshold, the detection frame of the current image frame is determined as a high confidence detection frame, and the detection frame and the tracking frame are matched. If the match is unsuccessful, a new tracker is created, and if the match is successful, the tracker is updated; when the confidence of the detection frame is less than the high confidence threshold and greater than the low confidence threshold, the detection frame of the current image frame is determined as a low confidence detection frame, and the detection frame and the tracking frame are matched. If the match is unsuccessful, the detection frame is discarded, and if the match is successful, the tracker is updated; when the confidence of the detection frame is less than the low confidence threshold, the detection frame of the current image frame is determined as an unreliable detection frame and the detection frame is discarded.

[0059] In some embodiments, the target detection model may be, but is not limited to, a neural network model.

[0060] In some embodiments, the confidence of the detection box is in the interval [0,1]. Assuming that the high confidence threshold is 0.5 and the low confidence threshold is 0.1, the detection box with 1>confidence>0.5 is a high confidence detection box, the detection box with 0.5>confidence>0.1 is a low confidence detection box, and the detection box with confidence<0.1 is an unreliable detection box.

[0061] In this embodiment, in order to avoid the problems of missed detection and false detection, the detection distortion condition is set to determine whether the detection result is distorted, and the distorted detection result is directly discarded. For the undistorted detection result, the detection result is matched with the corresponding tracking result to obtain the matching result, which can ensure the effect of target detection and target tracking.

[0062] In some embodiments, the detection result of the current image frame and the tracking result corresponding to the current image frame are matched to obtain the matching result corresponding to the current image frame, including: determining a first matching result corresponding to the current image frame based on the difference between the detection result of the current image frame and the tracking result corresponding to the current image frame; determining a second matching result corresponding to the current image frame based on the difference between the detection result of the previous image frame and the detection result of the current image frame; determining the size of the target in the current image frame based on the detection result of the current image frame; and weightedly fusing the second matching result and the first matching result based on the size of the target to obtain the matching result corresponding to the current image frame.

[0063] It can be understood that the larger the size of the target in the current image frame, the greater the overlap between the tracking frame and the detection frame is affected by the target size, making the overlap between the tracking frame and the detection frame less able to meet the matching requirements. However, the second matching feature result at this time can better meet the matching requirements. Therefore, it is necessary to adaptively adjust the weights corresponding to the second matching result and the first matching result according to the size of the target. The weights corresponding to the second matching result and the first matching result are closely related to the size of the target in the current image frame.

[0064] Exemplarily, the computer device can perform feature extraction on the detection results of the previous image frame to obtain the previous image features, and perform feature extraction on the detection results of the current image frame to obtain the current image features, and determine the second matching result corresponding to the current image frame based on the similarity between the previous image features and the current image features.

[0065] The computer device may use the intersection-and-union ratio between the detection result of the current image frame and the corresponding tracking result as the first matching result corresponding to the current image frame. The size of the detection box in the detection result of the current image frame is used as the size of the target in the current image frame. The second weight corresponding to the second matching result and the first weight corresponding to the first matching result are determined according to the size of the target. The second matching result and the first matching result are weighted and summed according to the second weight and the first weight to obtain the matching result corresponding to the current image frame.

[0066] In some embodiments, the computer device may use the Euclidean distance between the previous image feature and the current image feature as the matching result. It is understood that the calculation of similarity is not limited to the Euclidean distance, but may also be any one of cosine similarity, Manhattan distance or Mahalanobis distance.

[0067] In some embodiments, the computer device may extract the previous image features from the previous image frame based on a Dense SIFT method, and extract the current image features from the current image frame based on a Dense SIFT method.

[0068] In some embodiments, the computer device can determine each previous image frame whose corresponding matching result satisfies the matching success condition, and determine a second matching result corresponding to the current image frame based on the difference between the detection results of each previous image frame and the detection result of the current image frame.

[0069] In some embodiments, when the matching result corresponding to each image frame satisfies the matching success condition, the image feature of the image frame is determined as the image feature corresponding to the tracker. At this time, the image feature corresponding to the tracker belongs to the previous image feature of the previous image frame. The computer device can determine the second matching result based on the difference between the image feature corresponding to the tracker and the current image feature of the current image frame. It can be understood that when the matching result corresponding to the current image frame satisfies the matching success condition, the image feature corresponding to the tracker is updated to the current image feature.

[0070] In some embodiments, Figure 3 As shown, a schematic diagram of a process for updating the image features corresponding to the tracker is provided, including steps 302 to 320. Among them:

[0071] Step 302, obtaining the detection frame of the current image frame and the current image features.

[0072] Step 304, determining whether the current image frame is the first frame.

[0073] If the current image frame is the first frame, step 306 is executed; otherwise, step 318 is executed.

[0074] Step 306: Calculate the intersection-over-union ratio of the detection frame and the corresponding tracking frame to obtain a first matching result.

[0075] Step 308: Calculate the Euclidean distance between the image feature corresponding to the tracker and the current image feature to obtain a second matching result.

[0076] Step 310: Calculate a second weight corresponding to the second matching result and a first weight corresponding to the first matching result based on the size of the detection frame.

[0077] Step 312: Determine whether the matching result obtained by weighted summing the second matching result and the first matching result is greater than a matching threshold.

[0078] If the matching result is greater than the matching threshold, step 314 is executed; otherwise, step 318 is executed.

[0079] Step 314: Determine whether the tracker is matched successfully.

[0080] Step 316: Update the image features corresponding to the tracker.

[0081] Step 318, assign a new tracker.

[0082] Step 320: Keep the image features corresponding to the tracker without updating.

[0083] In some embodiments, when the size of the target does not meet the weight adjustment condition, the preset first initial weight is used as the second weight corresponding to the second matching result, and the preset second initial weight is used as the first weight corresponding to the first matching result. For example, when the size of the target is small, the second weight and the first weight can both be 0.5.

[0084] In some embodiments, the second weight changes in a curve as the target size increases.

[0085] In some embodiments, when the size of the target does not meet the weight adjustment condition, the second weight and the first weight remain stable. When the size of the target meets the weight adjustment condition, the second weight increases as the size of the target increases, that is, the second weight is positively correlated with the size of the target. The sum of the second weight and the first weight is consistent with the preset total weight.

[0086] In this embodiment, the second matching result is determined based on the difference between the detection result of the previous image frame and the detection result of the current image frame, and the second matching result can characterize whether the targets tracked by the tracker before and after are consistent. The first matching result is determined based on the difference between the detection result of the current image frame and the tracking result corresponding to the current image frame, and the first matching result can characterize whether the target predicted by the tracker is consistent with the target detected by the current image frame. The accuracy of the matching degree represented by the second matching result and the first matching result is closely related to the size of the target in the current image frame. The matching result obtained by weighted fusion of the second matching result and the first matching result according to the size of the target can accurately characterize the matching degree between the detection result and the corresponding tracking result.

[0087] In some embodiments, the second matching result and the first matching result are weightedly fused according to the size of the target to obtain a matching result corresponding to the current image frame, including: when the size of the target satisfies the weight adjustment condition, determining the second weight corresponding to the second matching result according to the size of the target and the weight adjustment function; determining the first weight corresponding to the first matching result according to the second weight; and performing weighted summation of the second matching result and the first matching result based on the second weight and the first weight to obtain a matching result corresponding to the current image frame.

[0088] The weight adjustment condition refers to a condition for determining whether to adjust the weights of the second matching result and the first matching result.

[0089] Exemplarily, when the size of the target is not less than the weight adjustment threshold, the size of the target is used as an input of the weight adjustment function to obtain a second weight corresponding to the second matching result. The difference between the preset total weight and the second weight is used as the first weight corresponding to the first matching result. The second matching result is weighted using the second weight to obtain a first weighted result. The first matching result is weighted using the first weight to obtain a second weighted result. The first weighted result and the second weighted result are summed to obtain a matching result corresponding to the current image frame.

[0090] In some embodiments, when the size of the target is within the weight adjustment interval, the size of the target satisfies the weight adjustment condition.

[0091] In some embodiments, the weight adjustment function may be a curve function. The curve function is a monotonically increasing function between 0 and a preset total weight. For example, when the preset total weight is 1, the weight adjustment function may be, but is not limited to, a Sigmoid function.

[0092] In this embodiment, the accuracy of the matching degree represented by the second matching result and the first matching result is closely related to the size of the target in the current image frame. When the size of the target satisfies the weight adjustment condition, the second weight corresponding to the second matching result is determined according to the size of the target and the weight adjustment function; the first weight corresponding to the first matching result is determined according to the second weight; the second matching result and the first matching result are weightedly summed based on the second weight and the first weight to obtain the matching result corresponding to the current image frame, which can accurately represent the matching degree between the detection result and the corresponding tracking result.

[0093] In some embodiments, the method also includes: when the interval frame number does not reach a preset frame number, using a transition threshold that matches the interval frame number as a matching threshold; the interval frame number refers to the number of previous image frames that meet the matching success condition continuously from the current image frame; when the interval frame number reaches the preset frame number, using a preset stability threshold as the matching threshold; wherein, when the matching result corresponding to the current image frame reaches the matching threshold, the matching result corresponding to the current image frame meets the matching success condition.

[0094] Exemplarily, from the beginning of tracking the target or reassociating the target after tracking is lost, the matching threshold gradually increases as the number of frames continuously tracked increases, and after continuously tracking the target for a preset number of frames, the matching threshold is fixed. The interval frame number is used to characterize the number of frames that are continuously tracked before the current image frame. When the interval frame number is less than the preset frame number, the computer device can determine the transition threshold based on the threshold adjustment function and the interval frame number, and use the transition threshold as the matching threshold. The input of the threshold adjustment function is the interval frame number, and the output is the transition threshold that matches the interval frame number. The functional relationship characterized by the threshold adjustment function is that the transition threshold increases as the interval frame number increases. The threshold adjustment function can be, but is not limited to, a linear function. When the interval frame number is not less than the preset frame number, the preset stable threshold is used as the matching threshold.

[0095] In some embodiments, when the interval frame number is not greater than the preset frame number, the interval frame number does not reach the preset frame number. When the interval frame number is greater than the preset frame number, the interval frame number reaches the preset frame number.

[0096] In this embodiment, when the interval frame number does not reach the preset frame number, the tracking performance of the tracker is unstable. A transition threshold that matches the interval frame number is used as the matching threshold, rather than directly using a fixed preset stable threshold. This can avoid the problem of resource waste caused by frequent creation of trackers. When the interval frame number reaches the preset frame number, the preset stable threshold is used as the matching threshold. At this time, the tracking performance of the tracker is stable, and a new tracker can be created in time when the tracker is inaccurate to ensure the tracking effect.

[0097] In some embodiments, when the matching result corresponding to the current image frame satisfies the matching success condition, the tracker is updated according to the detection result of the current image frame, including: when the interval frame number is non-zero and the matching result corresponding to the current image frame satisfies the matching success condition, the tracker is updated according to the detection result of the current image frame; the method also includes: when the interval frame number is zero and the matching result corresponding to the current image frame satisfies the matching success condition, a virtual trajectory is generated according to the detection result of the previous image frame, and the tracker is updated according to the virtual trajectory and the detection result of the current image frame.

[0098] It can be understood that when the interval frame number is non-zero and the matching result corresponding to the current image frame meets the matching success condition, it means that the current image frame is not the first image frame re-associated after the tracking is lost, and the tracker can be updated according to the detection result of the current image frame. When the interval frame number is zero and the matching result corresponding to the current image frame meets the matching success condition, it means that the current image frame is the first image frame re-associated after the tracking is lost, and the lost part in the middle is unknown. In order to ensure the stable update of the tracker, a virtual track is generated at this time, and the tracker is updated according to the virtual track and the detection result of the current image frame.

[0099] In some embodiments, the tracker may include a Kalman filter. It is understood that when the tracking algorithm constructs the motion model, the constructed Kalman filter is a linear uniform velocity model. Although nonlinear motion can be approximated as a linear model in a short time when the time is very short, it is assumed that when no detection frame is provided to the Kalman filter, the Kalman filter can only rely on its own prediction to recursively update the parameters, which will cause cumulative errors. Using the detection results of the previous image frame before the loss as a clue, the parameters of the new Kalman filter are smoothed online using a virtual trajectory, thereby eliminating the cumulative errors in the loss process. Among them, updating the Kalman filter according to the target represented by the virtual trajectory is the same as the process of updating the Kalman filter according to the target represented by the detection result.

[0100] In some embodiments, the computer device can generate a virtual track based on a virtual track generation algorithm. The detection results of the previous image frame and the current image frame can be used as input parameters of the virtual track generation algorithm. The target represented by the previous image frame is equivalent to the starting position of the virtual track, and the detection result of the current image frame is equivalent to the ending position of the virtual track. For example, the target represented by the last image frame before the tracking is lost is Z t1 , the target represented by the first image frame during reassociation is Z t2 , then the virtual trajectory can be Z t =Traj virtual (Z t1 ,Z t2, t), where t1 < t < t2. Among them, Traj virtual () represents the virtual trajectory generation algorithm, and Z t represents the target at time t.

[0101] In some embodiments, the computer device may determine the motion direction of the target based on the detection results of the prior image frames. The motion direction of the target, the detection results of the prior image frames, and the detection results of the current image frame are used as input parameters of the virtual trajectory generation algorithm to generate a virtual trajectory.

[0102] In this embodiment, when the number of intervening frames is non-zero and the matching result corresponding to the current image frame meets the matching success condition, the tracker is updated according to the detection results of the current image frame; when the number of intervening frames is zero and the matching result corresponding to the current image frame meets the matching success condition, a virtual trajectory is generated according to the detection results of the prior image frame, and the tracker is updated according to the virtual trajectory and the detection results of the current image frame, which can generate a virtual trajectory during the process of losing and re-associating the target to stably update the tracker.

[0103] In some embodiments, updating the tracker according to the detection results of the current image frame includes: determining a first direction according to the detection results of the current frame image; determining a second direction according to the detection results of the prior image frame; constructing a cost matrix according to the difference between the first direction and the second direction and the intersection over union between the detection results of the current frame image and the corresponding tracking results; and updating the tracker based on the cost matrix.

[0104] It can be understood that in order to reduce the influence of noise on the consistency of the velocity direction, a velocity direction consistency term is added to the cost matrix of the tracker, and the cost matrix is used to supervise the consistency between the trajectory direction formed by the detection results of the prior image frame and the direction formed by the detection results of the current image frame. Before the tracker is updated according to the detection results of the current image frame, it is updated according to the detection results of the prior image frame. Therefore, the tracking direction corresponding to the tracker is consistent with the trajectory direction formed by the detection results of the prior image frame. To avoid the influence of noise, instead of using the direction of the tracking result, the trajectory direction formed by the detection results of the prior image frame is adopted.

[0105] In some embodiments, the computer device may perform motion estimation on the target represented by the detection results of the prior image frame and the target represented by the detection results of the prior image frame to determine a first direction. Perform motion estimation on the target represented by the detection results of the prior image frame to determine a second direction.

[0106] In some embodiments, the computer device may determine a previous frame image that is closest to the current image frame and whose detection results are not discarded, and perform motion estimation on the detection results of the previous image frame and the detection results of the current image frame to obtain a first direction. The computer device may determine two previous frame images that are closest to the current image frame and whose detection results are not discarded, and perform motion estimation on the detection results of the two previous image frames to obtain a second direction. For example, (u1, v1) and (u2, v2) represent the target positions at two different times, then the direction obtained by motion estimation is

[0107] In some embodiments, Figure 4 As shown, a schematic diagram of a first direction and a second direction is provided. The angle deviation between the first direction and the second direction is Δθ.

[0108] In some embodiments, the cost matrix of the tracker includes a velocity direction consistency term and an intersection-and-union ratio term. The velocity direction consistency term may be an angular deviation between a first direction and a second direction. The intersection-and-union ratio term refers to an intersection-and-union ratio between a detection result of a current frame image and a corresponding tracking result.

[0109] In some embodiments, the computer device can calculate the cost of each state or decision based on the cost matrix. The cost can be expressed as a weighted sum of the intersection-and-union ratio and the speed-direction consistency term, where the intersection-and-union ratio measures the overlap between the predicted box and the real box, and the speed-direction consistency term measures the similarity between the speed and direction predicted by the tracker and the actual target speed and direction. After that, after calculating the costs of all states, the state or decision with the smallest cost is selected as the optimal solution. According to the optimal solution, the state or decision of the tracker is updated. Specifically, the position, size, direction and other parameters of the tracker can be updated according to the position, size, direction and other information of the optimal solution.

[0110] In some embodiments, the cost matrix in, Represents the tracking result. Z represents the detection result of the current image frame. Characterizes the detection result of the previous image frame. CIOU() is used to calculate the intersection-over-union ratio. v () is used to calculate the angle deviation. λ represents the weight corresponding to the angle deviation.

[0111] In this embodiment, a first direction is determined according to the detection result of the current frame image; a second direction is determined according to the detection result of the previous image frame; a cost matrix is ​​constructed according to the difference between the first direction and the second direction and the intersection-and-union ratio between the detection result of the current frame image and the corresponding tracking result; the tracker is updated based on the cost matrix, and a speed direction consistency item is introduced into the cost matrix to ensure the accuracy of the tracker update.

[0112] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0113] Based on the same inventive concept, the embodiment of the present application also provides a target tracking device for implementing the target tracking method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more target tracking device embodiments provided below can refer to the limitations on the target tracking method above, and will not be repeated here.

[0114] In an exemplary embodiment, Figure 5 As shown, a target tracking device 500 is provided, comprising: a matching module 502, an updating module 504 and a creating module 506, wherein:

[0115] The matching module 502 is used to match the detection result of the current image frame with the tracking result corresponding to the current image frame to obtain the matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame.

[0116] The updating module 504 is used to update the tracker according to the detection result of the current image frame when the matching result corresponding to the current image frame meets the matching success condition.

[0117] A creation module 506 is used to create a new tracker for the target if the matching result corresponding to the current image frame does not meet the matching success condition, if the detection result of the current image frame meets the detection accuracy condition; if the detection result of the current image frame does not meet the detection accuracy condition, discard the detection result of the current image frame.

[0118] In some embodiments, the matching module 502 is used to discard the detection result of the current image frame when the detection result of the current image frame meets the detection distortion condition; when the detection result of the current image frame does not meet the detection distortion condition, match the detection result of the current image frame and the tracking result corresponding to the current image frame to obtain the matching result corresponding to the current image frame.

[0119] In some embodiments, the matching module 502 is used to determine a first matching result corresponding to the current image frame based on the difference between the detection result of the current image frame and the tracking result corresponding to the current image frame; determine a second matching result corresponding to the current image frame based on the difference between the detection result of the previous image frame and the detection result of the current image frame; determine the size of the target in the current image frame based on the detection result of the current image frame; and perform weighted fusion of the second matching result and the first matching result based on the size of the target to obtain a matching result corresponding to the current image frame.

[0120] In some embodiments, the matching module 502 is used to determine the second weight corresponding to the second matching result according to the size of the target and the weight adjustment function when the size of the target satisfies the weight adjustment condition; determine the first weight corresponding to the first matching result according to the second weight; and perform weighted summation of the second matching result and the first matching result based on the second weight and the first weight to obtain the matching result corresponding to the current image frame.

[0121] In some embodiments, the matching module 502 is used to use a transition threshold that matches the interval frame number as a matching threshold when the interval frame number does not reach a preset frame number; the interval frame number refers to the number of previous image frames that meet the matching success condition continuously from the current image frame; when the interval frame number reaches the preset frame number, the preset stable threshold is used as the matching threshold; wherein, when the matching result corresponding to the current image frame reaches the matching threshold, the matching result corresponding to the current image frame meets the matching success condition.

[0122] In some embodiments, the update module 504 is used to update the tracker according to the detection result of the current image frame when the interval frame number is non-zero and the matching result corresponding to the current image frame meets the matching success condition; when the interval frame number is zero and the matching result corresponding to the current image frame meets the matching success condition, generate a virtual trajectory according to the detection result of the previous image frame, and update the tracker according to the virtual trajectory and the detection result of the current image frame.

[0123] In some embodiments, the update module 504 is used to determine a first direction based on the detection results of the current frame image; determine a second direction based on the detection results of the previous image frame; construct a cost matrix based on the difference between the first direction and the second direction and the intersection and union ratio between the detection results of the current frame image and the corresponding tracking results; and update the tracker based on the cost matrix.

[0124] Each module in the above target tracking device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store trackers. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target tracking method is implemented.

[0126] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a target tracking method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0127] Those skilled in the art will understand that Figure 6 or Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0130] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A target tracking method, characterized in that: The method comprises: In the case where the confidence of the detection result of the current image frame is not less than the low confidence threshold, a first matching result is determined according to the difference between the detection result of the current image frame and the tracking result corresponding to the current image frame, based on the DenseSIFT algorithm, a prior image feature is extracted from the detection result of the prior image frame, and a current image feature is extracted from the detection result of the current image frame, a second matching result is determined according to the similarity between the prior image feature and the current image feature, and the first matching result and the second matching result are weightedly fused to obtain a matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize the target detected from the current image frame; the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame; In the case where the interval frame number is non-zero and the matching result corresponding to the current image frame satisfies the matching success condition, a first direction is determined according to the detection result of the current image frame, and a second direction is determined according to the detection result of the previous image frame, a cost matrix is ​​constructed according to the difference between the first direction and the second direction and the intersection-and-union ratio between the detection result of the current image frame and the tracking result corresponding to the current image frame, and the tracker is updated based on the cost matrix; the interval frame number refers to the number of previous image frames that meet the matching success condition consecutively from the current image frame; When the interval frame number is zero and the matching result corresponding to the current image frame satisfies a matching success condition, a virtual track is generated according to the detection result of the previous image frame, and the tracker is updated according to the virtual track and the detection result of the current image frame; In the case where the matching result corresponding to the current image frame does not meet the matching success condition, if the confidence of the detection result of the current image frame is not less than the high confidence threshold, creating a new tracker for the target represented by the detection result of the current image frame; the low confidence threshold is less than the high confidence threshold; In the case where the matching result corresponding to the current image frame does not meet the matching success condition, if the confidence of the detection result of the current image frame is less than the high confidence threshold, the detection result of the current image frame is discarded.

2. The method according to claim 1, characterized in that The method further comprises: When the confidence of the detection result of the current image frame is less than the low confidence threshold, the detection result of the current image frame is discarded.

3. The method according to claim 1, characterized in that The weighted fusion of the first matching result and the second matching result to obtain the matching result corresponding to the current image frame includes: Determining the size of the target in the current image frame according to the detection result of the current image frame; The second matching result and the first matching result are weightedly fused according to the size of the target to obtain a matching result corresponding to the current image frame.

4. The method according to claim 3, characterized in that The step of weighting and fusing the second matching result and the first matching result according to the size of the target to obtain a matching result corresponding to the current image frame includes: When the size of the target satisfies a weight adjustment condition, determining a second weight corresponding to the second matching result according to the size of the target and a weight adjustment function; Determine a first weight corresponding to the first matching result according to the second weight; The second matching result and the first matching result are weightedly summed based on the second weight and the first weight to obtain a matching result corresponding to the current image frame.

5. The method according to claim 1, characterized in that The method further comprises: When the interval frame number does not reach the preset frame number, a transition threshold matching the interval frame number is used as a matching threshold; When the interval frame number reaches a preset frame number, a preset stability threshold is used as a matching threshold; Wherein, when the matching result corresponding to the current image frame reaches the matching threshold, the matching result corresponding to the current image frame satisfies the matching success condition.

6. A target tracking device, characterized in that: The device comprises: A matching module, for, when the confidence of the detection result of the current image frame is not less than a low confidence threshold, extracting a prior image feature from the detection result of the prior image frame and a current image feature from the detection result of the current image frame based on a first matching result determined according to the difference between the detection result of the current image frame and the tracking result corresponding to the current image frame based on a Dense SIFT algorithm, determining a second matching result according to a similarity between the prior image feature and the current image feature, and weightedly fusing the first matching result and the second matching result to obtain a matching result corresponding to the current image frame; the detection result of the current image frame is used to characterize a target detected from the current image frame; and the tracking result corresponding to the current image frame is used to characterize the target predicted by the tracker for the current image frame; An updating module, configured to determine a first direction according to a detection result of the current image frame and a second direction according to a detection result of a previous image frame when the interval frame number is non-zero and the matching result corresponding to the current image frame satisfies a matching success condition, construct a cost matrix according to a difference between the first direction and the second direction and an intersection-and-union ratio between the detection result of the current image frame and the tracking result corresponding to the current image frame, and update the tracker based on the cost matrix; the interval frame number refers to the number of previous image frames that meet the matching success condition consecutively from the current image frame; when the interval frame number is zero and the matching result corresponding to the current image frame satisfies the matching success condition, generate a virtual track according to the detection result of the previous image frame, and update the tracker according to the virtual track and the detection result of the current image frame; A creation module is used to create a new tracker for the target represented by the detection result of the current image frame if the matching result corresponding to the current image frame does not meet the matching success condition and the confidence of the detection result of the current image frame is not less than the high confidence threshold; and discard the detection result of the current image frame if the matching result corresponding to the current image frame does not meet the matching success condition and the confidence of the detection result of the current image frame is less than the high confidence threshold; the low confidence threshold is less than the high confidence threshold.

7. The device according to claim 6, characterized in that The matching module is also used to determine the size of the target in the current image frame according to the detection result of the current image frame; and perform weighted fusion on the second matching result and the first matching result according to the size of the target to obtain the matching result corresponding to the current image frame.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images