Target tracking detection method, device, equipment and storage medium

Abstract

The application relates to a target tracking detection method, device and equipment and a storage medium. The method comprises the following steps: performing position detection on each target object included in a current video frame to obtain a detection frame and a detection confidence degree corresponding to each target object; performing position prediction on each target object included in the current video frame to obtain a prediction tracking frame corresponding to each target object, and dividing a plurality of prediction intervals according to the prediction tracking frames; wherein each prediction interval comprises at least one prediction tracking frame; performing matching processing on the detection frames and the prediction tracking frames according to the confidence classes of the detection confidence degrees corresponding to the target objects and the prediction intervals; and updating the detection positions of the target objects in the current video frame according to the matching result. The method can improve the accuracy of multi-target detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a target tracking and detection method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of deep learning, the research on applying deep learning to object detection has become increasingly mature, and the application of deep learning-based object detection algorithms in practical engineering has become more and more widespread.

[0003] Currently, many application scenarios require multi-object detection. However, existing algorithms for multi-object detection have low accuracy. Therefore, improving the accuracy of multi-object detection is an urgent problem to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide a target tracking and detection method, apparatus, device, and storage medium that can improve the accuracy of multi-target detection in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a target tracking and detection method. The method includes:

[0006] Position detection is performed on each target object included in the current video frame to obtain the detection bounding box and detection confidence score for each target object; position prediction is performed on each target object included in the current video frame to obtain the predicted tracking bounding box for each target object, and multiple prediction intervals are divided based on each predicted tracking bounding box; wherein each divided prediction interval includes at least one of the predicted tracking bounding boxes; each detection bounding box and each predicted tracking bounding box are matched according to the confidence class of the detection confidence score corresponding to each target object and each prediction interval; the detection position of each target object in the current video frame is updated based on the matching results.

[0007] In one embodiment, the prediction intervals are divided according to each prediction tracking box, including: obtaining the depth value of each prediction tracking box in the current video frame; the depth value is used to characterize the distance of the target object corresponding to the prediction tracking box from the shooting point in the current video frame; sorting the depth values, and referring to the sorting results, dividing the prediction intervals according to each prediction tracking box.

[0008] In one embodiment, obtaining the depth value of each predicted tracking box in the current video frame includes: for each predicted tracking box, calculating the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame to obtain the depth value corresponding to the predicted tracking box.

[0009] In one embodiment, the depth values ​​are sorted, and based on the sorting results, multiple prediction intervals are divided according to each predicted tracking box. This includes: arranging the depth values ​​in ascending order to obtain a sorting result; traversing the predicted tracking boxes corresponding to each depth value according to the sorting result, and dividing the prediction intervals based on the center point of each currently traversed predicted tracking box to obtain multiple prediction intervals.

[0010] In one embodiment, the prediction interval is divided based on the center point of each currently traversed prediction tracking box, including: obtaining prediction tracking boxes located within a circular range with the center point as the center and a preset distance value as the radius; and determining the prediction interval corresponding to the currently traversed prediction tracking box based on the prediction tracking boxes within each circular range.

[0011] In one embodiment, the method further includes: for the currently traversed prediction tracking box, detecting whether the currently traversed prediction tracking box is located within a determined prediction interval; if not, performing the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

[0012] In one embodiment, the matching result includes a first matching result; the confidence category includes a high-confidence category, where the detection confidence of the high-confidence category is greater than a first confidence threshold; based on the confidence category of the detection confidence corresponding to each target object and each prediction interval, the detection boxes and each predicted tracking boxes are matched, including: obtaining the detection boxes of the target objects corresponding to the detection confidence of the high-confidence category as high-confidence detection boxes; calculating the correlation between each high-confidence detection box and each predicted tracking box included in each prediction interval; for each prediction interval, generating a first relational loss matrix corresponding to the prediction interval based on the correlation calculated based on each predicted tracking box included in the prediction interval; determining the first matching result based on the first relational loss matrix corresponding to each prediction interval; wherein, the first matching result includes successfully matched detection boxes and predicted tracking box groups, unmatched detection boxes, and unmatched predicted tracking boxes.

[0013] In one embodiment, the correlation between each high-confidence detection box and each predicted tracking box included in each prediction interval is calculated, including: for the target high-confidence detection box and the target predicted tracking box to be calculated, calculating the area intersection value and the height intersection value between the target high-confidence detection box and the target predicted tracking box; and obtaining the correlation between the target high-confidence detection box and the target predicted tracking box based on the area intersection value and the height intersection value.

[0014] In one embodiment, calculating the area intersection value between the target high-confidence detection box and the target predicted tracking box includes: obtaining the area of ​​the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtaining the area of ​​the union region of the target high-confidence detection box and the target predicted tracking box; and using the ratio of the area of ​​the intersection region to the area of ​​the union region as the area intersection value.

[0015] In one embodiment, calculating the height intersection value between the target high-confidence detection box and the target predicted tracking box includes: obtaining the height of the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtaining the height of the union region of the target high-confidence detection box and the target predicted tracking box; using the height of the intersection region and the height of the area of ​​the union region as the height intersection value.

[0016] In one embodiment, the matching result includes a second matching result; the confidence category includes a low-confidence category, where the detection confidence of the low-confidence category is less than a first confidence threshold and greater than a second confidence threshold; based on the confidence category of the detection confidence corresponding to each target object and each prediction interval, each detection box and each predicted tracking box are matched, including: obtaining the detection box of the target object corresponding to the detection confidence of the low-confidence category as a low-confidence detection box; calculating the correlation degree between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result; generating multiple second relationship loss matrices based on the correlation degree calculated based on the unmatched predicted tracking boxes included in the first matching result; determining the second matching result based on each second relationship loss matrix; wherein, the second matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0017] In one embodiment, the matching result includes a third matching result; based on the confidence category of the detection confidence level corresponding to each target object and each prediction interval, each detection box and each predicted tracking box are matched, including: calculating the correlation degree between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library; wherein, the tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result; generating multiple third relationship loss matrices based on the correlation degree calculated based on the historical predicted tracking boxes; determining the third matching result based on each third relationship loss matrix; wherein, the third matching result includes the successfully matched detection boxes and predicted tracking box groups, the unmatched detection boxes, and the unmatched historical predicted tracking boxes.

[0018] In one embodiment, the process of determining the first matching result, the second matching result, and the third matching result includes: taking each first relation loss matrix, each second relation loss matrix, and each third relation loss matrix as target relation loss matrices, and using a preset optimization matching algorithm to filter the correlation of each target relation loss matrix to obtain the first matching result, the second matching result, and the third matching result respectively.

[0019] In one embodiment, the tracking trajectory library also includes the lifecycle value corresponding to each historical predicted tracking box; after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the method further includes: cumulatively updating the lifecycle value of each historical predicted tracking box, and deleting the historical predicted tracking box from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking box is greater than a preset period threshold.

[0020] In one embodiment, updating the detection position of each target object in the current video frame according to the matching result includes: for the first matching result, the second matching result, and the third matching result that include a successfully matched detection box and a predicted tracking box group, replacing the predicted tracking box with the detection box in the detection box and the predicted tracking box group, and using the replaced predicted tracking box as the detection position of the target object.

[0021] In one embodiment, position detection is performed on each target object included in the current video frame to obtain the detection box and detection confidence corresponding to each target object. This includes: inputting the current video frame into a preset detection model for position detection, and obtaining the detection confidence of each detection box corresponding to each target object output by the preset detection model.

[0022] In one embodiment, position prediction is performed on each target object included in the current video frame to obtain a predicted tracking box corresponding to each target object. This includes: inputting the current video frame into a preset tracker after initialization and debugging to perform position prediction, and obtaining the predicted tracking box corresponding to each target object output by the preset tracker.

[0023] In one embodiment, the process of initializing and debugging the preset tracker includes: determining whether the current video frame is the first video frame in the current detection video; if so, inputting the current video frame and the detection boxes of each target object into the preset tracker, and using the preset tracker to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier included in other video frames in the current detection video; wherein, the tracking identifier is also used to associate and display with the detection position of the corresponding target object.

[0024] Secondly, this application also provides a target tracking and detection device. The device includes:

[0025] The position detection module is used to detect the position of each target object included in the current video frame, and obtain the detection box and detection confidence score corresponding to each target object.

[0026] The position prediction module is used to predict the position of each target object included in the current video frame, obtain the prediction tracking box corresponding to each target object, and divide multiple prediction intervals according to each prediction tracking box; wherein each prediction interval includes at least one prediction tracking box.

[0027] The matching module is used to match each detection box and each predicted tracking box according to the confidence class of the detection confidence of each target object and each prediction interval.

[0028] The update module is used to update the detection positions of each target object in the current video frame based on the matching results.

[0029] In one embodiment, the position prediction module is specifically used to: obtain the depth value of each predicted tracking box in the current video frame; the depth value is used to characterize the distance between the target object corresponding to the predicted tracking box and the shooting point in the current video frame; sort the depth values, and refer to the sorting results to divide the predicted tracking box into multiple prediction intervals.

[0030] In one embodiment, the position prediction module is specifically used to: for each predicted tracking box, calculate the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame, and obtain the depth value corresponding to the predicted tracking box.

[0031] In one embodiment, the location prediction module is specifically used to: arrange the depth values ​​in ascending order to obtain a sorting result; traverse the prediction tracking boxes corresponding to each depth value according to the sorting result, and divide the prediction interval based on the center point of each currently traversed prediction tracking box to obtain multiple prediction intervals.

[0032] In one embodiment, the position prediction module is specifically used to: obtain the predicted tracking boxes located within a circular range centered on the center point and with a preset distance value as the radius; and determine the prediction interval corresponding to the currently traversed predicted tracking box based on the predicted tracking boxes within each circular range.

[0033] In one embodiment, the apparatus further includes a detection execution module, configured to: for the currently traversed prediction tracking box, detect whether the currently traversed prediction tracking box is located within a determined prediction interval; if not, execute the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

[0034] In one embodiment, the matching result includes a first matching result; the confidence category includes a high-confidence category, and the detection confidence of the high-confidence category is greater than a first confidence threshold; the matching module is specifically used to: obtain the detection box of the target object corresponding to the detection confidence of the high-confidence category as a high-confidence detection box; calculate the correlation degree between each high-confidence detection box and each predicted tracking box included in each prediction interval; for each prediction interval, generate a first relation loss matrix corresponding to the prediction interval based on the correlation degree calculated based on each predicted tracking box included in the prediction interval; determine the first matching result based on the first relation loss matrix corresponding to each prediction interval; wherein, the first matching result includes a successfully matched detection box and predicted tracking box group, unmatched detection boxes, and unmatched predicted tracking boxes.

[0035] In one embodiment, the matching module is specifically configured to: for the target high-confidence detection box and the target predicted tracking box to be calculated, calculate the area intersection value between the target high-confidence detection box and the target predicted tracking box and calculate the height intersection value between the target high-confidence detection box and the target predicted tracking box; and obtain the correlation degree between the target high-confidence detection box and the target predicted tracking box based on the area intersection value and the height intersection value.

[0036] In one embodiment, the matching module is specifically configured to: obtain the area of ​​the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtain the area of ​​the union region of the target high-confidence detection box and the target predicted tracking box; and use the ratio of the area of ​​the intersection region to the area of ​​the union region as the area intersection-union value.

[0037] In one embodiment, the matching module is specifically configured to: obtain the height of the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtain the height of the union region of the target high-confidence detection box and the target predicted tracking box; and use the height of the intersection region and the height of the area of ​​the union region as the height intersection-union value.

[0038] In one embodiment, the matching result includes a second matching result; the confidence category includes a low-confidence category, wherein the detection confidence of the low-confidence category is less than a first confidence threshold and greater than a second confidence threshold; the matching module is specifically used to: obtain the detection box of the target object corresponding to the detection confidence of the low-confidence category as a low-confidence detection box; calculate the correlation degree between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result; generate multiple second relationship loss matrices based on the correlation degree calculated based on the unmatched predicted tracking boxes included in the first matching result; and determine the second matching result based on each second relationship loss matrix; wherein the second matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0039] In one embodiment, the matching result includes a third matching result; the matching module is specifically configured to: calculate the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library; wherein the tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result; generate multiple third relationship loss matrices based on the correlation calculated based on the historical predicted tracking boxes; determine the third matching result based on each third relationship loss matrix; wherein the third matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched historical predicted tracking boxes.

[0040] In one embodiment, the matching module is specifically used to: take each first relation loss matrix, each second relation loss matrix and each third relation loss matrix as target relation loss matrices, and use a preset optimized matching algorithm to filter the correlation of each target relation loss matrix to obtain the first matching result, the second matching result and the third matching result respectively.

[0041] In one embodiment, the tracking trajectory library also includes the lifecycle value corresponding to each historical predicted tracking box; after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the device further includes a period accumulation module, used to: cumulatively update the lifecycle value of each historical predicted tracking box, and delete the historical predicted tracking box from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking box is greater than a preset period threshold.

[0042] In one embodiment, the update module is specifically used to: replace the predicted tracking box with the detection box in the first matching result, the second matching result and the third matching result that include a successfully matched detection box and a predicted tracking box group, and use the replaced predicted tracking box as the detection position of the target object.

[0043] In one embodiment, the position detection module is specifically used to: input the current video frame into a preset detection model for position detection, and obtain the detection confidence scores of each detection box corresponding to each target object output by the preset detection model.

[0044] In one embodiment, the position prediction module is specifically used to: input the current video frame into the preset tracker after initialization and debugging to perform position prediction, and obtain the predicted tracking boxes corresponding to each target object output by the preset tracker.

[0045] In one embodiment, the position prediction module is specifically used to: determine whether the current video frame is the first video frame in the current detection video; if so, input the current video frame and the detection boxes of each target object into the preset tracker, and use the preset tracker to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier in other video frames in the current detection video; wherein, the tracking identifier is also used to associate and display with the detection position of the corresponding target object.

[0046] Thirdly, this application also provides a computer device, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described in any of the first aspects above.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0048] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0049] The aforementioned target tracking and detection method, apparatus, device, and storage medium, by performing position detection on each target object included in the current video frame to obtain a detection bounding box and detection confidence score for each target object; predicting the position of each target object included in the current video frame to obtain a predicted tracking bounding box for each target object; and dividing multiple prediction intervals, each containing at least one predicted tracking bounding box, based on the prediction tracking bounding box; matching each detection bounding box and each predicted tracking bounding box according to the confidence class of the detection confidence score corresponding to each target object and each prediction interval; and updating the detection position of each target object in the current video frame based on the matching results. In this way, by simultaneously acquiring the detection bounding box and the predicted tracking bounding box of the target object, and dividing the prediction interval for the predicted tracking box, interference and confusion between close target objects are avoided. Furthermore, the introduction of detection confidence scores during the detection process increases the accuracy of matching between each detection bounding box and each predicted tracking bounding box, thereby effectively improving the accuracy of determining the current detection position of each target object in the current video frame. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a diagram illustrating the application environment of a target tracking and detection method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a target tracking and detection method in one embodiment;

[0053] Figure 3 This is a schematic diagram of the process for initializing the tracker in one embodiment;

[0054] Figure 4 This is a schematic diagram of the process of dividing the prediction interval in one embodiment;

[0055] Figure 5 This is a schematic diagram of a video frame in one embodiment;

[0056] Figure 6 This is a schematic diagram of another method for dividing the prediction interval in one embodiment;

[0057] Figure 7 This is a flowchart illustrating the matching process in one embodiment;

[0058] Figure 8 This is a flowchart illustrating the process of calculating the correlation degree in one embodiment;

[0059] Figure 9 This is a schematic diagram of the intersection region in one embodiment;

[0060] Figure 10 This is a schematic diagram of the union region in one embodiment;

[0061] Figure 11 This is an intentional representation of the height of the intersection region in one embodiment;

[0062] Figure 12 This is a schematic diagram illustrating the height of the union region in one embodiment;

[0063] Figure 13 This is a flowchart illustrating another matching process in one embodiment;

[0064] Figure 14 This is a flowchart illustrating another matching process in one embodiment;

[0065] Figure 15 This is a flowchart illustrating a multi-target tracking algorithm in one embodiment;

[0066] Figure 16 This is a structural block diagram of a target tracking and detection device in one embodiment;

[0067] Figure 17 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0068] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that many specific details are set forth in the following description in order to provide a full understanding of this application, but this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0070] In the field of object detection, the current background is as follows: With the rise of deep learning in recent years, research on object detection has become increasingly mature, and deep learning-based object detection algorithms are being applied more and more widely in practical engineering. Consequently, the approach of driving the tracking process based on detection hypotheses has gradually become the mainstream form of multi-object tracking (MOT). Multi-object tracking problems mainly face the following challenges: 1) Frequent occlusion during tracking makes it difficult to accurately locate the target; 2) Different targets may have high appearance similarity, increasing the difficulty of maintaining target IDs; 3) Interactions between targets may cause tracking box drift. These problems limit the further development of MOT research and the accuracy of its application in practical systems. Therefore, combining research on the target occlusion problem and extending it to MOT algorithms plays a crucial role in solving the challenging problems faced by multi-object tracking. Furthermore, it should be noted that the inventors have devoted a great deal of creative effort from identifying the above-mentioned technical problems to the technical solutions described in the following embodiments.

[0071] In view of this, embodiments of this application provide a target tracking and detection method that can improve the accuracy of multi-target tracking and detection.

[0072] The target tracking and detection method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the shooting device 101 communicates with the computer device 102 via a network. A data storage system can store the data that the computer device 102 needs to process. The data storage system can be integrated into the computer device 102 or placed in the cloud or on another network server. The shooting device 101 transmits the currently detected video to the computer device 102. The computer device 102 obtains the current video frame from the current detected video and performs position detection on each target object included in the current video frame, obtaining the detection bounding box and detection confidence score corresponding to each target object; it then performs position prediction on each target object included in the current video frame, obtaining the predicted tracking bounding box corresponding to each target object, and divides multiple prediction intervals based on each predicted tracking bounding box; each prediction interval includes at least one predicted tracking bounding box; based on the confidence class of the detection confidence score corresponding to each target object and each prediction interval, it performs matching processing on each detection bounding box and each predicted tracking bounding box; and updates the detection position of each target object in the current video frame based on the matching results. Optionally, the shooting device 101 can transmit the current detection device value to the computer device 102 in real time, or the computer device 102 can store the detection videos transmitted by the shooting device 101 in the past, and perform detection on some or all of the video frames in the historically transmitted detection videos to determine the detection position of each target object in each video frame, thereby realizing the tracking trajectory of the target object in the detection video.

[0073] The computer device 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, portable wearable devices, or servers. IoT devices can include smart TVs, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, etc. The server can be implemented using a standalone server or a server cluster composed of multiple servers.

[0074] In one embodiment, such as Figure 2 As shown, a target tracking and detection method is provided, which can be applied to... Figure 1 Taking computer device 102 as an example, the following steps are included:

[0075] Step 201: Perform position detection on each target object included in the current video frame to obtain the detection box and detection confidence score corresponding to each target object.

[0076] The current video frame may include one or more target objects. Optionally, the target object can be all objects appearing in the current video frame, or the target object can be some objects of interest in the current video frame. The objects of interest are, for example, objects that occupy an area greater than a preset threshold in the current video frame, etc., but are not specifically limited here.

[0077] The main task of multi-target tracking is to simultaneously locate multiple target objects in a given video sequence, maintain their tracking identifiers, and continuously record their trajectories. Therefore, in this embodiment, for the video sequence of the currently detected video, each video frame can be used as the current video frame for subsequent tracking and detection, thereby determining the trajectory of each target object in the current detected video. Optionally, for multiple consecutive video frames with a repetition rate greater than a target threshold, only one target frame can be selected as the current video frame, and the detection positions of target objects in other repeated video frames can be updated using the detection results of the target frame, thereby saving computational complexity of the computer equipment.

[0078] In this embodiment, position detection is performed on each target object. Based on the detection bounding boxes and detection confidence scores corresponding to each target object obtained from the position detection, the results of position prediction are then correlated to determine a more accurate detection position of the target object from the results of position detection and position prediction. The detection confidence score characterizes the accuracy level of the detection bounding boxes corresponding to the target object.

[0079] Optionally, the computer device can use a preset detection model to perform position detection on the current video frame.

[0080] Optionally, the detection box can be constructed using multiple coordinates corresponding to the edges of the area occupied by the target object in the current video frame. For example, the detection box can be a rectangular detection box, without specific limitations.

[0081] Step 202: Predict the position of each target object in the current video frame to obtain the predicted tracking box corresponding to each target object, and divide the predicted tracking box into multiple prediction intervals. Each prediction interval includes at least one predicted tracking box.

[0082] Optionally, the computer device can use a preset tracker to predict the location of the target object. The preset tracker is used to predict the location of different objects.

[0083] Considering the priority and occlusion order of each target object in the current video frame, prediction intervals are divided for each predicted tracking bounding box. These intervals include the predicted tracking bounding boxes of each target object at different layer depths within the current video frame. In other words, by dividing the prediction intervals, target objects at different layer depths are segmented, and subsequent matching processing based on these intervals allows for separate matching of target objects at different layer depths, reducing interference and confusion caused by occlusion.

[0084] Optionally, the predicted tracking box can also be constructed using multiple coordinates corresponding to the edges of the area occupied by the target object in the current video frame. For example, the predicted tracking box can be a rectangular detection box, without specific limitation. It is understood that since the results of position detection and position prediction may differ, the detection box and the predicted tracking box corresponding to the same target object may have different positions in the current video frame.

[0085] Step 203: Match each detection box and each predicted tracking box according to the confidence category of the detection confidence level corresponding to each target object and each prediction interval.

[0086] For example, the confidence category of the detection confidence may include a high confidence category and a low confidence category. The detection confidence of the high confidence category indicates that the detection box of the target object given by the computer device is more accurate, while the detection confidence of the low confidence category indicates that the detection box of the target object given by the computer device is less accurate.

[0087] By combining confidence categories and prediction intervals, predicted tracking boxes at each layer depth can be matched separately. This facilitates data association matching between predicted tracking boxes and detection boxes, thereby improving matching accuracy. For example, detection boxes of high-confidence categories are first matched with predicted tracking boxes in each prediction interval, and then detection boxes of low-confidence categories are matched with predicted tracking boxes in each prediction interval. Since target objects corresponding to low-confidence categories may often have serious problems, classifying target objects based on detection confidence before performing matching calculations improves the performance of computer matching calculations and the accuracy of data association.

[0088] Step 204: Update the detection positions of each target object in the current video frame based on the matching results.

[0089] The computer device can first display the detection location of each target object based on each predicted tracking bounding box. Then, based on the matching results, if the detection bounding box and the predicted tracking bounding box of a certain target object match successfully, it means that the detection bounding box more accurately represents the location of the target object. Therefore, the detection location of the target object can be displayed based on the detection bounding box of the target object, thereby realizing location update.

[0090] The aforementioned target tracking and detection method, apparatus, device, and storage medium, by performing position detection on each target object included in the current video frame to obtain a detection bounding box and detection confidence score for each target object; predicting the position of each target object included in the current video frame to obtain a predicted tracking bounding box for each target object; and dividing multiple prediction intervals, each containing at least one predicted tracking bounding box, based on the prediction tracking bounding box; matching each detection bounding box and each predicted tracking bounding box according to the confidence class of the detection confidence score corresponding to each target object and each prediction interval; and updating the detection position of each target object in the current video frame based on the matching results. In this way, by simultaneously acquiring the detection bounding box and the predicted tracking bounding box of the target object, and dividing the prediction interval for the predicted tracking box, interference and confusion between close target objects are avoided. Furthermore, the introduction of detection confidence scores during the detection process increases the accuracy of matching between each detection bounding box and each predicted tracking bounding box, thereby effectively improving the accuracy of determining the current detection position of each target object in the current video frame.

[0091] In one embodiment, position detection is performed on each target object included in the current video frame to obtain the detection box and detection confidence score corresponding to each target object, including: inputting the current video frame into a preset detection model for position detection, and obtaining the detection confidence scores of each target object corresponding to each target object output by the preset detection model.

[0092] The computer device can input the current video frame into the preset detection model and obtain the detection confidence of each detection box corresponding to each target object output by the preset detection model.

[0093] The preset detection model can be obtained by training based on the initially built network model.

[0094] For example, during the training of the pre-defined detection model, a computer device can acquire multiple sample videos collected by urban road surveillance cameras, analyze the sample videos, and obtain multiple video image sequences. Multiple sample video frames are selected from each video image sequence, and training and testing datasets are obtained based on these sample video frames. The detectable objects included in each sample video frame may be, for example, vehicles or pedestrians, without specific limitations.

[0095] The computer equipment inputs each sample video frame from the training dataset into the initially built network model for iterative training to obtain candidate detection models. Then, it uses each sample video frame from the test dataset to test the candidate detection models, and iteratively updates them based on the test results to obtain the predictive detection model.

[0096] Optionally, the initial network model can be a YOLOX network structure model. In this way, the high stability and accuracy of the YOLOX network can be used to improve the detection accuracy of the preset detection model.

[0097] In this embodiment, a pre-trained preset detection model is used to obtain the detection box and detection confidence of the target object, thereby improving the efficiency and accuracy of the computer device in the location detection process.

[0098] In one embodiment, the position prediction of each target object included in the current video frame is performed to obtain the predicted tracking box corresponding to each target object. This includes: inputting the current video frame into a preset tracker after initialization and debugging to perform position prediction, and obtaining the predicted tracking box corresponding to each target object output by the preset tracker.

[0099] After initialization and debugging, the preset tracker can track and detect target objects in the video frames input to it, assigning corresponding tracking identifiers to each target object. Before assigning tracking identifiers, the preset tracker needs to be initialized to generate tracking identifiers for each target object. These tracking identifiers are used to distinguish the identity of each target object. Only then can tracking identifiers be assigned to target objects in different video frames and displayed in association.

[0100] Optionally, the preset tracker can be a Kalman tracker. The basic principle of Kalman filter target tracking is to estimate and predict the target state using a Kalman filter. The Kalman filter is a recursive filter that continuously updates the state estimate by estimating and predicting the system state, thereby achieving the tracking and prediction of the system state.

[0101] In one embodiment, please refer to Figure 3 The diagram illustrates a flowchart of an initialization process for a tracker provided in an embodiment of this application. The process of initializing and debugging the preset tracker includes:

[0102] Step 301: Determine whether the current video frame is the first video frame in the currently detected video.

[0103] Step 302: If yes, input the current video frame and the detection bounding boxes of each target object into the preset tracker. Use the preset tracker to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier in other video frames in the current detection video. The tracking identifiers are also used to associate and display the detection position of the corresponding target object.

[0104] If the first frame is detected, the preset tracker is initialized using the detection bounding boxes of each target object. That is, the first video frame and the detection bounding boxes of each target object are input into the preset tracker, which can identify each target object based on each detection bounding box, and then generate a tracking label corresponding to each target object.

[0105] The preset tracker records each target object and its corresponding tracking identifier. For video frames that are not the first frame in the current detection video, the initialized preset tracker can directly assign tracking identifiers to each target object in that video frame. Simultaneously, the preset tracker also predicts the position of each target object and outputs a predicted tracking bounding box for each target object.

[0106] In this way, the computer device can display the detection position of each target object, and at the same time display the tracking mark of the target object. As the video frames of the current detection video change continuously, the detection position and tracking mark of each target object can be seen to change continuously with each target object, thus realizing the position tracking and marking of each target object.

[0107] In one embodiment, such as Figure 4 The diagram illustrates a flowchart of a prediction interval division method provided in an embodiment of this application. Multiple prediction intervals are divided based on each prediction tracking frame, including:

[0108] Step 401: Obtain the depth value of each predicted tracking box in the current video frame. The depth value is used to characterize the distance between the target object corresponding to the predicted tracking box and the shooting point in the current video frame.

[0109] In one optional implementation, obtaining the depth value of each predicted tracking box in the current video frame includes: for each predicted tracking box, calculating the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame to obtain the depth value corresponding to the predicted tracking box.

[0110] Please refer to Figure 5 The illustrated video frame diagram includes the predicted tracking boxes for targets 1 to 4. It can be seen that the depth value 1 corresponding to the predicted tracking box 1 for target object 1 is the smallest, indicating that target object 1 is relatively close to the shooting point. The depth value 4 corresponding to the predicted tracking box 4 for target object 4 is the largest, indicating that target object 4 is relatively far from the shooting point. In other words, the smaller the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame (corresponding to a smaller depth value), the closer the target object is to the shooting point in that current video frame.

[0111] In another optional implementation, the depth value of each predicted tracking box in the current video frame is obtained, including: for each predicted tracking box, calculating the vertical distance between the top edge of the predicted tracking box and the bottom edge of the current video frame to obtain the depth value corresponding to the predicted tracking box.

[0112] Step 402: Sort the depth values ​​and, with reference to the sorting results, divide the prediction intervals according to each prediction tracking box.

[0113] In one implementation, the depth values ​​are sorted, and based on the sorting results, multiple prediction intervals are divided according to each predicted tracking box. This includes: arranging the depth values ​​in ascending order to obtain a sorting result; traversing the predicted tracking boxes corresponding to each depth value according to the sorting result, and dividing the prediction intervals based on the center point of each currently traversed predicted tracking box to obtain multiple prediction intervals.

[0114] For example, first, the first predicted tracking box corresponding to the first ranked depth value is determined. The computer device determines the center point of this first predicted tracking box and divides it into prediction interval 1. It is understood that prediction interval 1 includes this first predicted tracking box, and other predicted tracking boxes are relatively close to it. Next, the second predicted tracking boxes corresponding to the second ranked depth values ​​are traversed. The computer device determines the center point of this second predicted tracking box and divides it into prediction interval 2. It is understood that prediction interval 2 includes this second predicted tracking box, and other predicted tracking boxes are relatively close to it. This process of traversing the predicted tracking boxes in order yields multiple prediction intervals.

[0115] In an optional embodiment of this application, the method further includes: for the currently traversed prediction tracking box, detecting whether the currently traversed prediction tracking box is located within a determined prediction interval; if not, then performing the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

[0116] For example, when traversing to the second predicted tracking box, first determine whether the second predicted tracking box is assigned to prediction interval 1. If so, skip the second predicted tracking box, that is, do not execute the step of assigning prediction intervals based on the center point of the second predicted tracking box. If not, execute the step of assigning prediction intervals based on the center point of the currently traversed predicted tracking box.

[0117] This ensures that each predicted tracking box is assigned to a single prediction interval, achieving the effect of prioritizing occlusion based on layer depth and obtaining multiple prediction intervals at different layer depths. This avoids the problem of interference and confusion between adjacent predicted tracking boxes during association when occlusion occurs, thereby reducing the number of times the tracking labels for each target object are switched when displaying the detection location.

[0118] In one embodiment, the prediction interval is divided based on the center point of each currently traversed prediction tracking box, including: obtaining prediction tracking boxes located within a circular range with the center point as the center and a preset distance value as the radius; and determining the prediction interval corresponding to the currently traversed prediction tracking box based on the prediction tracking boxes within each circular range.

[0119] For example, please refer to Figure 6 This diagram illustrates a method for dividing prediction intervals according to an embodiment of this application. First, the predicted tracking box 1 is traversed, and predicted tracking boxes 1 to 3 are divided into prediction interval 1 based on predicted tracking box 1. Since predicted tracking boxes 2 and 3 have already been divided, they are skipped, and the division is directly based on predicted tracking box 4, thus placing predicted tracking boxes 4 and 5 into prediction interval 2. Similarly, predicted tracking boxes 6 to 8 are divided into prediction interval 4. Figure 6 As can be seen, the layer depth of each prediction interval in the current video frame is different, with prediction interval 3 being the deepest.

[0120] It should be noted that, in addition to dividing the prediction interval based on a circular range, other shapes can also be used to divide the prediction interval, and no specific limitations are made here.

[0121] It is understandable that the depth values ​​can also be arranged in descending order to obtain a sorting result, and the prediction interval can be divided with reference to this sorting result. Further details are omitted.

[0122] In this embodiment, the predicted tracking box is divided into prediction intervals based on layer depth information and the distance relationship between neighboring target objects, so that data association can be performed on each prediction interval in an orderly manner, thereby improving the robustness of target tracking under occlusion conditions.

[0123] In one embodiment, such as Figure 7 The diagram illustrates a matching process according to an embodiment of this application. Based on the confidence class of the detection confidence level corresponding to each target object and each prediction interval, the matching process is performed between each detection box and each predicted tracking box, including:

[0124] Step 701: Obtain the detection bounding box of the target object corresponding to the detection confidence of the high confidence category as the high confidence detection bounding box.

[0125] As mentioned above, the confidence categories include high confidence categories and low confidence categories.

[0126] For example, the detection confidence level of a high-confidence category is greater than the first confidence threshold; the detection confidence level of a low-confidence category is less than the first confidence threshold but greater than the second confidence threshold. It is understood that the first confidence threshold is greater than the second confidence threshold.

[0127] In other words, the computer device is equipped with a first confidence threshold and a second confidence threshold. For each target object, the computer device calculates the relationship between the detection confidence of the target object and the first confidence threshold and the second confidence threshold. If the detection confidence is greater than the first confidence threshold, it belongs to the high confidence category. If it is less than the first confidence threshold but greater than the second confidence threshold, it belongs to the low confidence category.

[0128] Alternatively, if the detection confidence of a target object is less than the second confidence threshold, it means that the accuracy is low, and the detection box corresponding to the target object is discarded and does not participate in the subsequent matching process.

[0129] Step 702: Calculate the correlation between each high-confidence detection box and each predicted tracking box included in each prediction interval.

[0130] Step 703: For each prediction interval, generate the first relation loss matrix corresponding to the prediction interval based on the correlation degree calculated based on each prediction tracking box included in the prediction interval.

[0131] For example, with Figure 6 For example, for prediction interval 1, the correlation between each high-confidence detection box and the predicted tracking boxes 1 to 3 is calculated, and the first relational loss matrix corresponding to prediction interval 1 is generated based on the obtained correlation.

[0132] Step 704: Determine the first matching result based on the first relation loss matrix corresponding to each prediction interval.

[0133] The matching results include the first matching result. The first matching result includes successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0134] In one embodiment, the computer device also inputs the unmatched detection boxes included in the first matching result into the preset tracker for re-initialization and debugging, so that the preset tracker generates tracking identifiers for the target objects corresponding to each unmatched detection box.

[0135] It should be noted that the initialization was performed using the first video frame above. However, if the current video frame is not the first frame, new target objects that are different from the target objects in the first frame may appear in the current video frame. Therefore, it is necessary to initialize again to generate tracking identifiers for the newly appearing target objects.

[0136] In one embodiment, a preset optimized matching algorithm can be used to filter the correlation of each first relation loss matrix to obtain a first matching result. In other words, the preset optimized matching algorithm can be used to filter out results with high correlation, where the detection box and the predicted tracking box corresponding to the result have a high correlation. In this way, multiple sets of highly correlated detection boxes and predicted tracking boxes can be filtered out, forming a group of successfully matched detection boxes and predicted tracking boxes.

[0137] For unmatched detection boxes and unmatched predicted tracking boxes, the correlation can be recalculated for matching.

[0138] In this embodiment, the detection boxes are divided according to the relationship between detection confidence and threshold, and then associated with the predicted tracking boxes in sequence, thereby increasing the accuracy of matching.

[0139] In one embodiment, such as Figure 8 The diagram illustrates a flowchart of a method for calculating correlation degree according to an embodiment of this application. The correlation degree between each high-confidence detection box and each predicted tracking box included in each prediction interval is calculated, including:

[0140] Step 801: For the target high-confidence detection box and target prediction tracking box to be calculated, calculate the area intersection value between the target high-confidence detection box and the target prediction tracking box, and calculate the height intersection value between the target high-confidence detection box and the target prediction tracking box.

[0141] In one implementation, calculating the area intersection-union value between the target high-confidence detection box and the target predicted tracking box includes: obtaining the area of ​​the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtaining the area of ​​the union region of the target high-confidence detection box and the target predicted tracking box; and using the ratio of the area of ​​the intersection region to the area of ​​the union region as the area intersection-union value.

[0142] In this context, the height of the target object in the video frame reflects depth information to some extent, and the height of the detection box mainly depends on the distance between the target object and the shooting device. This makes the height status an effective clue for distinguishing objects with overlapping heights. Therefore, in this embodiment, height information modulation IoU matching is introduced to obtain the correlation degree HMIoU, and then the HMIoU matching is used to achieve enhanced recognition and matching of target objects under occlusion conditions.

[0143] IoU, or Intersection Over Union, is the area over union value at this location. It is used to calculate the overlap rate between the predicted bounding box and the actual bounding box.

[0144] like Figure 9 The filled portion shows the intersection region of the high-confidence target detection box and the target predicted tracking box; as shown in the image. Figure 10 The filled portion shows the union region of the high-confidence target detection box and the target predicted tracking box.

[0145] Computer devices can calculate the area of ​​the intersection region and the area of ​​the union region, and use the ratio of the area of ​​the intersection region to the area of ​​the union region as the IoU.

[0146] In one implementation, calculating the height intersection-union value between the target high-confidence detection box and the target predicted tracking box includes: obtaining the height of the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtaining the height of the union region of the target high-confidence detection box and the target predicted tracking box; and using the ratio of the height of the intersection region to the height of the union region as the height intersection-union value.

[0147] This application introduces a height intersection-union value HIoU related to height information. For example... Figure 11 The height H1 of the intersection region between the high-confidence target detection box and the target predicted tracking box is shown in the figure; for example... Figure 12 The figure shows the height H2 of the union region of the high-confidence detection box and the predicted tracking box of the target. A computer device can calculate the ratio of the height of the intersection region to the height of the union region. Thus, HIoU is obtained.

[0148] Step 802: Obtain the correlation between the high-confidence detection box and the predicted tracking box of the target based on the area intersection value and the height intersection value.

[0149] Optionally, the computer device can calculate the product of the area intersection value and the height intersection value to obtain the correlation degree HMIoU.

[0150] In this embodiment, the accuracy of data association is improved by modulating the IoU. Data association based on IoU is prone to matching errors, which limits the tracking ability to handle occlusion. However, this application modulates the IoU based on height information to enhance the recognition of occlusion phenomena and improve the accuracy of multi-target tracking and detection.

[0151] In one embodiment, such as Figure 13 The diagram illustrates another matching process provided in an embodiment of this application. Based on the confidence class of the detection confidence level corresponding to each target object and each prediction interval, the detection bounding box and each predicted tracking bounding box are matched, including:

[0152] Step 1301: Obtain the detection bounding box of the target object corresponding to the detection confidence of the low confidence category as the low confidence detection bounding box.

[0153] It is understandable that the detection box of the target object corresponding to the detection confidence of the low confidence category is the other detection boxes in the valid detection boxes besides the detection boxes of the high and low confidence categories already used for matching mentioned above.

[0154] Step 1302: Calculate the correlation between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result.

[0155] The process of calculating the correlation between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result is the same as the process of calculating the correlation between each high-confidence detection box and each predicted tracking box included in each prediction interval, and will not be repeated here.

[0156] Step 1303: Generate multiple second relationship loss matrices based on the correlation degree calculated from the unmatched predicted tracking boxes included in the first matching result.

[0157] Optionally, for each unmatched predicted tracking box A, the correlation degree between each low-confidence detection box and predicted tracking box A is calculated, and a second relational loss matrix corresponding to predicted tracking box A is generated based on the obtained correlation degree. For each unmatched predicted tracking box B, the correlation degree between each low-confidence detection box and predicted tracking box B is calculated, and a second relational loss matrix corresponding to predicted tracking box B is generated based on the obtained correlation degree. This process continues until the correlation degree between all low-confidence detection boxes and all unmatched predicted tracking boxes is calculated, resulting in the second relational loss matrix corresponding to each unmatched predicted tracking box.

[0158] Alternatively, a second relational loss matrix can be generated based on the correlation degree calculated from the unmatched predicted tracking boxes included in the first matching result, corresponding to each low-confidence detection box.

[0159] Step 1304: Determine the second matching result based on the loss matrices of each second relation.

[0160] The matching results include a second matching result. This second matching result includes successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes. It is understood that the successfully matched detection boxes and predicted tracking boxes, as well as the unmatched detection boxes, all correspond to low-confidence categories.

[0161] In one embodiment, such as Figure 14 The diagram illustrates another matching process provided in an embodiment of this application. Based on the confidence class of the detection confidence level corresponding to each target object and each prediction interval, the detection bounding box and each predicted tracking bounding box are matched, including:

[0162] Step 1401: Calculate the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library. The tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result.

[0163] It is understandable that the second matching result may contain 0, 1, or more unmatched detection boxes. If there are 0, then the steps of calculating the correlation between the unmatched detection boxes in the second matching result and the historical predicted tracking boxes in the tracking trajectory library can be skipped.

[0164] The computer device has a tracking trajectory library deployed in it. The unmatched predicted tracking boxes included in the second matching result calculated for each video frame are added to the tracking trajectory library so that matching can be performed using other video frames that appear later.

[0165] The process of calculating the correlation between each unmatched detection box and the historical predicted tracking boxes included in the tracking trajectory library is the same as the process of calculating the correlation between each high-confidence detection box and each predicted tracking box included in each prediction interval, and will not be repeated here.

[0166] Step 1402: Generate multiple third-relationship loss matrices based on the correlation degree calculated from the historical prediction tracking boxes.

[0167] Optionally, for a given historical predicted tracking box X, the correlation degree between each unmatched detection box included in the second matching result and the historical predicted tracking box X is calculated, and a third relation loss matrix corresponding to the historical predicted tracking box X is generated based on the obtained correlation degree. For a given historical predicted tracking box Y, the correlation degree between each unmatched detection box included in the second matching result and the historical predicted tracking box Y is calculated, and a third relation loss matrix corresponding to the historical predicted tracking box Y is generated based on the obtained correlation degree, until the correlation degree between all low-confidence detection boxes in the second matching result and all historical predicted tracking boxes is calculated, thus obtaining the third relation loss matrix corresponding to each historical predicted tracking box.

[0168] Alternatively, the second relation loss matrix can be generated based on the correlation degree calculated from each historical predicted tracking box, corresponding to each unmatched detection box included in the second matching result.

[0169] Step 1403: Determine the third matching result based on the loss matrices of each third relationship.

[0170] The matching results include a third matching result; wherein the third matching result includes successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched historical predicted tracking boxes.

[0171] Optionally, if the third matching result includes multiple unmatched detection boxes, they can be discarded. However, for historical prediction tracking boxes that have not matched successfully, their lifecycle values ​​need to be updated.

[0172] The tracking trajectory library also includes the lifecycle values ​​corresponding to each historical predicted tracking box.

[0173] In this embodiment of the application, after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the method further includes: cumulatively updating the lifecycle value of each historical predicted tracking box, and deleting the historical predicted tracking box from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking box is greater than a preset period threshold.

[0174] For example, after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the lifetime value of each historical predicted tracking box involved in the calculation is incremented by 1. When the updated lifetime value of a historical predicted tracking box is greater than a preset lifetime threshold, it means that the object corresponding to the historical predicted tracking box may no longer appear. In this case, the historical predicted tracking box can be deleted from the tracking trajectory library, reducing the subsequent computational load of the computer device and avoiding unnecessary calculations.

[0175] Optionally, the preset period threshold can be set to 100, that is, after calculating 100 frames, the historical prediction tracking box is removed.

[0176] In this embodiment of the application, the performance of the multi-target tracking algorithm is improved by also introducing low-confidence detection boxes into the matching process. Low-confidence detection boxes usually correspond to severe occlusion problems. By retaining such results for processing, the integrity of the tracking trajectory is preserved.

[0177] In one embodiment, the process of determining the first matching result, the second matching result, and the third matching result includes: taking each first relation loss matrix, each second relation loss matrix, and each third relation loss matrix as target relation loss matrices, and using a preset optimization matching algorithm to filter the correlation of each target relation loss matrix to obtain the first matching result, the second matching result, and the third matching result respectively.

[0178] In other words, a preset optimized matching algorithm can be used to filter the correlation of each second relation loss matrix to obtain a second matching result. Similarly, a preset optimized matching algorithm can be used to filter the correlation of each third relation loss matrix to obtain a third matching result. This process can filter out multiple sets of highly correlated detection boxes and predicted tracking boxes, forming a group of successfully matched detection boxes and predicted tracking boxes.

[0179] Optionally, the preset optimized matching algorithm can be the Hungarian algorithm, thereby improving the accuracy and efficiency of determining each matching result.

[0180] In one embodiment, updating the detection position of each target object in the current video frame according to the matching result includes: for the first matching result, the second matching result, and the third matching result that include a successfully matched detection box and a predicted tracking box group, replacing the predicted tracking box with the detection box in the detection box and the predicted tracking box group, and using the replaced predicted tracking box as the detection position of the target object.

[0181] In this way, the updated detection positions and tracking markers of each target object are displayed together, enabling accurate tracking of multiple targets.

[0182] Optionally, the computer device can store the matching results corresponding to each video frame, so that the trajectory of each target object in the current detection video can be obtained based on each matching result.

[0183] For ease of understanding, the target tracking and detection method provided in this application will be described below with a complete embodiment. Please refer to... Figure 15 The diagram illustrates the flow of a multi-target tracking algorithm. Specifically, it includes:

[0184] (1) Acquire video images. This involves acquiring video footage from urban road surveillance cameras, streaming it to the server, and obtaining a sequence of video images.

[0185] (2) Data preprocessing and offline training of the detection model. Specifically, the dataset samples required for training are created using video image sequences, and the YOLOX network structure is trained based on the dataset samples to obtain the object detection model.

[0186] (3) YOLOX Online Target Detection. This involves acquiring the detection results of the current video frame output by the target detection model, and classifying them into high-confidence and low-confidence detection results based on a threshold.

[0187] If it is the first frame detection, the Kalman tracker is initialized using the detection result, and each target is assigned a different tracking ID to distinguish the identity of each target.

[0188] (4) Prioritizing Occlusion in Tracking Results. Specifically, the tracking result `Trt` predicted by the Kalman tracker is obtained, and the layer depth (vertical distance between the bottom of the tracking box and the bottom of the image) is calculated for each result. The layer depth values ​​are then sorted from smallest to largest. A traversal is then performed, using the center of the target tracking box corresponding to the depth value as the center, and selecting a target box within a certain radius as an interval. This process continues sequentially through the remaining tracking boxes until all tracking boxes have been traversed, resulting in multiple small intervals. Finally, each small interval is sequentially matched with the detection results.

[0189] (5) Data Association and Cascaded Matching. This involves introducing height information modulation (IOU) matching and using HMIoU matching to enhance target recognition and matching under occlusion conditions. HMIoU = IoU * HIoU. 1) First, calculate the HMIoU values ​​between the high-confidence detection box in the current frame and the tracking boxes in each depth layer interval sequentially to obtain a pairwise HMIoU relationship loss matrix. Based on the HMIoU loss matrix, use the Hungarian algorithm to match the initial tracking trajectory with the high-resolution bounding box in the current frame, obtaining three results: successfully matched tracking-detection result pairs, unmatched tracking trajectories, and unmatched current frame detection results. Update the position of the tracked target using the successfully matched current frame detection results, keeping the target's tracking ID unchanged, and initialize a new Kalman tracker for the unmatched detection results. 2) Then, calculate the HMIoU value between the low-confidence detection result in the current frame and the tracking result that failed to match in the previous step, obtain the relationship matrix, and use the Hungarian algorithm for matching and association to obtain successfully matched tracking-detection result pairs (update the position of the tracked target using the successfully matched detection result in the current frame, and keep the target's tracking ID unchanged), unmatched detection results, and unmatched tracking results. Add the unmatched tracking results to the tracking loss trajectory library. 3) Finally, continue to use the unmatched low-confidence detection results to calculate the HMIoU value between them and the tracking boxes in the tracking loss trajectory library, use the Hungarian algorithm to obtain the matching results, obtain successfully matched tracking-detection boxes (update the position of the tracked target using the successfully matched detection result in the current frame, and keep the target ID value unchanged), discard the unmatched detection results, and increment the lifetime of the unmatched tracking results by 1. When its lifetime is greater than 100, remove it from the tracking loss trajectory library.

[0190] Testing revealed that existing tracking algorithms often result in inaccurate tracking IDs when two target objects intersect, leading to swapped tracking IDs. However, the method provided in this application maintains stable tracking IDs for both target objects when they intersect, ensuring continuous tracking without swapping and achieving high tracking accuracy. Optionally, this method can be effectively applied in practical engineering applications such as intelligent surveillance, urban security, and air and sea sentry duties.

[0191] 1) This application embodiment takes into account the interaction between target neighborhoods. By dividing the occlusion priority according to the layer depth of the tracking results, the problem of mutual interference and confusion in the process of associating data of nearby targets when occlusion occurs is avoided, thereby effectively reducing the number of tracking ID switching; (2) This application embodiment divides the detection results by threshold and associates them with the tracking results in sequence, thereby increasing the accuracy of matching; (3) This application embodiment improves the performance of multi-target tracking algorithm by introducing low confidence detection results into the algorithm. Low detection confidence usually corresponds to severe occlusion problems. By retaining such results for processing, the integrity of the tracking trajectory is preserved; (4) This application embodiment improves the accuracy of data association by modulating IoU. Data association based on IoU is prone to matching errors, thereby limiting the tracking ability to handle occlusion. By modulating IoU, the enhanced recognition of occlusion phenomena is achieved.

[0192] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0193] Based on the same inventive concept, this application also provides a target tracking and detection apparatus for implementing the target tracking and detection method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more target tracking and detection apparatus embodiments provided below can be found in the limitations of the target tracking and detection method described above, and will not be repeated here.

[0194] In one embodiment, such as Figure 16 As shown, a target tracking and detection device 1600 is provided, comprising: a position detection module 1601, a position prediction module 1602, a matching module 1603, and an update module 1604, wherein:

[0195] Secondly, this application also provides a target tracking and detection device. The device includes:

[0196] The position detection module 1601 is used to detect the position of each target object included in the current video frame and obtain the detection box and detection confidence score corresponding to each target object.

[0197] The position prediction module 1602 is used to predict the position of each target object included in the current video frame, obtain the prediction tracking box corresponding to each target object, and divide multiple prediction intervals according to each prediction tracking box; wherein each prediction interval includes at least one prediction tracking box.

[0198] The matching module 1603 is used to match each detection box and each predicted tracking box according to the confidence category of the detection confidence level of each target object and each prediction interval.

[0199] The update module 1604 is used to update the detection position of each target object in the current video frame based on the matching results.

[0200] In one embodiment, the position prediction module 1602 is specifically used to: obtain the depth value of each predicted tracking box in the current video frame; the depth value is used to characterize the distance between the target object corresponding to the predicted tracking box and the shooting point in the current video frame; sort the depth values, and refer to the sorting results to divide multiple prediction intervals according to each predicted tracking box.

[0201] In one embodiment, the position prediction module 1602 is specifically used to: for each predicted tracking box, calculate the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame to obtain the depth value corresponding to the predicted tracking box.

[0202] In one embodiment, the location prediction module 1602 is specifically used to: arrange the depth values ​​in ascending order to obtain a sorting result; traverse the prediction tracking boxes corresponding to each depth value according to the sorting result, and divide the prediction interval based on the center point of each currently traversed prediction tracking box to obtain multiple prediction intervals.

[0203] In one embodiment, the position prediction module 1602 is specifically used to: obtain the prediction tracking boxes located within a circular range with the center point as the center and a preset distance value as the radius; and determine the prediction interval corresponding to the currently traversed prediction tracking box based on the prediction tracking boxes within each circular range.

[0204] In one embodiment, the apparatus further includes a detection execution module, configured to: for the currently traversed prediction tracking box, detect whether the currently traversed prediction tracking box is located within a determined prediction interval; if not, execute the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

[0205] In one embodiment, the matching result includes a first matching result; the confidence category includes a high-confidence category, where the detection confidence of a high-confidence category is greater than a first confidence threshold; the matching module 1603 is specifically used to: obtain the detection box of the target object corresponding to the detection confidence of the high-confidence category as a high-confidence detection box; calculate the correlation degree between each high-confidence detection box and each predicted tracking box included in each prediction interval; for each prediction interval, generate a first relational loss matrix corresponding to the prediction interval based on the correlation degree calculated based on each predicted tracking box included in the prediction interval; and determine the first matching result based on the first relational loss matrix corresponding to each prediction interval; wherein, the first matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0206] In one embodiment, the matching module 1603 is specifically used to: for the target high-confidence detection box and the target predicted tracking box to be calculated, calculate the area intersection value between the target high-confidence detection box and the target predicted tracking box and calculate the height intersection value between the target high-confidence detection box and the target predicted tracking box; and obtain the correlation degree between the target high-confidence detection box and the target predicted tracking box based on the area intersection value and the height intersection value.

[0207] In one embodiment, the matching module 1603 is specifically used to: obtain the area of ​​the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtain the area of ​​the union region of the target high-confidence detection box and the target predicted tracking box; and use the ratio of the area of ​​the intersection region to the area of ​​the union region as the area intersection-union value.

[0208] In one embodiment, the matching module 1603 is specifically used to: obtain the height of the intersection region of the target high-confidence detection box and the target predicted tracking box, and obtain the height of the union region of the target high-confidence detection box and the target predicted tracking box; and use the height of the intersection region and the height of the area of ​​the union region as the height intersection-union value.

[0209] In one embodiment, the matching result includes a second matching result; the confidence category includes a low-confidence category, where the detection confidence of a low-confidence category is less than a first confidence threshold and greater than a second confidence threshold; the matching module 1603 is specifically used to: obtain the detection box of the target object corresponding to the detection confidence of the low-confidence category as a low-confidence detection box; calculate the correlation between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result; generate multiple second relationship loss matrices based on the correlation calculated based on the unmatched predicted tracking boxes included in the first matching result; and determine the second matching result based on each second relationship loss matrix; wherein the second matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0210] In one embodiment, the matching result includes a third matching result; the matching module 1603 is specifically used to: calculate the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library; wherein, the tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result; generate multiple third relationship loss matrices based on the correlation calculated based on the historical predicted tracking boxes; determine the third matching result based on each third relationship loss matrix; wherein, the third matching result includes the successfully matched detection boxes and predicted tracking box group, the unmatched detection boxes, and the unmatched historical predicted tracking boxes.

[0211] In one embodiment, the matching module 1603 is specifically used to: take each first relation loss matrix, each second relation loss matrix and each third relation loss matrix as target relation loss matrices, and use a preset optimized matching algorithm to filter the correlation of each target relation loss matrix to obtain the first matching result, the second matching result and the third matching result respectively.

[0212] In one embodiment, the tracking trajectory library also includes the lifecycle value corresponding to each historical predicted tracking box; after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the device further includes a period accumulation module, used to: cumulatively update the lifecycle value of each historical predicted tracking box, and delete the historical predicted tracking box from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking box is greater than a preset period threshold.

[0213] In one embodiment, the update module 1604 is specifically used to: for the first matching result, the second matching result, and the third matching result that include a successfully matched detection box and a predicted tracking box group, replace the predicted tracking box with the detection box in the detection box and the predicted tracking box group, and use the replaced predicted tracking box as the detection location of the target object.

[0214] In one embodiment, the position detection module 1601 is specifically used to: input the current video frame into a preset detection model for position detection, and obtain the detection confidence scores of the detection boxes corresponding to each target object output by the preset detection model.

[0215] In one embodiment, the position prediction module 1602 is specifically used to: input the current video frame into the preset tracker after initialization and debugging to perform position prediction, and obtain the predicted tracking boxes corresponding to each target object output by the preset tracker.

[0216] In one embodiment, the position prediction module 1602 is specifically used to: determine whether the current video frame is the first video frame in the current detection video; if so, input the current video frame and the detection boxes of each target object into a preset tracker, and use the preset tracker to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier included in other video frames in the current detection video; wherein, the tracking identifier is also used to associate and display with the detection position of the corresponding target object.

[0217] Each module in the aforementioned target tracking and detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0218] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 17 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores target tracking and detection data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a target tracking and detection method.

[0219] Those skilled in the art will understand that Figure 17 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0220] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0221] Position detection is performed on each target object included in the current video frame to obtain the detection bounding box and detection confidence score for each target object; position prediction is performed on each target object included in the current video frame to obtain the predicted tracking bounding box for each target object, and multiple prediction intervals are divided based on each predicted tracking bounding box; wherein each divided prediction interval includes at least one of the predicted tracking bounding boxes; each detection bounding box and each predicted tracking bounding box are matched according to the confidence class of the detection confidence score corresponding to each target object and each prediction interval; the detection position of each target object in the current video frame is updated based on the matching results.

[0222] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the depth value of each predicted tracking box in the current video frame; the depth value is used to characterize the distance of the target object corresponding to the predicted tracking box from the shooting point in the current video frame; sorting the depth values, and referring to the sorting result, dividing the predicted interval into multiple predicted intervals according to each predicted tracking box.

[0223] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each predicted tracking box, calculates the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame to obtain the depth value corresponding to the predicted tracking box.

[0224] In one embodiment, when the processor executes the computer program, it further performs the following steps: arranging the depth values ​​in ascending order to obtain a sorting result; traversing the prediction tracking boxes corresponding to each depth value according to the sorting result, and dividing the prediction interval based on the center point of each currently traversed prediction tracking box to obtain multiple prediction intervals.

[0225] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the predicted tracking boxes located within a circular range centered on the center point and with a preset distance value as the radius; and determining the prediction interval corresponding to the currently traversed predicted tracking box based on the predicted tracking boxes within each circular range.

[0226] In one embodiment, when the processor executes the computer program, it further implements the following steps: for the currently traversed prediction tracking box, it detects whether the currently traversed prediction tracking box is located within a determined prediction interval; if not, it performs the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

[0227] In one embodiment, the matching result includes a first matching result; the confidence category includes a high-confidence category, and the detection confidence of the high-confidence category is greater than a first confidence threshold; when the processor executes the computer program, it further implements the following steps: obtaining the detection box of the target object corresponding to the detection confidence of the high-confidence category as a high-confidence detection box; calculating the correlation degree between each high-confidence detection box and each predicted tracking box included in each prediction interval; for each prediction interval, generating a first relational loss matrix corresponding to the prediction interval based on the correlation degree calculated based on each predicted tracking box included in the prediction interval; determining the first matching result based on the first relational loss matrix corresponding to each prediction interval; wherein, the first matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0228] In one embodiment, when the processor executes the computer program, it further performs the following steps: for the target high-confidence detection box and the target prediction tracking box to be calculated, calculate the area intersection value between the target high-confidence detection box and the target prediction tracking box and calculate the height intersection value between the target high-confidence detection box and the target prediction tracking box; and obtain the correlation degree between the target high-confidence detection box and the target prediction tracking box based on the area intersection value and the height intersection value.

[0229] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the area of ​​the intersection region of the target high-confidence detection box and the target prediction tracking box, and obtaining the area of ​​the union region of the target high-confidence detection box and the target prediction tracking box; and using the ratio of the area of ​​the intersection region to the area of ​​the union region as the area intersection-union value.

[0230] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the height of the intersection region of the target high-confidence detection box and the target prediction tracking box, and obtaining the height of the union region of the target high-confidence detection box and the target prediction tracking box; using the height of the intersection region and the height of the area of ​​the union region as the height intersection-union value.

[0231] In one embodiment, the matching result includes a second matching result; the confidence category includes a low-confidence category, wherein the detection confidence of the low-confidence category is less than a first confidence threshold and greater than a second confidence threshold; when the processor executes the computer program, it further implements the following steps: obtaining the detection box of the target object corresponding to the detection confidence of the low-confidence category as a low-confidence detection box; calculating the correlation degree between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result; generating multiple second relationship loss matrices based on the correlation degree calculated based on the unmatched predicted tracking boxes included in the first matching result; determining the second matching result based on each second relationship loss matrix; wherein the second matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0232] In one embodiment, the matching result includes a third matching result; when the processor executes the computer program, it further implements the following steps: calculating the correlation degree between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library; wherein, the tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result; generating multiple third relation loss matrices based on the correlation degree calculated based on the historical predicted tracking boxes; determining the third matching result based on each third relation loss matrix; wherein, the third matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched historical predicted tracking boxes.

[0233] In one embodiment, when the processor executes the computer program, it further implements the following steps: taking each first relation loss matrix, each second relation loss matrix, and each third relation loss matrix as target relation loss matrices, and using a preset optimization matching algorithm to filter the correlation of each target relation loss matrix to obtain the first matching result, the second matching result, and the third matching result respectively.

[0234] In one embodiment, the tracking trajectory library also includes the lifecycle value corresponding to each historical predicted tracking box; after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the processor further implements the following steps when executing the computer program: cumulatively updating the lifecycle value of each historical predicted tracking box, and deleting the historical predicted tracking box from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking box is greater than a preset period threshold.

[0235] In one embodiment, when the processor executes the computer program, it further performs the following steps: for the first matching result, the second matching result, and the third matching result that include a successfully matched detection box and a predicted tracking box group, the predicted tracking box is replaced with a detection box in the detection box and the predicted tracking box group, and the replaced predicted tracking box is used as the detection position of the target object.

[0236] In one embodiment, when the processor executes the computer program, it further performs the following steps: inputting the current video frame into a preset detection model for position detection, and obtaining the detection confidence scores of each detection box corresponding to each target object output by the preset detection model.

[0237] In one embodiment, when the processor executes the computer program, it further performs the following steps: inputting the current video frame into a preset tracker after initialization and debugging to perform position prediction, and obtaining the predicted tracking boxes corresponding to each target object output by the preset tracker.

[0238] In one embodiment, when the processor executes the computer program, it further implements the following steps: determining whether the current video frame is the first video frame in the current detection video; if so, inputting the current video frame and the detection boxes of each target object into the preset tracker, and using the preset tracker to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier included in other video frames in the current detection video; wherein, the tracking identifier is also used to associate and display with the detection position of the corresponding target object.

[0239] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0240] Position detection is performed on each target object included in the current video frame to obtain the detection bounding box and detection confidence score for each target object; position prediction is performed on each target object included in the current video frame to obtain the predicted tracking bounding box for each target object, and multiple prediction intervals are divided based on each predicted tracking bounding box; wherein each divided prediction interval includes at least one of the predicted tracking bounding boxes; each detection bounding box and each predicted tracking bounding box are matched according to the confidence class of the detection confidence score corresponding to each target object and each prediction interval; the detection position of each target object in the current video frame is updated based on the matching results.

[0241] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the depth value of each predicted tracking box in the current video frame; the depth value is used to characterize the distance of the target object corresponding to the predicted tracking box from the shooting point in the current video frame; sorting the depth values, and referring to the sorting results, dividing the predicted intervals into multiple predicted intervals according to each predicted tracking box.

[0242] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each predicted tracking box, calculates the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame to obtain the depth value corresponding to the predicted tracking box.

[0243] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: arranging the depth values ​​in ascending order to obtain a sorting result; traversing the prediction tracking boxes corresponding to each depth value according to the sorting result, and dividing the prediction interval based on the center point of each currently traversed prediction tracking box to obtain multiple prediction intervals.

[0244] In one embodiment, when the computer program is executed by a processor, it further implements the following steps: obtaining the predicted tracking boxes located within a circular range centered at the center point and with a preset distance value as the radius; and determining the prediction interval corresponding to the currently traversed predicted tracking box based on the predicted tracking boxes within each circular range.

[0245] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: for the currently traversed prediction tracking box, detecting whether the currently traversed prediction tracking box is located within a determined prediction interval; if not, performing the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

[0246] In one embodiment, the matching result includes a first matching result; the confidence category includes a high-confidence category, and the detection confidence of the high-confidence category is greater than a first confidence threshold; when the computer program is executed by the processor, it further implements the following steps: obtaining the detection box of the target object corresponding to the detection confidence of the high-confidence category as a high-confidence detection box; calculating the correlation degree between each high-confidence detection box and each predicted tracking box included in each prediction interval; for each prediction interval, generating a first relational loss matrix corresponding to the prediction interval based on the correlation degree calculated based on each predicted tracking box included in the prediction interval; determining the first matching result based on the first relational loss matrix corresponding to each prediction interval; wherein, the first matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0247] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: for the target high-confidence detection box and the target prediction tracking box to be calculated, calculate the area intersection value between the target high-confidence detection box and the target prediction tracking box and calculate the height intersection value between the target high-confidence detection box and the target prediction tracking box; and obtain the correlation degree between the target high-confidence detection box and the target prediction tracking box based on the area intersection value and the height intersection value.

[0248] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the area of ​​the intersection region of the target high-confidence detection box and the target prediction tracking box, and obtaining the area of ​​the union region of the target high-confidence detection box and the target prediction tracking box; and using the ratio of the area of ​​the intersection region to the area of ​​the union region as the area intersection-union value.

[0249] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the height of the intersection region of the target high-confidence detection box and the target prediction tracking box, and obtaining the height of the union region of the target high-confidence detection box and the target prediction tracking box; using the height of the intersection region and the height of the area of ​​the union region as the height intersection-union value.

[0250] In one embodiment, the matching result includes a second matching result; the confidence category includes a low-confidence category, wherein the detection confidence of the low-confidence category is less than a first confidence threshold and greater than a second confidence threshold; when the computer program is executed by the processor, it further implements the following steps: obtaining the detection box of the target object corresponding to the detection confidence of the low-confidence category as a low-confidence detection box; calculating the correlation degree between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result; generating multiple second relationship loss matrices based on the correlation degree calculated based on the unmatched predicted tracking boxes included in the first matching result; determining the second matching result based on each second relationship loss matrix; wherein the second matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0251] In one embodiment, the matching result includes a third matching result; when the computer program is executed by the processor, it further implements the following steps: calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library; wherein the tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result; generating multiple third relation loss matrices based on the correlation calculated based on the historical predicted tracking boxes; determining the third matching result based on each third relation loss matrix; wherein the third matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched historical predicted tracking boxes.

[0252] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: taking each first relation loss matrix, each second relation loss matrix and each third relation loss matrix as target relation loss matrices respectively, and using a preset optimization matching algorithm to filter the correlation of each target relation loss matrix to obtain the first matching result, the second matching result and the third matching result respectively.

[0253] In one embodiment, the tracking trajectory library also includes the lifecycle value corresponding to each historical predicted tracking box; after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the computer program, when executed by the processor, further implements the following steps: cumulatively updating the lifecycle value of each historical predicted tracking box, and deleting the historical predicted tracking box from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking box is greater than a preset period threshold.

[0254] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for the first matching result, the second matching result, and the third matching result that include a successfully matched detection box and a predicted tracking box group, the predicted tracking box is replaced with a detection box in the detection box and the predicted tracking box group, and the replaced predicted tracking box is used as the detection location of the target object.

[0255] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: inputting the current video frame into a preset detection model for position detection, and obtaining the detection confidence scores of each detection box corresponding to each target object output by the preset detection model.

[0256] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: inputting the current video frame into a preset tracker after initialization and debugging to perform position prediction, and obtaining the predicted tracking boxes corresponding to each target object output by the preset tracker.

[0257] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: determining whether the current video frame is the first video frame in the current detection video; if so, inputting the current video frame and the detection boxes of each target object into the preset tracker, and using the preset tracker to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier included in other video frames in the current detection video; wherein, the tracking identifier is also used to associate and display with the detection position of the corresponding target object.

[0258] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0259] Position detection is performed on each target object included in the current video frame to obtain the detection bounding box and detection confidence score for each target object; position prediction is performed on each target object included in the current video frame to obtain the predicted tracking bounding box for each target object, and multiple prediction intervals are divided based on each predicted tracking bounding box; wherein each divided prediction interval includes at least one of the predicted tracking bounding boxes; each detection bounding box and each predicted tracking bounding box are matched according to the confidence class of the detection confidence score corresponding to each target object and each prediction interval; the detection position of each target object in the current video frame is updated based on the matching results.

[0260] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the depth value of each predicted tracking box in the current video frame; the depth value is used to characterize the distance of the target object corresponding to the predicted tracking box from the shooting point in the current video frame; sorting the depth values, and referring to the sorting results, dividing the predicted intervals into multiple predicted intervals according to each predicted tracking box.

[0261] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each predicted tracking box, calculates the vertical distance between the bottom edge of the predicted tracking box and the bottom edge of the current video frame to obtain the depth value corresponding to the predicted tracking box.

[0262] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: arranging the depth values ​​in ascending order to obtain a sorting result; traversing the prediction tracking boxes corresponding to each depth value according to the sorting result, and dividing the prediction interval based on the center point of each currently traversed prediction tracking box to obtain multiple prediction intervals.

[0263] In one embodiment, when the computer program is executed by a processor, it further implements the following steps: obtaining the predicted tracking boxes located within a circular range centered at the center point and with a preset distance value as the radius; and determining the prediction interval corresponding to the currently traversed predicted tracking box based on the predicted tracking boxes within each circular range.

[0264] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: for the currently traversed prediction tracking box, detecting whether the currently traversed prediction tracking box is located within a determined prediction interval; if not, performing the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

[0265] In one embodiment, the matching result includes a first matching result; the confidence category includes a high-confidence category, and the detection confidence of the high-confidence category is greater than a first confidence threshold; when the computer program is executed by the processor, it further implements the following steps: obtaining the detection box of the target object corresponding to the detection confidence of the high-confidence category as a high-confidence detection box; calculating the correlation degree between each high-confidence detection box and each predicted tracking box included in each prediction interval; for each prediction interval, generating a first relational loss matrix corresponding to the prediction interval based on the correlation degree calculated based on each predicted tracking box included in the prediction interval; determining the first matching result based on the first relational loss matrix corresponding to each prediction interval; wherein, the first matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0266] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: for the target high-confidence detection box and the target prediction tracking box to be calculated, calculate the area intersection value between the target high-confidence detection box and the target prediction tracking box and calculate the height intersection value between the target high-confidence detection box and the target prediction tracking box; and obtain the correlation degree between the target high-confidence detection box and the target prediction tracking box based on the area intersection value and the height intersection value.

[0267] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the area of ​​the intersection region of the target high-confidence detection box and the target prediction tracking box, and obtaining the area of ​​the union region of the target high-confidence detection box and the target prediction tracking box; and using the ratio of the area of ​​the intersection region to the area of ​​the union region as the area intersection-union value.

[0268] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the height of the intersection region of the target high-confidence detection box and the target prediction tracking box, and obtaining the height of the union region of the target high-confidence detection box and the target prediction tracking box; using the height of the intersection region and the height of the area of ​​the union region as the height intersection-union value.

[0269] In one embodiment, the matching result includes a second matching result; the confidence category includes a low-confidence category, wherein the detection confidence of the low-confidence category is less than a first confidence threshold and greater than a second confidence threshold; when the computer program is executed by the processor, it further implements the following steps: obtaining the detection box of the target object corresponding to the detection confidence of the low-confidence category as a low-confidence detection box; calculating the correlation degree between each low-confidence detection box and the unmatched predicted tracking boxes included in the first matching result; generating multiple second relationship loss matrices based on the correlation degree calculated based on the unmatched predicted tracking boxes included in the first matching result; determining the second matching result based on each second relationship loss matrix; wherein the second matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

[0270] In one embodiment, the matching result includes a third matching result; when the computer program is executed by the processor, it further implements the following steps: calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library; wherein the tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result; generating multiple third relation loss matrices based on the correlation calculated based on the historical predicted tracking boxes; determining the third matching result based on each third relation loss matrix; wherein the third matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched historical predicted tracking boxes.

[0271] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: taking each first relation loss matrix, each second relation loss matrix and each third relation loss matrix as target relation loss matrices respectively, and using a preset optimization matching algorithm to filter the correlation of each target relation loss matrix to obtain the first matching result, the second matching result and the third matching result respectively.

[0272] In one embodiment, the tracking trajectory library also includes the lifecycle value corresponding to each historical predicted tracking box; after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the computer program, when executed by the processor, further implements the following steps: cumulatively updating the lifecycle value of each historical predicted tracking box, and deleting the historical predicted tracking box from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking box is greater than a preset period threshold.

[0273] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for the first matching result, the second matching result, and the third matching result that include a successfully matched detection box and a predicted tracking box group, the predicted tracking box is replaced with a detection box in the detection box and the predicted tracking box group, and the replaced predicted tracking box is used as the detection location of the target object.

[0274] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: inputting the current video frame into a preset detection model for position detection, and obtaining the detection confidence scores of each detection box corresponding to each target object output by the preset detection model.

[0275] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: inputting the current video frame into a preset tracker after initialization and debugging to perform position prediction, and obtaining the predicted tracking boxes corresponding to each target object output by the preset tracker.

[0276] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: determining whether the current video frame is the first video frame in the current detection video; if so, inputting the current video frame and the detection boxes of each target object into the preset tracker, and using the preset tracker to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier included in other video frames in the current detection video; wherein, the tracking identifier is also used to associate and display with the detection position of the corresponding target object.

[0277] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0278] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0279] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0280] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A target tracking and detection method, characterized in that, The method includes: Position detection is performed on each target object included in the current video frame to obtain the detection box and detection confidence score corresponding to each target object; Position prediction is performed on each target object included in the current video frame to obtain a prediction tracking box corresponding to each target object, and multiple prediction intervals are divided according to each prediction tracking box; wherein each of the divided prediction intervals includes at least one prediction tracking box. Based on the confidence category of the detection confidence level corresponding to each target object and each prediction interval, the detection box and the prediction tracking box are matched. Update the detection position of each target object in the current video frame according to the matching result; The step of dividing multiple prediction intervals according to each prediction tracking box includes: Obtain the depth value of each of the predicted tracking boxes in the current video frame; the depth value is used to characterize the distance between the target object corresponding to the predicted tracking box and the shooting point in the current video frame. The depth values ​​are sorted, and based on the sorting results, multiple prediction intervals are divided according to the prediction tracking boxes. The process of sorting the depth values ​​and, with reference to the sorting results, dividing the predicted tracking boxes into multiple predicted intervals includes: Arrange the depth values ​​in ascending order to obtain the sorting result; According to the sorting result, the predicted tracking boxes corresponding to each depth value are traversed, and the prediction intervals are divided based on the center point of each currently traversed predicted tracking box to obtain multiple prediction intervals.

2. The method according to claim 1, characterized in that, The method further includes: For the currently traversed predicted tracking box, detect whether the currently traversed predicted tracking box is located within the already determined prediction interval; If not, then proceed with the step of dividing the prediction interval based on the center point of the currently traversed prediction tracking box.

3. The method according to claim 1, characterized in that, The matching result includes a first matching result; the confidence category includes a high-confidence category, where the detection confidence of the high-confidence category is greater than a first confidence threshold; the matching process of each detection box and each predicted tracking box based on the confidence category of the detection confidence corresponding to each target object and each prediction interval includes: Obtain the detection bounding box of the target object corresponding to the detection confidence of the high confidence category as the high confidence detection bounding box; Calculate the correlation between each of the high-confidence detection boxes and each of the predicted tracking boxes included in each of the predicted intervals; For each prediction interval, a first relation loss matrix corresponding to the prediction interval is generated based on the correlation degree calculated based on each prediction tracking box included in the prediction interval. The first matching result is determined based on the first relational loss matrix corresponding to each prediction interval; wherein the first matching result includes a group of successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

4. The method according to claim 3, characterized in that, The step of calculating the correlation between each of the high-confidence detection boxes and each predicted tracking box included in each of the predicted intervals includes: For the target high-confidence detection box and the target predicted tracking box to be calculated, calculate the area intersection value between the target high-confidence detection box and the target predicted tracking box, and calculate the height intersection value between the target high-confidence detection box and the target predicted tracking box; The correlation between the target high-confidence detection box and the target prediction tracking box is obtained based on the area intersection value and the height intersection value.

5. The method according to claim 3, characterized in that, The matching result includes a second matching result; the confidence category includes a low confidence category, where the detection confidence of the low confidence category is less than a first confidence threshold and greater than a second confidence threshold; the matching process of each detection box and each predicted tracking box based on the confidence category of the detection confidence corresponding to each target object and each prediction interval includes: Obtain the detection bounding box of the target object corresponding to the detection confidence of the low confidence category as the low confidence detection bounding box; Calculate the correlation between each of the low-confidence detection boxes and the unmatched predicted tracking boxes included in the first matching result; Based on the correlation degree calculated from the unmatched predicted tracking boxes included in the first matching result, multiple second relationship loss matrices are generated. The second matching result is determined based on each of the second relation loss matrices; wherein the second matching result includes successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched predicted tracking boxes.

6. The method according to claim 5, characterized in that, The matching result includes a third matching result; the matching process of each detection box and each predicted tracking box based on the confidence category of the detection confidence level corresponding to each target object and each prediction interval includes: Calculate the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library; wherein, the tracking trajectory library includes multiple historical video frames corresponding to the unmatched predicted tracking boxes included in the second matching result. Based on the correlation degree calculated from the historical prediction tracking box, multiple third-relationship loss matrices are generated; The third matching result is determined based on the loss matrices of each third relationship; wherein the third matching result includes successfully matched detection boxes and predicted tracking boxes, unmatched detection boxes, and unmatched historical predicted tracking boxes.

7. The method according to claim 6, characterized in that, The tracking trajectory library also includes the lifecycle value corresponding to each of the historical predicted tracking boxes; after calculating the correlation between the unmatched detection boxes included in the second matching result and the historical predicted tracking boxes included in the tracking trajectory library, the method further includes: The lifecycle values ​​of each historical predicted tracking frame are cumulatively updated, and the historical predicted tracking frame is deleted from the tracking trajectory library if the updated lifecycle value of the historical predicted tracking frame is greater than a preset period threshold.

8. The method according to claim 6, characterized in that, The step of updating the detection position of each target object in the current video frame according to the matching result includes: For the first matching result, the second matching result, and the third matching result that include a successfully matched detection box and a predicted tracking box group, the predicted tracking box is replaced by the detection box in the detection box and the predicted tracking box group, and the replaced predicted tracking box is used as the detection position of the target object.

9. The method according to claim 1, characterized in that, The step of performing position detection on each target object included in the current video frame to obtain the detection box and detection confidence score corresponding to each target object includes: The current video frame is input into a preset detection model for position detection, and the detection confidence scores of the detection boxes corresponding to each target object output by the preset detection model are obtained. The step of predicting the position of each target object included in the current video frame to obtain a prediction tracking box corresponding to each target object includes: The current video frame is input into the preset tracker after initialization and debugging for position prediction, and the predicted tracking box corresponding to each target object is obtained from the output of the preset tracker.

10. The method according to claim 9, characterized in that, The process of initializing and debugging the preset tracker includes: Determine whether the current video frame is the first video frame in the currently detected video; If so, the current video frame and the detection boxes of each target object are input into the preset tracker, and the preset tracker is used to generate tracking identifiers corresponding to each target object, so that the preset tracker can predict the position of the target objects corresponding to each tracking identifier in other video frames in the current detection video; The tracking identifier is also used to associate and display the corresponding detection location of the target object.

11. A target tracking and detection device, characterized in that, The device includes: The position detection module is used to detect the position of each target object included in the current video frame, and obtain the detection box and detection confidence score corresponding to each target object respectively; The position prediction module is used to predict the position of each target object included in the current video frame, obtain the prediction tracking box corresponding to each target object, and divide multiple prediction intervals according to each prediction tracking box; wherein each prediction interval includes at least one prediction tracking box. The matching module is used to match each detection box and each prediction tracking box according to the confidence category of the detection confidence level corresponding to each target object and each prediction interval; The update module is used to update the detection position of each target object in the current video frame according to the matching result; The step of dividing multiple prediction intervals according to each prediction tracking box includes: Obtain the depth value of each of the predicted tracking boxes in the current video frame; the depth value is used to characterize the distance between the target object corresponding to the predicted tracking box and the shooting point in the current video frame. The depth values ​​are sorted, and based on the sorting results, multiple prediction intervals are divided according to the prediction tracking boxes. The process of sorting the depth values ​​and, with reference to the sorting results, dividing the predicted tracking boxes into multiple predicted intervals includes: Arrange the depth values ​​in ascending order to obtain the sorting result; According to the sorting result, the predicted tracking boxes corresponding to each depth value are traversed, and the prediction intervals are divided based on the center point of each currently traversed predicted tracking box to obtain multiple prediction intervals.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

13. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images