Target trajectory matching method, electronic device and computer readable storage medium

Abstract

This application discloses a target trajectory matching method, an electronic device, and a computer-readable storage medium, comprising: acquiring all initially selected targets and their corresponding detection boxes in the current frame; determining all obtained target trajectories and their corresponding historical features and historical boxes; determining a first matching result between the initially selected targets and target trajectories based on the visual features of each initially selected target and the historical features of each target trajectory; determining a second matching result between the initially selected targets and target trajectories based on the position and size of the detection boxes of each initially selected target and the historical boxes of each target trajectory, as well as their overlapping areas; filtering the trajectory matching results of the initially selected targets based on the positional relationship between the matched detection boxes and historical boxes in the second matching result; acquiring other unmatched targets; and determining the trajectory matching results of other targets based on the intersection-union ratio (IUU) between the detection boxes and historical boxes of other targets. This scheme can improve the efficiency and accuracy of target trajectory matching.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a target trajectory matching method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Continuous target trajectory matching is a crucial branch of computer vision, primarily tasked with detecting targets within video frames and performing cross-frame matching and association. However, current technologies typically involve complex trajectory prediction calculations before matching, resulting in high computational load and low matching efficiency. Typical prediction methods include Kalman filtering. Furthermore, conventional matching methods suffer from low accuracy in complex scenes with occlusion or changing lighting conditions. Therefore, improving the efficiency and accuracy of target trajectory matching has become a pressing issue. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a target trajectory matching method, electronic device, and computer-readable storage medium that can improve the efficiency and accuracy of target trajectory matching.

[0004] To address the aforementioned technical problems, this application provides a target trajectory matching method, comprising: acquiring all initially selected targets and their corresponding detection boxes in the current frame; determining all obtained target trajectories and their corresponding historical features and historical boxes; determining a first matching result between the initially selected targets and the target trajectories based on the visual features of each initially selected target and the historical features of each target trajectory; determining a second matching result between the initially selected targets and the target trajectories based on the respective positions and sizes of the detection boxes of each initially selected target and the historical boxes of each target trajectory, as well as their overlapping areas; filtering the trajectory matching results of the initially selected targets from the first matching result and the second matching result based on the positional relationship between the matched detection boxes and historical boxes in the second matching result; acquiring other unmatched targets in the current frame; and determining the trajectory matching results of the other targets based on the intersection-union ratio (IUU) between the detection boxes and historical boxes of the other targets.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0007] The beneficial effects of this application are as follows: Unlike existing technologies, this application obtains all preliminary targets and their corresponding detection boxes from the current frame, and determines all target trajectories obtained up to the current time, as well as the historical features and historical boxes corresponding to the target trajectories. Based on the visual features of each preliminary target and the historical features of each target trajectory, the preliminary targets and target trajectories are matched according to the visual features to obtain a first matching result. Based on the position and size of the detection boxes of each preliminary target and the historical boxes of each target trajectory, as well as the overlapping area between the detection boxes and historical boxes, the preliminary targets and target trajectories are matched according to the spatiotemporal position and size between the bounding boxes to obtain a second matching result, thus eliminating the need for position prediction for matching. Based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, the final trajectory matching result is selected from the first and second matching results for the preliminary targets. This allows for the selection of a better matching result from two different dimensions of matching results based on the motion level reflected by the positional relationship, reducing the impact of factors such as occlusion or changes in illumination on the accuracy of the matching results and improving the accuracy of trajectory matching. The system acquires other unmatched targets in the current frame. Based on the intersection-union ratio (IUU) between the detection boxes of other targets and all historical boxes, it matches the remaining unmatched targets with their trajectories to obtain the trajectory matching results of other targets, ensuring the accuracy and comprehensiveness of trajectory matching. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one implementation method of the target trajectory matching method of this application; Figure 2 This is a flowchart illustrating another implementation of the target trajectory matching method of this application; Figure 3 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 4 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0011] The target trajectory matching method provided in this application is used to identify and match the trajectory of targets in video frames, and its corresponding execution subject is a processing unit capable of data processing.

[0012] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the target trajectory matching method of this application. The method includes: S101: Obtain all initially selected targets and their corresponding detection boxes in the current frame, and determine all obtained target trajectories and their corresponding historical features and historical boxes.

[0013] Specifically, all initially selected targets and their corresponding detection boxes are obtained from the current frame, and all target trajectories obtained up to the current time, as well as the historical features and historical boxes corresponding to the target trajectories, are determined.

[0014] It should be noted that the current frame is the latest video frame in the video stream up to the present. When the target is matched with the target trajectory, the latest position of the target in the video frame will be merged into the target trajectory. The historical features of the target trajectory are obtained based on the visual features matched by the target corresponding to the target trajectory. The historical bounding box of the target trajectory corresponds to the detection bounding box of the target corresponding to the target in the historical video frames.

[0015] In one embodiment, all detected targets and their corresponding detection boxes in the current frame are acquired, the confidence level of each detection box is determined, targets with confidence levels higher than a confidence threshold are selected as initial targets, and the historical features matched by the targets corresponding to the obtained target trajectories are determined, along with the corresponding target boxes in the most recent video frames, as historical boxes. The historical features are obtained by fusing visual features of the targets corresponding to the target trajectories in historical video frames.

[0016] In one embodiment, all detected targets in the current frame are acquired and their occlusion rates are determined. Targets with occlusion rates below an occlusion threshold are selected as initial targets. Detection boxes of the initial targets are acquired. Historical features matching the obtained target trajectories are determined, along with the corresponding target bounding boxes in the most recent video frames, which are then used as historical bounding boxes. The historical features are obtained based on the visual features corresponding to unoccluded targets in the most recent subset of video frames up to the present.

[0017] In some implementation scenarios, targets in video frames are detected using an object detection model. This model includes a neural network layer for feature extraction and target recognition. After pre-training, the model outputs the targets in the video frame and their corresponding bounding boxes. The detected targets output a confidence score based on the quality of the targets within the bounding boxes. Targets with confidence scores higher than a certain threshold are selected as initial targets.

[0018] In some implementation scenarios, targets in video frames are obtained using a target recognition model. The target recognition model includes a classification network for classifying and recognizing the required targets. The target recognition model is pre-trained and can output the required type of target. Based on the visual features of the target, the occlusion rate of the target is determined, and targets with an occlusion rate lower than the occlusion threshold are selected as initial targets.

[0019] S102: Based on the visual features of each preliminary target and the historical features of each target trajectory, determine the first matching result between the preliminary target and the target trajectory. Based on the position and size of the detection box of each preliminary target and the historical box of each target trajectory, as well as the overlapping area between them, determine the second matching result between the preliminary target and the target trajectory.

[0020] Specifically, based on the visual features of each preliminary target and the historical features of each target trajectory, the preliminary targets and target trajectories are matched according to the visual features to obtain the first matching result.

[0021] Understandably, based on the visual features of each initial target and the historical features of each target trajectory, the visual feature similarity between each initial target and each target trajectory is determined. Based on the visual feature similarity, the target trajectory with the optimal matching cost is selected for at least some of the initial targets, and the first matching result between the initial targets and the target trajectories is obtained. The first matching result includes the matched trajectories, unmatched trajectories and unmatched detection boxes obtained according to the visual features.

[0022] Furthermore, based on the position and size of the detection box of each initial target and the historical box of each target trajectory, as well as the overlapping area between the detection box and the historical box, the initial targets and target trajectories are matched according to the spatiotemporal position and size between the bounding boxes to obtain a second matching result, thus enabling matching without the need for position prediction.

[0023] In one embodiment, for each detection box of a preliminary target and the historical box corresponding to each target trajectory, the overlapping area and intersection area corresponding to the overlapping area between the detection box and the historical box are obtained, the center distance and area difference between the detection box and the historical box are determined, and the bounding box similarity index between the detection box and the historical box is determined by using the proportion of the overlapping area and the intersection area, as well as the center distance and area difference. Based on the bounding box similarity index, the target trajectory with the optimal matching cost is selected for at least some of the preliminary targets, and the second matching result between the preliminary targets and the target trajectory is obtained.

[0024] In one embodiment, for each detection box of a preliminary target and the historical box corresponding to each target trajectory, the overlapping area between the detection box and the historical box, the center position of each detection box and the historical box, and the border size of each detection box and the historical box are obtained. Based on the proportion of the overlapping area to the bounding box, and the difference between the distance between the center positions and the border size, the bounding box similarity index between the detection box and the historical box is determined. Based on the bounding box similarity index, the target trajectory with the optimal matching cost is selected for at least some of the preliminary targets, and a second matching result between the preliminary targets and the target trajectories is obtained.

[0025] Understandably, the second matching result includes matched trajectories, unmatched trajectories, and unmatched detection boxes obtained according to the spatiotemporal position and size between the bounding boxes.

[0026] It should be noted that the first and second matching results can be used to determine the matching result with the lowest cost using the Hungarian algorithm or the bipartite graph matching algorithm.

[0027] S103: Based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, filter the trajectory matching results of the initial target from the first matching result and the second matching result.

[0028] Specifically, based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, the final trajectory matching result is selected from the first and second matching results for the initial target.

[0029] Understandably, by selecting the better matching result from two different dimensions based on the level of motion reflected by the positional relationship, the accuracy of the matching result can be reduced due to factors such as occlusion or changes in lighting, thereby improving the accuracy of trajectory matching.

[0030] In one embodiment, the intersection-union ratio (CIRR) between the matched detection boxes and historical boxes in the second matching result is obtained. When the CIRR is greater than a preset CIRR threshold, the second matching result is used as the trajectory matching result corresponding to the target. When the CIRR is less than or equal to the preset CIRR threshold, the first matching result is used as the trajectory matching result corresponding to the target. When the target only obtains the first matching result, the first matching result is used as the trajectory matching result corresponding to the target.

[0031] In one embodiment, the center distance between the matched detection box and the historical box in the second matching result is obtained. When the target only obtains the first matching result or the first matching result and the second matching result do not conflict, the first matching result is used as the trajectory matching result corresponding to the target. When the first matching result and the second matching result conflict, if the center distance is less than the center distance threshold, the second matching result is used as the trajectory matching result corresponding to the target; otherwise, the first matching result is used as the trajectory matching result corresponding to the target.

[0032] S104: Obtain other unmatched targets in the current frame, and determine the trajectory matching results of other targets based on the intersection-union ratio between the detection boxes and historical boxes of other targets.

[0033] Specifically, other unmatched targets in the current frame are obtained. Based on the intersection-union ratio between the detection boxes of other targets and all historical boxes, the remaining unmatched targets are matched with the target trajectories to obtain the trajectory matching results of other targets, ensuring the accuracy and comprehensiveness of trajectory matching.

[0034] It should be noted that other unmatched targets in the current frame include targets that were not selected as initial targets, as well as targets among the initial targets that failed to match the target trajectory.

[0035] In one embodiment, a matching cost matrix is ​​constructed based on the cross-union ratio (CURRR) between the detection boxes of other targets and the historical boxes. The cost in the matching cost matrix is ​​negatively correlated with the CURRR. Hungarian matching is performed based on the matching cost matrix to obtain the matching results between other targets and target trajectories.

[0036] In one implementation, based on the cross-union ratio (CUR) between the detection boxes and historical boxes of other targets, other targets with CURs greater than a screening threshold are filtered out, and a matching cost matrix is ​​constructed for the filtered targets. The cost in the matching cost matrix is ​​negatively correlated with the CUR. Hungarian matching is performed based on the matching cost matrix to obtain the matching results between other targets and target trajectories.

[0037] It is understandable that the final matching result in the current frame is the intersection of the trajectory matching results of the initially selected target and other targets. Furthermore, the trajectory matching result of the current frame can be used to update the latest target trajectory, as well as the historical features and bounding boxes corresponding to the target trajectory. The trajectory matching result can be used for target tracking or target clustering, thereby managing and labeling targets in specific scenarios.

[0038] The above scheme obtains all preliminary targets and their corresponding detection boxes from the current frame, and determines all target trajectories obtained up to the current time, as well as their corresponding historical features and historical boxes. Based on the visual features of each preliminary target and the historical features of each target trajectory, the preliminary targets and target trajectories are matched according to the visual features to obtain a first matching result. Based on the position and size of the detection boxes of each preliminary target and the historical boxes of each target trajectory, as well as the overlapping area between the detection boxes and historical boxes, the preliminary targets and target trajectories are matched according to the spatiotemporal position and size between the bounding boxes to obtain a second matching result, thus eliminating the need for position prediction. Based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, the final trajectory matching result is selected from the first and second matching results for the preliminary targets. This allows for the selection of a better matching result from two different dimensions of matching results based on the motion level reflected by the positional relationship, reducing the impact of factors such as occlusion or changes in lighting on the accuracy of the matching results and improving the accuracy of trajectory matching. The system acquires other unmatched targets in the current frame. Based on the intersection-union ratio (IUU) between the detection boxes of other targets and all historical boxes, it matches the remaining unmatched targets with their trajectories to obtain the trajectory matching results of other targets, ensuring the accuracy and comprehensiveness of trajectory matching.

[0039] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the target trajectory matching method of this application, the method comprising: S201: Obtain all initially selected targets and their corresponding detection boxes in the current frame, and determine all obtained target trajectories and their corresponding historical features and historical boxes.

[0040] Specifically, all initially selected targets in the current frame and their corresponding detection boxes are obtained, and all target trajectories obtained up to the current time and their corresponding historical features and historical boxes are determined.

[0041] In one embodiment, obtaining all preliminary targets and their corresponding detection boxes in the current frame includes: obtaining the detection boxes corresponding to all targets in the current frame and their corresponding confidence scores; classifying at least some targets into high-confidence targets and low-confidence targets based on the confidence scores; wherein, high-confidence targets are selected as preliminary targets.

[0042] Specifically, the detection boxes and their corresponding confidence levels for all targets in the current frame are obtained. At least some targets are classified according to their confidence levels, and the corresponding targets are divided into high-confidence targets and low-confidence targets. Among them, high-confidence targets are selected as initial targets and are used for trajectory matching first to ensure the accuracy of trajectory matching.

[0043] In some implementation scenarios, confidence levels correspond to confidence thresholds. Targets with confidence levels greater than the confidence threshold are considered high-confidence targets, while targets with confidence levels less than or equal to the confidence threshold are considered low-confidence targets.

[0044] In some implementation scenarios, confidence levels correspond to a first confidence threshold and a second confidence threshold, and the first confidence threshold is greater than the second confidence threshold. Targets with a confidence level greater than the first confidence threshold are considered high-confidence targets, targets with a confidence level between the first and second confidence thresholds are considered low-confidence targets, and targets with a confidence level less than the second confidence threshold are discarded.

[0045] It should be noted that obtaining the detection boxes and their corresponding confidence scores for all targets in the current frame includes: inputting the video stream into the target detection model, obtaining the detection boxes for all targets in the current frame output by the target detection model, and the confidence score set by the target detection model for each target in the detection box; wherein, the target detection model includes a lightweight network, and the target detection model is trained using a video stream including the training target.

[0046] Specifically, the video stream is input into the target detection model so that the target detection model can detect targets in each video frame. The model outputs the detection boxes corresponding to all targets in the current frame, as well as the confidence level set by the target detection model for each target in the detection box, thereby ensuring the efficiency and reliability of target detection.

[0047] It should be noted that the object detection model includes a lightweight network to improve the real-time performance of detection and has low requirements for device computing memory, making it easy to implement in practical applications. The object detection model is trained using a video stream that includes the training target to ensure the accuracy of the output of the object detection model.

[0048] In some implementation scenarios, the object detection module includes the YOLOv13n network, also known as the YouOnly Look Once v13n network, which is a lightweight network. Different versions of the YOLO network can be selected according to specific needs in different implementation scenarios.

[0049] Understandably, the required set of training targets is obtained, and the object detection module is trained in a supervised manner using a video stream that includes the training targets. The input video stream is processed frame by frame, and the detection box results are output. The detection result of the k-th frame is denoted as... N represents the total number of detection results, including the detection boxes corresponding to all targets in the current frame and the confidence scores of the detection box matches.

[0050] S202: Based on the visual features of each preliminary target and the historical features of each target trajectory, determine the first matching result between the preliminary target and the target trajectory. Based on the position and size of the detection box of each preliminary target and the historical box of each target trajectory, as well as the overlapping area between them, determine the second matching result between the preliminary target and the target trajectory.

[0051] Specifically, based on the visual features of each preliminary target and the historical features of each target trajectory, the preliminary targets and target trajectories are matched according to the visual features to obtain a first matching result. Based on the position and size of the detection box of each preliminary target and the historical box of each target trajectory, as well as the overlapping area between the detection box and the historical box, the preliminary targets and target trajectories are matched according to the spatiotemporal position and size between the bounding boxes to obtain a second matching result.

[0052] In one embodiment, the first matching result between the preliminary target and the target trajectory is determined based on the visual features of each preliminary target and the historical features of each target trajectory, including: constructing a visual cost matrix based on the visual feature similarity between the visual features of each preliminary target and the historical features of each target trajectory; wherein the cost in the visual cost matrix is ​​negatively correlated with the visual similarity; and performing Hungarian matching based on the visual cost matrix to obtain the first matching result between the preliminary target and the target trajectory.

[0053] Specifically, the visual features of each initially selected target and the historical features of each target trajectory are obtained and their similarity is calculated. The visual feature similarity between each visual feature and each historical feature is obtained. The visual cost matrix is ​​constructed using the visual feature similarity, where the cost in the visual cost matrix is ​​negatively correlated with the visual similarity.

[0054] It should be noted that the visual features of the initially selected targets can be extracted using a re-identification (ReID) network to ensure the accuracy of the visual features. The visual feature similarity between the visual features and historical features is usually calculated using the cosine distance between the features. The cost is based on the cosine distance between the visual features of the target within each detection box and the historical features of the target trajectory. Construct a visual cost matrix The above process can be expressed using the following formula: (1) in, The cost in the visual cost matrix is ​​negatively correlated with visual similarity; that is, the lower the visual similarity, the higher the cost. The visual features of the target Historical characteristics of the target trajectory.

[0055] Furthermore, Hungarian matching is performed based on the visual cost matrix to obtain the first matching result between the initially selected target and the target trajectory, thereby ensuring the reliability of the first matching result.

[0056] In one embodiment, a second matching result between the preliminary target and the target trajectory is determined based on the position and size of the detection box of each preliminary target and the historical box of each target trajectory, as well as their overlapping areas. This includes: for each preliminary target's detection box and each target trajectory's historical box, obtaining the overlap height and overlap width corresponding to the overlapping area between the detection box and the historical box; determining the bounding box similarity index between the detection box and the historical box based on the overlap height and overlap width, the center distance between the detection box and the historical box, and the rectangle width and height of the circumscribed rectangle; constructing a motion cost matrix using the bounding box similarity index between each detection box and the historical box; and performing Hungarian matching based on the motion cost matrix to obtain the second matching result between the preliminary target and the target trajectory. The cost in the motion cost matrix is ​​negatively correlated with the bounding box similarity index.

[0057] Specifically, for each detection box and historical box, the overlapping area between the detection box and the historical box is obtained, the overlap height and overlap width of the overlapping area are determined, and the overlap height of the overlapping area is calculated based on the coordinates of the detection box and the coordinates of the nearest historical box in the trajectory cache. and overlap width The above process can be expressed by the following formula: (2) (3) (4) (5) (6) (7) The coordinate system uses the top-left corner of the image as its origin. This represents the coordinates of the bottom right corner of the detection box. This represents the coordinates of the top-left corner of the detection box. Similarly, Represents the coordinates of the bottom right corner of the history box. This represents the coordinates of the top-left corner of the history bounding box. When the detection box and the history bounding box overlap, this represents the overlap height. and overlap width A positive number indicates the overlap height when the detection box and the history box do not overlap. and overlap width It is 0.

[0058] Furthermore, the center distance between the detection box and the historical boxes, as well as the width and height of the bounding rectangles corresponding to the detection box and the historical boxes, are obtained. Height similarity is calculated using the overlap height, width similarity is calculated using the overlap width, and center consistency index is calculated using the center distance and the width and height of the bounding rectangles. The bounding box similarity index (BBSI) is determined by combining multiple results. The above process is expressed by the following formula: (8) (9) (10) (11) in, The bounding box similarity index. For high similarity, For width similarity, As the center consistency index, The height of the historical frame. The width of the history frame. For the height of the detection frame, For the detection frame width, A constant is set to avoid division by zero. Represents the coordinates of the center point of the detection box. Represents the coordinates of the center point of the history box. The height of the bounding rectangle. The width of the bounding rectangle. This is the intersection-union ratio (IUU) between the detected bounding box and the historical bounding boxes. The bounding box similarity index is positively correlated with the overlap height and overlap width, and negatively correlated with the ratio of the center distance to the sum of the width and height of the circumscribed rectangle.

[0059] Furthermore, the motion distance cost is calculated based on the bounding box similarity index. The motion cost matrix is ​​obtained. The above process can be expressed by the following formula: (12) in, Let be the bounding box similarity index between the i-th detection box and the j-th historical box. The cost in the motion cost matrix is ​​negatively correlated with the bounding box similarity index, meaning that the smaller the boundary similarity index, the greater the cost.

[0060] Understandably, Hungarian matching based on the motion cost matrix yields a second matching result between the initial target and the target trajectory. This allows for the comprehensive matching of moving targets by considering overlapping areas, center distance, and shape and size consistency, thus avoiding position prediction and saving computational resources.

[0061] It should be noted that, based on the above cost matrix and Parallel Hungarian matching is performed to obtain high-confidence detection boxes and each tracking trajectory. The two matching results are as follows. The first matching result, based on visual features, includes: (Trajectory matching) (Trajectory mismatch) (Detection box mismatch), the second matching result obtained based on the position and size of the detection box after the target motion includes: (Trajectory matching) (Trajectory mismatch) (Detection box mismatch).

[0062] S203: Based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, filter the trajectory matching results of the initial target from the first matching result and the second matching result.

[0063] Specifically, based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, the final trajectory matching result is selected from the first and second matching results for the initial target.

[0064] In one embodiment, based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, the trajectory matching results of the preliminary target are filtered from the first matching result and the second matching result, including: obtaining the preliminary target and target trajectory matched in the second matching result; generating a motion score for the preliminary target using the intersection-union ratio (IUU) between the detection boxes of the corresponding preliminary target and the historical boxes of the target trajectory; wherein the motion score is negatively correlated with the corresponding IUU; in response to the motion score of the preliminary target being greater than a scoring threshold, the first matching result corresponding to the preliminary target is used as the trajectory matching result; in response to the motion score of the preliminary target being less than or equal to the scoring threshold, the second matching result corresponding to the preliminary target is used as the trajectory matching result; in response to the preliminary target not obtaining a motion score and the preliminary target having a corresponding first matching result, the first matching result corresponding to the preliminary target is used as the trajectory matching result.

[0065] Specifically, the initial target and target trajectory matched in the second matching result are obtained. The intersection-union ratio (IUR) between the detection box of the corresponding initial target and the historical box of the target trajectory is determined. The IUR is used to generate the motion score of the corresponding initial target. The motion score is negatively correlated with the corresponding IUR, that is, the smaller the IUR, the higher the motion score. The above process is expressed by the following formula: (13) in, Representing the trajectory In the Frame motion scoring Representing the trajectory In the The bounding box of the target frame, also known as the detection box. Representing the trajectory In the The most recent historical target bounding box of the frame, also known as the history box.

[0066] Understandably, the motion score of the initial target is compared with the scoring threshold. If the motion score of the initial target is greater than the scoring threshold, the first matching result corresponding to the initial target is taken as the trajectory matching result. Thus, when the target's motion level is high, the result based on visual feature matching is selected. However, if the target only includes the second matching result and the motion score is greater than the scoring threshold, the target will not have a matching trajectory because it does not have a corresponding first matching result. This allows the second matching result with low accuracy to be eliminated.

[0067] Furthermore, if the motion score of the initially selected target is less than or equal to the scoring threshold, the second matching result corresponding to the initially selected target is used as the trajectory matching result. This allows for the selection of matching results based on spatiotemporal location and size when the target's motion level is low, reducing the impact of inaccurate visual feature matching caused by angle or lighting conditions. Therefore, the optimal feature is adaptively selected based on the motion level to leverage the advantages of different features in different scenarios, improving the accuracy and robustness of trajectory matching.

[0068] It should be noted that if the initially selected target does not receive a motion score but there is a first matching result for the initially selected target, this scenario is very likely to occur when the target is missed or the target moves too fast. In this case, the first matching result for the initially selected target will be used as the trajectory matching result to ensure the comprehensiveness of the trajectory matching.

[0069] S204: Obtain other unmatched targets in the current frame, and determine the trajectory matching results of other targets based on the intersection-union ratio between the detection boxes and historical boxes of other targets.

[0070] Specifically, other unmatched targets in the current frame are obtained. Based on the intersection-union ratio between the detection boxes of other targets and all historical boxes, the remaining unmatched targets are matched with the target trajectories to obtain the trajectory matching results of other targets.

[0071] In one embodiment, obtaining other unmatched targets in the current frame and determining the trajectory matching results of other targets based on the intersection-union ratio (IU) between the detection boxes and historical boxes of other targets includes: obtaining the IU between the historical boxes of unmatched target trajectories and the detection boxes of low-confidence targets; matching low-confidence targets with IU greater than a first IU threshold with the unmatched target trajectories to obtain trajectory matching results for at least some low-confidence targets; obtaining the IU between the historical boxes of unmatched target trajectories and the detection boxes of all unmatched targets; matching unmatched targets with IU greater than a second IU threshold with the unmatched target trajectories to obtain trajectory matching results for at least some unmatched targets; wherein the second IU threshold is greater than the first IU threshold.

[0072] Specifically, the intersection-union ratio (IUR) between the historical bounding boxes of unmatched target trajectories and the detection boxes of low-confidence targets is obtained. Low-confidence targets with IUR greater than a first IUR threshold are filtered out. Based on the corresponding IUR values, a cost matrix is ​​constructed between the filtered low-confidence targets and the unmatched target trajectories. Hungarian matching is then performed based on the cost matrix to obtain trajectory matching results for at least some of the low-confidence targets. Therefore, a first IUR threshold is set for low-confidence targets, and corresponding targets are selected for matching to ensure the matching accuracy of low-confidence targets.

[0073] Furthermore, the intersection-union ratio (IUR) between the historical bounding boxes of the remaining unmatched target trajectories and the detection boxes of all unmatched targets is obtained. Targets with IUR greater than a second IUR threshold that are still unmatched are filtered out. Based on the corresponding IUR, a cost matrix is ​​constructed between the filtered targets and the unmatched target trajectories. Hungarian matching is then performed based on the cost matrix to obtain trajectory matching results for at least some of the unmatched targets. The second IUR threshold is greater than the first IUR threshold, thus setting a relatively low threshold for low-confidence targets, performing a single match to determine matching target trajectories for low-confidence targets as much as possible. After matching both high-confidence and low-confidence targets, a higher threshold is set by the second IUR threshold to filter out targets and target trajectories that can still attempt to match, ensuring the accuracy of trajectory matching.

[0074] In some implementation scenarios, the second crossover ratio (CUP) threshold is 0.8 and the first CUP threshold is 0.5. These values ​​can be set according to specific requirements based on the specific scenario, and this application does not impose any specific restrictions on them.

[0075] S205: Based on the mutually matched targets and target trajectories, update the obtained target trajectory, and perform weighted fusion of the corresponding visual features of the target and the historical features of the target trajectory to update the historical features of the target trajectory; wherein, the weight of the historical features is greater than the weight of the visual features.

[0076] Specifically, based on the mutually matched targets and target trajectories, the obtained target trajectories are updated, and the corresponding visual features of the targets are obtained and weighted and fused with the historical features of the target trajectories, thereby updating the historical features of the target trajectories.

[0077] It should be noted that historical features have a greater weight than visual features. Therefore, historical features are used as the core, supplemented by the latest visual features, to ensure the accuracy of features related to the target's appearance. The process of updating historical features is expressed by the following formula: (14) in, These are historical features from before the update. It is the visual feature corresponding to the target in the current frame. It is a weight hyperparameter.

[0078] S206: Generate new target trajectories using unmatched initial targets; wherein, if any target trajectory fails to match a target for more than a preset number of frames, the corresponding target trajectory is discarded.

[0079] Specifically, the initial targets that do not match the target trajectory are obtained, and the corresponding initial targets are added as new targets. New target trajectories are generated for the new targets. Thus, when a target with a high confidence level fails to match the target trajectory, a target trajectory is generated for the corresponding target, and the first detection box of the target trajectory is obtained.

[0080] Furthermore, the number of frames in which the obtained target trajectory has not been matched with the target up to the current time is counted. If any target trajectory fails to match the target for more than a preset number of frames, the corresponding target trajectory is removed.

[0081] In some implementation scenarios, if any target trajectory fails to match the target for 15 consecutive frames, the corresponding target trajectory is discarded. The discard condition can be set to a preset number of consecutive occurrences or a preset total number of occurrences; the specific setting can be customized according to requirements, and this application does not impose any specific restrictions.

[0082] Please see Figure 3 , Figure 3This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 301 and a processor 302 coupled to each other. The memory 301 stores program data (not shown in the figure). The processor 302 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0083] Please see Figure 4 , Figure 4 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0084] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.

Claims

1. A target trajectory matching method characterized by, include: Obtain all initially selected targets and their corresponding detection boxes in the current frame, and determine all obtained target trajectories and their corresponding historical features and historical boxes; Based on the visual features of each of the preliminary targets and the historical features of each of the target trajectories, a first matching result between the preliminary targets and the target trajectories is determined. Based on the position and size of the detection boxes of each of the preliminary targets and the historical boxes of each of the target trajectories, as well as their overlapping areas, a second matching result between the preliminary targets and the target trajectories is determined. Based on the positional relationship between the matched detection boxes and historical boxes in the second matching result, the trajectory matching results of the initially selected target are filtered from the first matching result and the second matching result; Obtain other unmatched targets in the current frame, and determine the trajectory matching result of the other targets based on the intersection-union ratio between the detection boxes of the other targets and the historical boxes.

2. The target trajectory matching method according to claim 1, characterized in that, The step of determining the first matching result between the preliminary target and the target trajectory based on the visual features of each of the preliminary targets and the historical features of each target trajectory includes: A visual cost matrix is ​​constructed based on the visual feature similarity between the visual features of each of the initially selected targets and the historical features of each target trajectory; wherein, the cost in the visual cost matrix is ​​negatively correlated with the visual similarity. Hungarian matching is performed based on the visual cost matrix to obtain the first matching result between the initially selected target and the target trajectory.

3. The target trajectory matching method according to claim 1, characterized in that, The determination of a second matching result between the initial target and the target trajectory based on the position and size of the detection box of each of the initial targets and the historical box of each target trajectory, as well as their overlapping areas, includes: For each detection box of the initial target and the historical box of each target trajectory, obtain the overlap height and overlap width corresponding to the overlapping area between the detection box and the historical box. Based on the overlap height and overlap width, as well as the center distance between the detection box and the historical box and the rectangle width and height of the bounding rectangle, determine the bounding box similarity index between the detection box and the historical box. A motion cost matrix is ​​constructed using the bounding box similarity index between each detection box and a historical box. Hungarian matching is then performed based on the motion cost matrix to obtain a second matching result between the initially selected target and the target trajectory. The cost in the motion cost matrix is ​​negatively correlated with the bounding box similarity index.

4. The target trajectory matching method according to claim 1, characterized by, The step of filtering the trajectory matching results of the initially selected target from the first matching results and the second matching results based on the positional relationship between the matched detection boxes and historical boxes in the second matching results includes: Obtain the initial target and target trajectory matched in the second matching result, and generate the corresponding motion score of the initial target by using the intersection-union ratio (IUU) between the detection box of the corresponding initial target and the historical box of the target trajectory; wherein, the motion score is negatively correlated with the corresponding IUU. In response to the motion score of the initial target being greater than the score threshold, the first matching result corresponding to the initial target is taken as the trajectory matching result; In response to the motion score of the initial target being less than or equal to the score threshold, the second matching result corresponding to the initial target is taken as the trajectory matching result; In response to the fact that the initial target does not receive a motion score and the initial target has a corresponding first matching result, the first matching result corresponding to the initial target is taken as the trajectory matching result.

5. The target trajectory matching method according to claim 1, characterized by, The step of obtaining all initially selected targets and their corresponding detection boxes in the current frame includes: Obtain the bounding boxes and their corresponding confidence scores for all targets in the current frame; Based on the confidence level, at least some targets are divided into high-confidence targets and low-confidence targets; wherein, the high-confidence targets are selected as the initial targets.

6. The target trajectory matching method according to claim 5, characterized in that, The step of obtaining the detection boxes corresponding to all targets in the current frame and their corresponding confidence scores includes: The video stream is input into the object detection model to obtain the detection boxes corresponding to all targets in the current frame output by the object detection model, as well as the confidence level set by the object detection model for each target in the detection box; wherein, the object detection model includes a lightweight network, and the object detection model is trained using the video stream including the training target.

7. The target trajectory matching method according to claim 5, characterized in that, The step of obtaining other unmatched targets in the current frame and determining the trajectory matching result of the other targets based on the intersection-union ratio between the detection boxes of the other targets and the historical boxes includes: Obtain the intersection-union ratio (IUGR) between the historical bounding boxes of unmatched target trajectories and the detection boxes of the low-confidence targets; match the low-confidence targets with IUGR values ​​greater than a first IUGR threshold with the unmatched target trajectories to obtain trajectory matching results for at least some of the low-confidence targets. Obtain the intersection-union ratio (IUGR) between the historical bounding boxes of unmatched target trajectories and the detection boxes of all unmatched targets. Match the unmatched targets with the unmatched target trajectories whose IUGR is greater than a second IUGR threshold to obtain trajectory matching results for at least some of the unmatched targets; wherein, the second IUGR threshold is greater than the first IUGR threshold.

8. The target trajectory matching method according to any one of claims 1-7, characterized in that, After obtaining other unmatched targets in the current frame and determining the trajectory matching result of the other targets based on the intersection-union ratio between the detection boxes of the other targets and the historical boxes, the method further includes: Based on the matching targets and target trajectories, the obtained target trajectories are updated by weighted fusion of the corresponding visual features of the targets and the historical features of the target trajectories, and the historical features of the target trajectories are updated; where the weight of the historical features is greater than the weight of the visual features. New target trajectories are generated using the unmatched initial targets; wherein, if any target trajectory fails to match a target for more than a preset number of frames, the corresponding target trajectory is discarded.

9. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, it implements the method as described in any one of claims 1-8.

Images