Target tracking method and device, electronic device, computer readable medium

Abstract

The present disclosure provides a target tracking method and apparatus, an electronic device, and a computer-readable medium, belonging to the field of computer technology. The method includes: obtaining a video stream of a preset acquisition area; performing target detection on the t-th video frame of the video stream to determine a first candidate box of at least one candidate target in the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream; classifying the candidate targets according to the confidence levels of the candidate targets and the first intersection over union between the first candidate boxes of the respective candidate targets to determine at least one candidate target at a matching level; respectively matching the at least one candidate target at a matching level according to the tracker set in the (t - 1)-th state of the video stream and a preset matching strategy to determine the target tracking result of the t-th video frame; and determining the tracking trajectory of the target in the video stream according to the target tracking results of the T video frames.

Description

Technical Field

[0001] This disclosure belongs to the field of computer technology, and particularly relates to a target tracking method and device, an electronic device, and a non-transitory computer-readable medium. Background Art

[0002] Object detection and tracking is a very important task in computer vision. It can estimate the information of objects (such as vehicles, people, or objects) within the field of view through input data from sensors, and detect and track single or multiple objects within the field of view. In application scenarios such as smart cities and traffic analysis, the effect of object detection and tracking is particularly important. The object tracking methods in related technologies are easily affected by perspectives, light, and crowded objects, and have a high risk of object loss, resulting in poor object tracking effects. Summary of the Invention

[0003] This disclosure aims to at least solve one of the technical problems existing in the related technologies, and provides a target tracking method and device, an electronic device, and a non-transitory computer-readable medium.

[0004] In a first aspect, an embodiment of this disclosure provides a target tracking method, which includes:

[0005] Obtain a video stream of a preset acquisition area;

[0006] Perform object detection on the t-th video frame of the video stream to determine a first candidate box of at least one candidate object in the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream;

[0007] Classify the candidate objects according to the confidence levels of the candidate objects and the first intersection over union between the first candidate boxes of the candidate objects, and determine at least one candidate object at a matching level;

[0008] Match each of the at least one candidate object at a matching level according to the tracker set in the (t - 1)-th state of the video stream and a preset matching strategy, and determine the target tracking result of the t-th video frame; where the target tracking result includes the identifier and position of the target in the t-th video frame;

[0009] Determine the tracking trajectory of the target in the video stream according to the target tracking results of the T video frames of the video stream, where the tracking trajectory includes the video frames corresponding to the target and the positions in the corresponding video frames.

[0010] In a second aspect, an embodiment of this disclosure provides a target tracking device, which includes:

[0011] A video stream acquisition module, configured to obtain a video stream of a preset acquisition area;

[0012] A target detection module, configured to perform target detection on the t-th video frame of the video stream, and determine a first candidate bounding box of at least one candidate target in the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream;

[0013] A target grading module, configured to grade the candidate targets according to the confidence levels of the candidate targets and the first intersection over union between the first candidate bounding boxes of the respective candidate targets, and determine at least one candidate target at a matching level;

[0014] A target matching module, configured to respectively match the at least one candidate target at a matching level according to the tracker set in the (t - 1)-th state of the video stream and a preset matching strategy, and determine a target tracking result of the t-th video frame; wherein, the target tracking result includes the identifier and position of the target in the t-th video frame;

[0015] A trajectory determination module, configured to determine a tracking trajectory of the target in the video stream according to the target tracking results of the T video frames of the video stream, where the tracking trajectory includes the video frames corresponding to the target and the positions in the corresponding video frames.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0017] One or more processors; a memory, configured to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned target tracking method.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable medium, on which a computer program is stored, where the computer program implements the steps in the above-mentioned target tracking method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1a and Figure 1b are schematic diagrams of the same target under different circumstances. [[ID=?]]

[0021] Figure 2 is a flowchart of the target tracking method according to an embodiment of the present disclosure.

[0022] Figure 3 is a schematic diagram of target grading according to an embodiment of the present disclosure.

[0023] Figure 4 is a schematic diagram of an acquisition area according to an embodiment of the present disclosure.

[0024] Figure 5 This is a block diagram of a target tracking device according to an embodiment of the present disclosure.

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0028] In applications such as smart cities and traffic analytics, the effectiveness of multi-target multi-camera tracking (MTMC) is crucial. However, several challenges remain. The goal of cross-camera vehicle tracking is to identify the same vehicle passing through multiple intersections, but because the features of cameras across intersections are easily affected by perspective, lighting, and congested traffic, there is a high risk of vehicle loss.

[0029] Figure 1a and Figure 1b These are schematic diagrams illustrating the same objective under different conditions. For example... Figure 1a and Figure 1b As shown in the figure, the target within the white rectangle is the same vehicle. Due to the different angles of the different cameras and the different lighting conditions, the success rate of tracking this target in related technologies is relatively low.

[0030] The present disclosure proposes an effective and accurate system for multi-object multi-camera tracking. In the single-camera multi-tracking stage, the present disclosure proposes a single-camera multi-tracking method with multi-level matching, which makes full use of object detection information and splices tracking abnormal results, and can improve the accuracy and stability of tracking results. In the multi-camera matching stage, the present disclosure proposes a matching strategy based on spatio-temporal constraints, which greatly reduces the matching range and thus improves the matching accuracy.

[0031] Figure 2 The flowchart of the object tracking method according to the embodiment of the present disclosure. The object tracking method can be applied to an object tracking device, which can be implemented in a software and / or hardware manner and is generally integrated in an electronic device (such as a display device). The electronic device can be a terminal device or a server, etc. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.; the server can be a cloud server or an edge server, etc.

[0032] As Figure 2 shown, the object tracking method includes:

[0033] Step S11, obtaining a video stream of a preset acquisition area;

[0034] Step S12, performing object detection on the t-th video frame of the video stream to determine a first candidate box of at least one candidate object in the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream;

[0035] Step S13, grading the candidate objects according to the confidence of the candidate objects and the first intersection-over-union ratio between the first candidate boxes of the candidate objects, and determining at least one candidate object at a matching level;

[0036] Step S14, respectively matching the at least one candidate object at a matching level according to the tracker set in the (t - 1)-th state of the video stream and a preset matching strategy, and determining the object tracking result of the t-th video frame; wherein, the object tracking result includes the identifier and position of the object in the t-th video frame;

[0037] Step S15, determining the tracking trajectory of the object in the video stream according to the object tracking results of the T video frames of the video stream, where the tracking trajectory includes the video frames corresponding to the object and the positions in the corresponding video frames.

[0038] For example, the preset acquisition area can be the geographical area corresponding to the image captured by the camera, such as an intersection or square in a city. This disclosure does not impose any limitations on this.

[0039] In some embodiments, the video stream of the acquisition area can be obtained in step S11. The video stream can be a video stream acquired in real time by the camera, or a video stream of a certain duration acquired in advance. This disclosure does not limit this.

[0040] In some embodiments, the video stream may include multiple video frames, each video frame having a timestamp to indicate the time when the video frame was captured; there may be a certain time interval between each video frame, for example, when the frame rate is 20 frames / second, the time interval between adjacent video frames is 0.05 seconds.

[0041] In some embodiments, for the first frame of a video stream, target detection can be performed directly on the first frame to determine the detection box of the target in the first frame. A preset target detection network can be used to detect the first frame. The target detection network can be a convolutional neural network, such as the latest YOLOv5x6. This disclosure does not limit the specific type of target detection network.

[0042] In some embodiments, a set of trackers for the video stream in state 1 can be created. The set of trackers includes trackers for targets in the first video frame. Each tracker includes information such as the target's identifier, location, feature information, single-target prediction method, and prediction parameters.

[0043] In some embodiments, the target's identifier ID is used to identify the target so as to distinguish different targets; the target's position may be represented, for example, as the image coordinates of the diagonal vertex of the target's detection box, or the coordinates of the top-left vertex of the detection box and the height and width of the detection box.

[0044] In some embodiments, a pre-defined re-identification (reID) network can be used to extract features from the region image corresponding to the target's detection box to obtain the target's feature information. This re-identification network can be a residual network, such as ResNet101; this disclosure does not limit the specific network type of the re-identification network.

[0045] In some embodiments, the single-target prediction method can be set to at least one, such as the median optical flow prediction method, the ECO (Efficient Convolution Operators) single-target tracking method, etc. This disclosure does not limit the number or specific type of single-target prediction methods.

[0046] In some embodiments, the prediction parameters may include the prediction parameters of the Kalman prediction model (including the predicted speed of the target, etc.) adopted in a multi-object tracking method (e.g., SORT multi-object tracking algorithm, DeepSORT multi-object tracking algorithm); they may also include the prediction parameters of each single-object prediction method. The present disclosure does not limit the specific type and the number of the prediction parameters.

[0047] After creating the set of trackers for the first state of the video stream, target detection and tracking can be performed on subsequent video frames.

[0048] In some embodiments, for any video frame after the first video frame (referred to as the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream), in step S12, a target detection network can be used to perform target detection on the t-th video frame to obtain the first candidate box of the candidate target in the t-th video frame, that is, the position of the candidate target in the t-th video frame; the confidence of the candidate target can also be determined, which is used to represent the probability that the candidate target is a target of a preset category (e.g., vehicle, pedestrian).

[0049] In some embodiments, in step S13, the candidate targets can be classified according to the confidence of the candidate targets and the first intersection over union (IoU) between the first candidate boxes of each candidate target to determine at least one candidate target at a matching level. The first IoU is the ratio of the intersection area to the union area between the first candidate boxes, which is used to represent the overlapping degree between the first candidate boxes. The lower the IoU, the smaller the overlapping degree, that is, the less occlusion between each other; conversely, the higher the IoU, the greater the overlapping degree, that is, the more occlusion between each other.

[0050] In some embodiments, a confidence threshold and an IoU threshold can be set respectively to classify the candidate targets. For example, a set of relatively high confidence thresholds and relatively low IoU thresholds are set, and the candidate targets meeting this set of thresholds are considered as high-quality and unoccluded targets; a set of relatively low confidence thresholds and relatively low IoU thresholds are set, and the candidate targets meeting this set of thresholds are considered as low-quality and unoccluded targets; a set of relatively low confidence thresholds and relatively high IoU thresholds are set, and the candidate targets meeting this set of thresholds are considered as low-quality and occluded targets. In this way, different matching strategies can be set for different types of targets to efficiently and accurately match the targets.

[0051] In some embodiments, in step S14, the candidate targets at at least one matching level can be matched according to the tracker set of the video stream in state t-1 and a preset matching strategy to determine the target tracking result of the video frame t. For example, multiple precise matches can be performed on high-quality and unobstructed targets to improve the matching accuracy and success rate of these targets. Similarly, different matching strategies can be set for low-quality and unobstructed targets and low-quality and occluded targets.

[0052] In some embodiments, the target tracking result includes the identifier and location of the target in the t-th video frame. The target in the t-th video frame may include the target that was successfully matched among the candidate targets, or the target that failed to match but was determined to be newly added (newly appeared in the video frame).

[0053] In some embodiments, after matching is completed, strategies for creating, updating, and remediating trackers in the tracker set can be set for targets of different matching levels to improve the tracking performance of targets at various levels.

[0054] In some embodiments, by processing each video frame of the video stream sequentially through steps S12-S14, the target tracking result of the T-frame video frame of the video stream can be obtained. Then, in step S15, the tracking trajectory of the target in the video stream can be determined based on the target tracking result of the T-frame video frame. The tracking trajectory includes the video frame corresponding to the target and its position within that video frame. Thus, the multi-level matching single-camera multi-tracking processing is completed.

[0055] According to embodiments of this disclosure, target detection can be performed on video frames of a video stream to determine candidate bounding boxes of candidate targets in the video frames; the candidate targets are classified according to their confidence level and the intersection-over-union ratio (IoU) between candidate bounding boxes to determine at least one candidate target at a matching level; multiple candidate targets at matching levels are matched according to the tracker set of the previous state and a preset matching strategy to determine the target tracking result of the video frame; and the tracking trajectory of the target in the video stream is determined based on the target tracking result of the T-frame video frames of the video stream. This fully utilizes target detection information, thereby improving the accuracy and stability of the tracking results.

[0056] The target tracking method according to embodiments of this disclosure will now be described in detail.

[0057] As mentioned above, the video stream of the acquisition area can be acquired in step S11, and the video frames of the video stream can be used for target detection in step S12 through the target detection network.

[0058] For the t-th video frame, in step S12, a target detection network is used to perform target detection on the t-th video frame to obtain the first candidate box of at least one candidate target in the t-th video frame, and the confidence level of the candidate target can also be determined.

[0059] In some embodiments, in step S13, the candidate targets are graded according to their confidence levels and the first intersection-union ratios between the first candidate boxes of each candidate target, to determine at least one candidate target with a matching level. For any candidate target, step S13 may include:

[0060] If the confidence level of the candidate target is greater than or equal to the first confidence threshold, and the maximum value of the first intersection-union ratio of the candidate target is less than or equal to the first intersection-union ratio threshold, the candidate target is determined as the first candidate target of the first matching level.

[0061] If the confidence level of the candidate target is less than the first confidence threshold and greater than or equal to the second confidence threshold, and the maximum value of the first intersection-union ratio of the candidate target is less than or equal to the first intersection-union ratio threshold, the candidate target is determined as the second candidate target of the second matching level.

[0062] If the confidence level of a candidate target is less than the first confidence threshold and greater than or equal to the second confidence threshold, and the maximum value of the first crossover ratio (CUP) of the candidate target is greater than the first CUP threshold and less than or equal to the second CUP threshold, then the candidate target is determined as the third candidate target of the third matching level.

[0063] Wherein, the first confidence threshold is greater than the second confidence threshold, and the first cross-union ratio threshold is less than the second cross-union ratio threshold.

[0064] Figure 3 This is a schematic diagram illustrating the target hierarchy of an embodiment of this disclosure. For example... Figure 3 As shown, the rectangles represent high-quality, unobstructed targets, low-quality, unobstructed targets, and low-quality, occluded targets, respectively. It is evident that high-quality, unobstructed targets provide the most accurate information, low-quality, unobstructed targets offer some information, and low-quality, occluded targets provide the least information and are prone to mismatches during tracking.

[0065] In some embodiments, a set of higher first confidence thresholds and lower first intersection-over-union (IoU) thresholds can be set, and candidate targets that meet these thresholds are considered high-quality and unobstructed targets. That is, if the confidence of a candidate target is greater than or equal to the first confidence threshold, and the maximum value of the first IoU of the candidate target is less than or equal to the first IoU threshold, the candidate target is determined as the first candidate target of the first matching level. For example, the first confidence threshold (conf) can be set to 0.3, and the first IoU threshold can be set to 0.3. This disclosure does not limit the specific values ​​of the first confidence threshold and the first IoU threshold.

[0066] The maximum value of the first intersection-union ratio (IUR) between the candidate target and other candidate targets represents the candidate target with the highest degree of overlap with it. If the IUR corresponding to the candidate target with the highest degree of overlap is still relatively small, it can be said that the candidate target is unobstructed.

[0067] In some embodiments, a set of lower second confidence thresholds and lower first intersection-over-union (IoU) thresholds can be set, and candidate targets that meet these thresholds are considered low-quality and unoccluded targets. That is, if the confidence of a candidate target is less than the first confidence threshold but greater than or equal to the second confidence threshold, and the maximum value of the first IoU of the candidate target is less than or equal to the first IoU threshold, the candidate target is determined as a second candidate target of the second matching level. For example, the second confidence threshold conf can be set to 0.1. This disclosure does not limit the specific value of the second confidence threshold.

[0068] The first confidence threshold (e.g., 0.3) is greater than the second confidence threshold (e.g., 0.1). Furthermore, candidate targets with a first confidence threshold less than the second confidence threshold are considered invalid targets and are either not further processed or directly deleted.

[0069] In some embodiments, a set of lower confidence thresholds and higher intersection-over-union (IoU) thresholds can be set, and candidate targets that meet these thresholds are considered low-quality and occluded targets. That is, if the confidence of a candidate target is less than a first confidence threshold but greater than or equal to a second confidence threshold, and the maximum value of the first IoU of the candidate target is greater than the first IoU threshold but less than or equal to the second IoU threshold, the candidate target is determined as a third candidate target of the third matching level. For example, the second IoU threshold can be set to 0.45. This disclosure does not limit the specific value of the second IoU threshold.

[0070] The first crossover ratio (CUP) threshold (e.g., 0.3) is less than the second CUP threshold (e.g., 0.45). Furthermore, candidate targets with a first CUP greater than the second CUP threshold are considered invalid targets and are either not further processed or directly deleted.

[0071] In this way, different matching strategies can be set for different types of targets to achieve efficient and accurate target matching. For example, when there are 10 candidate targets in the t-th video frame, the number of candidate targets for the first matching level, the second matching level, and the third matching level may be 3, 3, and 4, respectively.

[0072] In some embodiments, after the hierarchical classification in step S13, multi-level matching can be performed in step S14. Based on the tracker set of the video stream in state t-1 and the preset matching strategy, the candidate targets of the at least one matching level are matched respectively to determine the target tracking result of the video frame t.

[0073] In some embodiments, when processing of the (t-1)th video frame is completed, the tracker set of the (t-1)th state can be updated and obtained. The tracker set of the (t-1)th state includes at least one of the following: a first tracker of a first historical target that was successfully matched in the (t-1)th video frame, and a second tracker of a second historical target that was not matched in the (t-1)th video frame.

[0074] The first historical target that is successfully matched in the (t-1)th video frame may include targets that were not lost in the (t-1)th video frame, and may also include targets that were added in the (t-1)th video frame; the second historical target that is not matched in the (t-1)th video frame may include targets that were lost in the (t-1)th video frame, but have not yet been deleted (they may be deleted after several consecutive frames are lost).

[0075] In some embodiments, four matching operations can be performed sequentially at four matching levels to achieve target tracking. Step S14 may include:

[0076] Step S1411: Extract features from the first candidate target to obtain the first feature information of the first candidate target;

[0077] Step S1412: Based on the first tracker of the first historical target, the first prediction box of the first historical target in the t-th video frame is determined by adopting the corresponding single target prediction method.

[0078] Step S1413: Determine the first loss matrix between the first historical target and the first candidate target based on the first feature information of the first candidate target, the first candidate bounding box of the first candidate target, and the first predicted bounding box of the first historical target in the t-th video frame.

[0079] Step S1414: Based on the first loss matrix, determine the first matching result between the first historical target and the first candidate target. The first matching result includes at least one of the first candidate target that is successfully matched, the first candidate target that fails to match, and the first historical target that fails to match in the first matching level.

[0080] Step S1415: For the first candidate target that is successfully matched in the first matching result, the first candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the first historical target that is matched, and the position of the target is the position of the first candidate box of the target.

[0081] For example, a precise match can be made between a high-quality, unobstructed candidate target (the first candidate target at the first matching level) and the first historical target that was successfully matched in the (t-1)th frame of the video.

[0082] In some embodiments, in step S1411, a re-identification network can be used to extract features from the region image corresponding to the candidate bounding box of the first candidate target in the t-th video frame, thereby obtaining the target's feature information (referred to as the first feature information). In this way, only the features of high-quality and unobstructed candidate targets can be extracted, making the extracted features more accurate, thereby improving the matching accuracy, reducing the probability of false matching, and reducing the number of candidate targets for feature extraction, thus reducing the computational load.

[0083] In some embodiments, in step S1412, single-target prediction is performed on each first historical target according to the single-target prediction method and corresponding prediction parameters in the first tracker, to obtain the first predicted bounding box of the first historical target in the t-th video frame. For example, using the median optical flow prediction method and the ECO single-target tracking method, two first predicted bounding boxes of the first historical target are obtained according to these two methods.

[0084] In some embodiments, in step S1413, a first loss matrix between the first historical target and the first candidate target can be determined based on the first feature information of the first candidate target, the first candidate bounding box of the first candidate target, and the first predicted bounding box of the first historical target in the t-th video frame. Step S1413 may include:

[0085] Based on the first feature information of the first candidate target and the feature information of the first historical target, a first feature distance matrix is ​​determined;

[0086] Based on the first candidate bounding box of the first candidate target and the first predicted bounding box of the first historical target in the t-th video frame, determine the first intersection-union distance matrix;

[0087] The first loss matrix is ​​determined based on the first feature distance matrix and the first intersection-union distance matrix.

[0088] In other words, the feature distance, such as the feature cosine distance, between the first feature information of each first candidate target and the feature information of each first historical target can be calculated separately to obtain the first feature distance matrix. For example, when there are 3 first candidate targets and 4 first historical targets, a 3*4 first feature distance matrix can be obtained.

[0089] In some embodiments, the intersection-union ratio (IUR) between the first candidate bounding boxes of each first candidate target and the first predicted bounding boxes of each first historical target can be calculated to obtain a first IUR distance matrix. For example, when there are 3 first candidate targets and 4 first historical targets, a 3*4 first IUR distance matrix can be obtained. This disclosure does not limit the specific calculation method of the first IUR distance matrix.

[0090] In some embodiments, the prediction method for the first historical target includes at least one single-target prediction method, and the first intersection-over-union (IoU) distance matrix correspondingly includes at least one IoU distance matrix. When multiple single-target prediction methods are employed, multiple first IoU distance matrices can be obtained respectively.

[0091] In some embodiments, the first loss matrix M1 can be determined based on the first feature distance matrix and the first intersection-union ratio (IU / U) distance matrix. M1 is, for example, expressed as:

[0092] M1=α1*A1+β1*B1+γ1*C1 (1)

[0093] In formula (1), A1 represents the first feature distance matrix, B1 represents the first intersection-over-union (IoU) distance matrix obtained using the median optical flow prediction method, and C1 represents the first IoU distance matrix obtained using the ECO single-target tracking method. α1, β1, and γ1 represent the weights of A1, B1, and C1, respectively. The weights α1, β1, and γ1 can be normalized, i.e., α1 + β1 + γ1 = 1; alternatively, the weights can be left unnormalized, and this disclosure does not impose any restrictions on this. It should be understood that those skilled in the art can set specific values ​​for the weights α1, β1, and γ1 according to actual circumstances, and this disclosure does not impose any restrictions on this.

[0094] In some embodiments, in step S1414, a first matching result between the first historical target and the first candidate target may be determined according to the first loss matrix. The first matching result includes at least one of the first candidate target that is successfully matched, the first candidate target that fails to match, and the first historical target that fails to match in the first matching level.

[0095] A first distance threshold can be set. If any item in the first loss matrix is ​​less than or equal to the first distance threshold, the first historical target and the first candidate target are considered to be successfully matched. If any item in the first loss matrix is ​​greater than the first distance threshold, the first historical target and the first candidate target are considered to be unmatched. In this way, through matching processing, at least one of the following can be obtained: a successfully matched first historical target and a first candidate target; an unmatched first candidate target; and an unmatched first historical target, which constitutes the first matching result.

[0096] This disclosure does not impose any restrictions on the specific value of the first distance threshold.

[0097] In some embodiments, in step S1415, for the first candidate target that is successfully matched in the first matching result, the first candidate target can be determined as the target in the t-th video frame, wherein the identifier of the target is the identifier ID of the first historical target that is matched, which is used to represent the same target; the position of the target is the position of the first candidate box of the target.

[0098] In this way, it is possible to achieve accurate matching between high-quality and unobstructed candidate targets and historical targets that were successfully matched in the previous frame, thereby improving the accuracy of the matching.

[0099] In some embodiments, after the first match, if there are first candidate targets and first historical targets that failed to match in the first match result, the first candidate targets and first historical targets that failed to match in the first match can be matched again.

[0100] In some embodiments, step S14 may further include:

[0101] Step S1421: Based on the prediction parameters of the third historical target, a Kalman prediction model is used to predict and determine the second prediction box of the third historical target in the t-th video frame. The third historical target is the first historical target that failed to match in the first matching result.

[0102] Step S1422: Determine the second intersection-union distance matrix based on the first candidate box of the fourth candidate target and the second predicted box of the third historical target, wherein the fourth candidate target is the first candidate target that failed to match in the first matching result;

[0103] Step S1423: Determine the third intersection-union distance matrix based on the first candidate box of the fourth candidate target and the first predicted box of the third historical target in the t-th video frame;

[0104] Step S1424: Determine the second loss matrix based on the second cross-union-ratio (CUNR) distance matrix and the third CUNR distance matrix;

[0105] Step S1425: Based on the second loss matrix, determine the second matching result between the fourth candidate target and the third historical target. The second matching result includes at least one of the successfully matched fourth candidate target, the unmatched fourth candidate target, and the unmatched third historical target.

[0106] Step S1426: For the fourth candidate target that is successfully matched in the second matching result, the fourth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the matched third historical target, and the position of the target is the position of the first candidate box of the target.

[0107] For example, the first historical target that failed to match in the first matching result can be called the third historical target, and the first candidate target that failed to match in the first matching result can be called the fourth candidate target, so as to facilitate differentiation.

[0108] In some embodiments, in step S1421, the prediction parameters of the Kalman prediction model can be obtained based on the first tracker of the third historical target; by performing prediction using the Kalman prediction model, the second prediction box of the third historical target in the t-th video frame can be determined. This disclosure does not limit the specific prediction process.

[0109] In some embodiments, in step S1422, a second intersection-over-union (IoU) distance matrix is ​​determined based on the first candidate bounding box of the fourth candidate target and the second predicted bounding box of the third historical target. This disclosure does not limit the specific calculation method of the second IoU distance matrix.

[0110] In some embodiments, in step S1423, a third intersection-with-union (IoU) distance matrix is ​​determined based on the first candidate bounding box of the fourth candidate target and the first predicted bounding box of the third historical target in the t-th video frame (obtained in the aforementioned step S1412). Similarly, multiple third IoU distance matrices can be obtained when employing various single-target prediction methods. This disclosure does not limit the specific calculation method of the third IoU distance matrix.

[0111] In some embodiments, in step S1424, a second loss matrix M2 is determined based on the second cross-union ratio (CUP) distance matrix and the third CUP distance matrix. M2 is expressed, for example, as:

[0112] M2=α2*P1+β2*B2+γ2*C2 (2)

[0113] In formula (2), P1 represents the second cross-union distance matrix, B2 represents the third cross-union distance matrix obtained using the median optical flow prediction method, and C2 represents the third cross-union distance matrix obtained using the ECO single-target tracking method. α2, β2, and γ2 represent the weights of P, B2, and C2, respectively. The weights α2, β2, and γ2 can be normalized, i.e., α2 + β2 + γ2 = 1; alternatively, the weights can be left unnormalized, and this disclosure does not impose any restrictions on this. It should be understood that those skilled in the art can set specific values ​​for the weights α2, β2, and γ2 according to actual circumstances, and this disclosure does not impose any restrictions on this.

[0114] In some embodiments, in step S1425, a second matching result between the fourth candidate target and the third historical target can be determined based on the second loss matrix. The second matching result includes at least one of the fourth candidate target that is successfully matched, the fourth candidate target that fails to match, and the third historical target that fails to match.

[0115] A second distance threshold can be set. If any item in the second loss matrix is ​​less than or equal to the second distance threshold, the corresponding fourth candidate target and the third historical target are considered to have matched successfully; if any item in the second loss matrix is ​​greater than the second distance threshold, the corresponding fourth candidate target and the third historical target are considered to have failed to match. Thus, through matching processing, target pairs consisting of successfully matched fourth candidate targets and third historical targets, and at least one of unmatched fourth candidate targets and unmatched third historical targets can be obtained, which is the second matching result.

[0116] The second distance threshold may differ from the first distance threshold. This disclosure does not impose any restrictions on the specific value of the second distance threshold.

[0117] In some embodiments, in step S1426, for the fourth candidate target that is successfully matched in the second matching result, the fourth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier ID of the matching third historical target, used to represent the same target; the position of the target is the position of the first candidate box of the target.

[0118] In this way, based on the first match, a second match can be made between high-quality and unobstructed candidate targets that failed the first match and historical targets that were successfully matched in the previous frame, thereby further improving the accuracy and success rate of the match.

[0119] In some embodiments, after the second matching, if a fourth candidate target (a high-quality, unobstructed candidate target that failed to match) exists in the second matching result, then this fourth candidate target can be matched with the second historical target (a tracker that has been lost) that failed to match in the (t-1)th frame of the video frame, i.e., a third matching, in order to remedy the historical targets that failed to match effectively in the previous frame or several previous frames. It is possible that the historical target was occluded and lost, and may reappear in subsequent frames.

[0120] In some embodiments, step S14 may further include:

[0121] Step S1431: Determine the second feature distance matrix based on the first feature information of the fifth candidate target and the feature information of the second historical target. The fifth candidate target is the fourth candidate target that failed to match in the second matching result.

[0122] Step S1432: Based on the second feature distance matrix, determine the third matching result between the fifth candidate target and the second historical target. The third matching result includes at least one of the successfully matched fifth candidate target, the unmatched fifth candidate target, and the unmatched second historical target.

[0123] Step S1433: For the fifth candidate target that is successfully matched in the third matching result, the fifth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the matched second historical target, and the position of the target is the position of the first candidate box of the target.

[0124] For example, the fourth candidate target that failed to match in the second matching result (that is, the first candidate target that failed to match in both matches) can be called the fifth candidate target to facilitate differentiation.

[0125] In some embodiments, in step S1431, a second feature distance matrix is ​​determined based on the first feature information of the fifth candidate target and the feature information of the second historical target. The feature distance, such as the feature cosine distance, between the first feature information of each fifth candidate target and the feature information of each second historical target can be calculated to obtain the second feature distance matrix A2.

[0126] In some embodiments, in the case of match remedy, the judgment can be made using only the feature distance. That is, the second feature distance matrix is ​​directly used as the loss matrix M = A2. In step S1432, based on the second feature distance matrix, a third matching result between the fifth candidate target and the second historical target is determined. The third matching result includes at least one of the successfully matched fifth candidate target, the unmatched fifth candidate target, and the unmatched second historical target.

[0127] In some embodiments, similarly, a third distance threshold can be set. If any item in the second feature distance matrix is ​​less than or equal to the third distance threshold, the corresponding fifth candidate target and the second historical target are considered to be successfully matched; if any item in the second feature distance matrix is ​​greater than the third distance threshold, the corresponding fifth candidate target and the second historical target are considered to be unmatched. Thus, through matching processing, at least one of the following can be obtained: a successfully matched fifth candidate target and a successfully matched second historical target; an unmatched fifth candidate target; and an unmatched second historical target, i.e., a third matching result.

[0128] The third distance threshold may differ from both the first and second distance thresholds. This disclosure does not impose any restrictions on the specific value of the third distance threshold.

[0129] In some embodiments, in step S1433, for the fifth candidate target that is successfully matched in the third matching result, the fifth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the matching second historical target, used to indicate the same target; the position of the target is the position of the first candidate box of the target.

[0130] In this way, based on the secondary matching, it is possible to salvage the matching between high-quality and unobstructed candidate targets that failed the secondary matching and the second historical target (the tracker that has been lost), thereby further improving the matching success rate.

[0131] In some embodiments, after three matches, if there are still unmatched historical targets (also referred to as unmatched trackers), a fourth match can be performed with low-quality, unoccluded candidate targets and low-quality, occluded candidate targets. The fourth match can be performed in a similar manner to the second match.

[0132] In some embodiments, step S14 may further include:

[0133] Step S1441: Based on the prediction parameters of the fourth historical target, a Kalman prediction model is used to predict and determine the third prediction box of the fourth historical target in the t-th video frame. The fourth historical target includes the third historical target that failed to match in the second matching result and the second historical target that failed to match in the third matching result.

[0134] Step S1442: Determine the fourth prediction box of the fourth historical target in the t-th video frame according to the single-target prediction method of the fourth historical target;

[0135] Step S1443: Determine the third loss matrix based on the first candidate box of the sixth candidate target and the third and fourth prediction boxes of the fourth historical target. The sixth candidate target includes the second candidate target and the third candidate target.

[0136] Step S1444: Based on the third loss matrix, determine the fourth matching result between the sixth candidate target and the fourth historical target. The fourth matching result includes at least one of the following: a successfully matched sixth candidate target, a failed matched sixth candidate target, and a failed matched fourth historical target.

[0137] Step S1445: For the sixth candidate target that is successfully matched in the fourth matching result, the sixth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the fourth historical target that is matched, and the position of the target is the position of the first candidate box of the target.

[0138] For example, a fourth historical target can be set, including a third historical target that failed to match in the second matching result and a second historical target that failed to match in the third matching result; a sixth candidate target can also be set, including a second candidate target (low quality and unoccluded) and a third candidate target (low quality and occluded) to facilitate differentiation.

[0139] In some embodiments, in step S1441, the prediction parameters of the Kalman prediction model can be obtained based on the tracker of the fourth historical target, and prediction is performed using the Kalman prediction model to determine the third prediction box of the fourth historical target in the t-th video frame. This disclosure does not limit the specific prediction process.

[0140] In some embodiments, in step S1442, the fourth prediction box of the fourth historical target in the t-th video frame is determined according to the single-target prediction method of the fourth historical target. This disclosure does not limit the specific prediction process.

[0141] In some embodiments, in step S1443, a third loss matrix is ​​determined based on the first candidate box of the sixth candidate target and the third and fourth predicted boxes of the fourth historical target. That is, an intersection-union-range (IUGR) distance matrix is ​​determined based on the first candidate box and the third predicted box; the number of IUGR matrices corresponds to the number of single-target prediction methods; and the third loss matrix is ​​obtained by weighted summing of the IUGR matrices. The representation of the third loss matrix is ​​similar to that of formula (2), and will not be repeated here.

[0142] In some embodiments, step S1444 involves determining a fourth matching result between the sixth candidate target and the fourth historical target based on the third loss matrix. The fourth matching result includes at least one of the sixth candidate target that is successfully matched, the sixth candidate target that fails to match, and the fourth historical target that fails to match.

[0143] In some embodiments, a fourth distance threshold can be set similarly. Through matching processing, a target pair consisting of a successfully matched sixth candidate target and a fourth historical target is obtained, along with at least one of a failed match sixth candidate target and a failed match fourth historical target, which is the fourth matching result.

[0144] In some embodiments, in step S1445, for the sixth candidate target that is successfully matched in the fourth matching result, the sixth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the fourth historical target that is matched, used to indicate the same target; the position of the target is the position of the first candidate box of the target.

[0145] In this way, based on the first three matches, a fourth match can be made between the tracker of low-quality, unobstructed / obstructed candidate targets and all remaining historical targets, further improving the success rate of the match.

[0146] In some embodiments, if there is still a first candidate target that has not been matched after the first three matching processes described above, then the target can be considered a newly emerging target.

[0147] In some embodiments, step S14 may further include:

[0148] Step S1451: For the fifth candidate target that failed to match in the third matching result, the fifth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is a newly created identifier, and the position of the target is the position of the first candidate box of the target.

[0149] In other words, the fifth candidate target that fails to match in the third matching result can be directly determined as the target in the t-th video frame, and a new identifier is created for this target to represent the new target and distinguish it from the existing targets; the position of this target is the position of the first candidate box of the target.

[0150] This completes the entire matching process in step S14, yielding the final target tracking result for the t-th video frame. This target tracking result includes the identifiers and positions of all targets in the t-th video frame as determined in the preceding steps.

[0151] In some embodiments, after obtaining the target tracking result of the t-th video frame, the trackers in the tracker set of the (t-1)-th state can be created, updated, and remedied as needed to obtain the tracker set of the t-th state, so that it can be used again when processing the next frame (t+1-th frame).

[0152] In some embodiments, after step S14, the method further includes:

[0153] A first tracker for the target is created in the tracker set. The first tracker for the target includes the target's identifier, location, feature information, single-target prediction method, and prediction parameters.

[0154] For example, if no matching candidate box is found after the first three matching steps, it can be assumed that a new, high-quality, unobstructed target has entered the field of view and appears in the t-th frame. In this case, a first tracker for the target can be created from the tracker set based on the target's candidate box, including the target's identifier, location, feature information, single-target prediction method, and prediction parameters. At this point, the Kalman prediction model can be initialized, the prediction parameters of each single-target prediction method can be initialized, optical flow point sampling can be performed, and the target's feature information can be extracted and saved through a re-identification network.

[0155] In this way, it is possible to create trackers for new targets for subsequent processing.

[0156] In some embodiments, for each target matched in the previous four matches, the tracker for each target can be updated.

[0157] In some embodiments, after step S14, the method further includes:

[0158] A seventh candidate target is determined, which includes: the first candidate target that was successfully matched in the first matching result, the fourth candidate target that was successfully matched in the second matching result, the fifth candidate target that was successfully matched in the third matching result, and the sixth candidate target that was successfully matched in the fourth matching result;

[0159] Update the tracker of the seventh candidate target based on its location.

[0160] In other words, a seventh candidate target can be set, including the first candidate target that was successfully matched in the first matching result, the fourth candidate target that was successfully matched in the second matching result, the fifth candidate target that was successfully matched in the third matching result, and the sixth candidate target that was successfully matched in the fourth matching result, that is, all targets matched in the previous four matching results. The tracker for each seventh candidate target is updated according to the position of the candidate box of each seventh candidate target.

[0161] In some embodiments, the step of updating the tracker of the seventh candidate target includes:

[0162] Based on the location of the seventh candidate target, update the prediction parameters of the Kalman prediction model in the tracker of the seventh candidate target;

[0163] For the single-target prediction method of the seventh candidate target, a second intersection-union ratio is determined between the candidate bounding box of the seventh candidate target and the prediction bounding box determined by the single-target prediction method;

[0164] If the second cross-union ratio is greater than the third cross-union ratio threshold, initialize the prediction parameters corresponding to the single-target prediction method;

[0165] If the second cross-union ratio is less than or equal to the third cross-union ratio threshold, the prediction parameters corresponding to the single-target prediction method are updated according to the position of the seventh candidate target.

[0166] For example, the parameters need to be updated for all seventh candidate targets.

[0167] In some embodiments, based on the position of the seventh candidate target in the previous frame or several previous frames and its position in the t-th frame, the prediction parameters of the Kalman prediction model in the corresponding tracker can be updated, such as updating the prediction speed in the Kalman prediction model, in order to improve the accuracy of subsequent predictions.

[0168] In some embodiments, for each seventh candidate target's single-target prediction method, it can be first determined whether the single-target prediction has shifted. Specifically, a second intersection-union ratio (IU) can be determined between the candidate bounding box of the seventh candidate target and the predicted bounding box determined by the corresponding single-target prediction method; if the second IU is greater than a preset third IU threshold, then the single-target prediction is considered to have shifted. In this case, the prediction parameters corresponding to the single-target prediction method can be initialized, for example, by using the region image of the t-th video frame corresponding to the candidate bounding box to completely update all information used for prediction, thereby achieving initialization.

[0169] In some embodiments, if the second cross-union ratio is less than or equal to a preset third cross-union ratio threshold, it can be considered that the single-target prediction has not shifted. In this case, the prediction parameters corresponding to the single-target prediction method can be updated in a conventional manner, such as using the region image of the t-th video frame corresponding to the candidate box to partially update various information used for prediction.

[0170] This approach improves the accuracy and effectiveness of parameter updates, thereby enhancing the precision of target tracking predictions for subsequent video frames.

[0171] In some embodiments, for high-quality and unoccluded targets and low-quality and unoccluded targets, the feature information (re-identification features) in the tracker can be updated; for low-quality and occluded targets, updating the feature information (re-identification features) in the tracker may lead to inaccurate subsequent processing or even mismatches, so there is no need to update the feature information.

[0172] In some embodiments, the step of updating the tracker of the seventh candidate target further includes:

[0173] For the first candidate target among the seventh candidate targets, the feature information in the tracker of the first candidate target is updated according to the first feature information of the first candidate target;

[0174] For the second candidate target among the seventh candidate targets, feature extraction is performed on the second candidate target to obtain the second feature information of the second candidate target;

[0175] Based on the second feature information, update the feature information in the tracker for the second candidate target.

[0176] In other words, for the first candidate target (a high-quality and unobstructed target) among the seventh candidate targets, the first feature information has already been obtained in the previous step S1411. In this case, the feature information in the tracker of the first candidate target can be directly updated based on the first feature information of the first candidate target.

[0177] In some embodiments, for the second candidate target (a low-quality, unobstructed target) among the seventh candidate targets, a re-identification network can be used in the current step to extract features from the region image corresponding to the candidate bounding box of the second candidate target in the t-th video frame, thereby obtaining the second feature information of the second candidate target. Then, based on the second feature information, the feature information in the tracker of the second candidate target is updated.

[0178] In this way, the re-identification features in the trackers for both high-quality and unobstructed targets and low-quality and unobstructed targets can be updated, thereby improving the accuracy of target tracking prediction for subsequent video frames.

[0179] In some embodiments, after step S14, the method further includes:

[0180] For the fourth historical target that failed to match in the fourth matching result, the prediction parameters of the Kalman prediction model in the tracker of the fourth historical target are updated according to the position of the fourth prediction box of the fourth historical target.

[0181] In other words, for a historical target that fails to match after four attempts (the fourth historical target that failed to match), the number of frames lost by the tracker of the fourth historical target can be determined (if the current t-th frame is lost, the number of lost frames is set to 1). If the number of lost frames reaches a preset frame threshold (e.g., 5 frames), it can be considered that the fourth historical target has left the acquisition area, and the tracker of the fourth historical target is deleted.

[0182] In some embodiments, if the number of lost frames does not reach a preset frame threshold, the tracker for the fourth historical target can be salvaged. Based on the position of the fourth predicted bounding box of the fourth historical target obtained using a single-target prediction method, the prediction parameters of the Kalman prediction model in the tracker of the fourth historical target are updated so that tracking and prediction can continue using the Kalman prediction model.

[0183] In this way, the stability of the tracker can be improved even when the detection results flicker (disappear for a few frames), thereby improving the accuracy of target tracking prediction.

[0184] Through the above processing, the entire process of creating, updating, remedying, and deleting trackers in the tracker set of state t-1 is completed, resulting in the tracker set of state t, which can be used again in the processing of the next frame (frame t+1). When processing the video frame t+1, the targets corresponding to each tracker in the tracker set of state t can be used as historical targets and matched with the candidate targets detected in the video frame t+1.

[0185] In some embodiments, by processing each video frame of the video stream sequentially through steps S12-S14, the target tracking result of the T-frame video frame of the video stream can be obtained. Then, in step S15, the tracking trajectory of the target in the video stream can be determined based on the target tracking result of the T-frame video frame. The tracking trajectory includes the video frame corresponding to the target and its position within that video frame. Thus, the multi-level matching single-camera multi-tracking processing is completed.

[0186] According to embodiments of this disclosure, it is also possible to match the tracking trajectories of video streams from multiple cameras. In some embodiments, there can be multiple preset acquisition areas, and video streams from each acquisition area can be acquired by multiple cameras to obtain video streams from multiple acquisition areas.

[0187] In some embodiments, after determining the tracking trajectory of the target in the video stream in step S15, the method further includes:

[0188] Step S16: Based on the geographical locations of the multiple acquisition areas and the tracking trajectories of the targets in the video streams of the multiple acquisition areas, the targets are grouped by the spatial correlation between the tracking trajectories to obtain multiple spatial groups. Each spatial group includes at least one target and the tracking trajectory of that target.

[0189] Step S17: For any spatial group, based on the start and end times of the tracking trajectory of the target in the spatial group and the passage time interval between the acquisition areas where the target is located, the targets in the spatial group are grouped to obtain multiple time subgroups.

[0190] Step S18: Cluster the target tracking trajectories in each time subgroup to obtain the trajectory matching results of the spatial group. The trajectory matching results include the tracking trajectories of the same target in video streams from different acquisition areas.

[0191] Step S19: Determine the trajectory matching results of the video streams in the multiple acquisition areas based on the trajectory matching results of the multiple spatial groups.

[0192] For example, multi-camera matching involves matching the trajectory of the target in the video stream of each individual camera's capture area after obtaining the target's tracking trajectory, based on the target's features and other information, to achieve cross-camera tracking.

[0193] In some embodiments, spatial and temporal correlations between tracking trajectories can be used to filter individual tracking trajectories in order to improve the accuracy of subsequent trajectory matching.

[0194] In some embodiments, in step S16, targets are grouped based on the spatial correlation between the tracking trajectories of the targets in the video streams of the plurality of acquisition areas, resulting in multiple spatial groups. For example, for two geographically adjacent acquisition areas, their spatially correlated target tracking trajectories can be divided into two groups: one group consists of targets moving from acquisition area 1 to acquisition area 2, and the other group consists of targets moving from acquisition area 2 to acquisition area 1. This avoids two targets with opposite trajectory directions from participating in subsequent clustering, thereby reducing the matching error rate.

[0195] In some embodiments, step S16 may include:

[0196] Based on the geographical locations of the first and second acquisition areas, and the tracking trajectories of targets in the video streams of the first and second acquisition areas, the first spatial group and the second spatial group are determined.

[0197] Wherein, the first acquisition area and the second acquisition area are any two adjacent acquisition areas among the plurality of acquisition areas.

[0198] The first spatial group includes targets that enter the second acquisition area from the first acquisition area, and the second spatial group includes targets that enter the first acquisition area from the second acquisition area.

[0199] For example, any two adjacent acquisition areas in multiple acquisition areas are referred to as the first acquisition area and the second acquisition area. The first spatial group and the second spatial group can be determined based on the geographical location of the first acquisition area and the second acquisition area, as well as the tracking trajectory of the target in the video stream of the first acquisition area and the second acquisition area.

[0200] Figure 4 This is a schematic diagram of the acquisition area according to an embodiment of this disclosure. Figure 4 As shown, in some embodiments, the acquisition area (camera scene) can be divided into the following 9 areas: the lower left area is 1, the areas clockwise are 2, 3, 4, 5, 6, 7, and 8, and the middle area is 9. When the tracking trajectory of a target ends in area 2, it is considered that the target is heading to the left side of the intersection. When the tracking trajectory of a target ends in area 6, it is considered that the target is heading to the right side of the intersection.

[0201] In some embodiments, the acquisition area can be divided into the following 5 regions: 1+2 = West, 3+4 = North, 5+6 = East, 7+8 = South, and 9 = Center. That is, the 9 regions can be simplified to 5 regions. When the tracking trajectory of a target ends in the west and enters the adjacent acquisition area in the east, then based on the relative time sequence, it can be assumed that the target is heading west; otherwise, it is heading east.

[0202] In some embodiments, if the second acquisition area is located at an intersection adjacent to the left of the first acquisition area, then a target leaving area 2 of the first acquisition area (the end area of ​​the tracking trajectory is area 2 of the first acquisition area) will enter from area 5 of the second acquisition area (the starting area of ​​the tracking trajectory is area 5 of the second acquisition area); furthermore, a target leaving area 6 of the second acquisition area (the end area of ​​the tracking trajectory is area 6 of the second acquisition area) will enter from area 1 of the first acquisition area (the starting area of ​​the tracking trajectory is area 2 of the first acquisition area). In this way, the first spatial grouping and the second spatial grouping can be determined.

[0203] In some embodiments, the first spatial group includes a target entering the second acquisition area from the first acquisition area, and the second spatial group includes a target entering the first acquisition area from the second acquisition area.

[0204] By processing each pair of adjacent acquisition areas within multiple acquisition areas separately, multiple spatial groups can be obtained. Each spatial group includes at least one target and its tracking trajectory. When an acquisition area has multiple adjacent acquisition areas in different directions, the target in that acquisition area may be assigned to multiple spatial groups; for example, the target in that acquisition area might correspond to... Figure 4 At a crossroads, there may be adjacent acquisition areas in four directions. Each direction has a first spatial group and a second spatial group, corresponding to a total of eight spatial groups. Targets within this acquisition area may also be repeatedly assigned to different spatial groups. This disclosure does not limit the specific number of spatial groups or the specific grouping method of the targets.

[0205] By grouping tracking trajectories according to their spatial correlation, tracking trajectories without spatial correlation can be filtered out, thereby improving the success rate of cross-camera tracking and matching.

[0206] In some embodiments, in step S17, for any spatial group, the targets in the spatial group are grouped according to the start time and end time of the tracking trajectory of the target in the spatial group, and the passage time interval between the acquisition areas where the target is located, to obtain multiple time subgroups.

[0207] In other words, the tracking trajectories can be further filtered based on their temporal correlation. Specifically, the start and end times of the tracking trajectory can be determined by the timestamps of the starting and ending video frames within the target's tracking trajectory.

[0208] In some embodiments, prior to step S17, the method may further include:

[0209] Based on the distance between the first collection area and the second collection area, the maximum speed limit, the minimum speed limit, and the average vehicle speed, the passage time interval between the first collection area and the second collection area is determined.

[0210] In other words, based on the distance dn between the adjacent first and second collection areas, as well as the maximum speed limit, minimum speed limit, and average vehicle speed vn between the first and second collection areas, the time required for the target to travel from the first collection area to the second collection area, or from the second collection area to the first collection area, can be calculated. This is called the transit time interval, which is the interval corresponding to the minimum time tn_min and the maximum time tn_max for traveling this distance.

[0211] For example, if the distance between the first and second data collection areas is 1.5km, the maximum speed limit is 90km / h, the minimum speed limit is 30km / h, and the average vehicle speed is 60km / h, then the passage times can be calculated to be 1 minute, 3 minutes, and 1.5 minutes, respectively. In this case, the passage time range can be set to [1 minute, 3 minutes].

[0212] In some embodiments, step S17 may include:

[0213] For the first spatial group corresponding to the first acquisition area and the second acquisition area, if the end time of the target's tracking trajectory in the first acquisition area is before the start time of the target's tracking trajectory in the second acquisition area, and the time difference between the end time and the start time is within the passage time interval, the corresponding targets are grouped into the same time subgroup.

[0214] In other words, for a first spatial group of targets entering the second acquisition area from the first acquisition area, if the end time of the tracking trajectory of the target in the first acquisition area is before the start time of the tracking trajectory of the target in the second acquisition area, and the time difference between the end time and the start time is within the passage time interval, then the corresponding targets may be the same target and can be classified into the same time subgroup for subsequent clustering.

[0215] Conversely, if the end time of the target's tracking trajectory in the first acquisition area is after the start time of the target's tracking trajectory in the second acquisition area, or if the time difference between the end time and the start time is not within the passage time interval, then the corresponding targets cannot be the same target and will not be assigned to the same time subgroup.

[0216] In some embodiments, a similar method is used to determine whether a first spatial group of a target entering the first acquisition area from the second acquisition area belongs to the same temporal subgroup.

[0217] In this way, the temporal correlation between tracking trajectories can be used to further filter each tracking trajectory, further narrowing the scope of subsequent clustering, thereby reducing the matching error rate.

[0218] In some embodiments, in step S18, the tracking trajectories of the target in each time subgroup can be clustered to obtain the trajectory matching results of the spatial group. The trajectory matching results include the tracking trajectories of the same target in video streams from different acquisition areas.

[0219] In some embodiments, the clustering method may be hierarchical clustering. Among them, step S18 includes: using the hierarchical clustering method to cluster the tracking trajectories of the targets in each time sub-group respectively, and obtaining the trajectory matching result of this spatial group.

[0220] It should be understood that those skilled in the art can set the clustering method according to the actual situation. For example, the K-means clustering method can also be used, etc. The present disclosure does not limit the specific clustering method.

[0221] In some embodiments, in step S19, according to the trajectory matching results of all the multiple spatial groups, the trajectory matching result of the video streams of the multiple acquisition areas can be determined. In this way, the entire processing process of target matching for multiple cameras is realized.

[0222] In this way, by using spatio-temporal conditions (spatial correlation and temporal correlation) to filter the tracking trajectories of the targets in multi-camera target matching, most of the unmatched tracking trajectories can be filtered out, thereby greatly improving the accuracy of hierarchical clustering and the accuracy of multi-target tracking across cameras, and solving the technical problem of low accuracy of multi-target tracking across cameras.

[0223] According to an embodiment of the present disclosure, a target tracking device is also provided. Figure 5 It is a block diagram of the target tracking device according to an embodiment of the present disclosure. As Figure 5 shown, the device includes:

[0224] A video stream acquisition module 51, configured to acquire the video stream of a preset acquisition area;

[0225] A target detection module 52, configured to perform target detection on the t-th video frame of the video stream, and determine the first candidate box of at least one candidate target in the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream;

[0226] A target grading module 53, configured to grade the candidate targets according to the confidence of the candidate targets and the first intersection over union between the first candidate boxes of the candidate targets, and determine at least one candidate target at a matching level;

[0227] A target matching module 54, configured to match the at least one candidate target at a matching level respectively according to the tracker set in the (t - 1)-th state of the video stream and a preset matching strategy, and determine the target tracking result of the t-th video frame; wherein, the target tracking result includes the identifier and position of the target in the t-th video frame;

[0228] The trajectory determination module 55 is used to determine the tracking trajectory of the target in the video stream based on the target tracking results of the T-frame video frames of the video stream. The tracking trajectory includes the video frame corresponding to the target and its position in the corresponding video frame.

[0229] In some embodiments, for any candidate target, the target classification module is configured to:

[0230] If the confidence level of a candidate target is greater than or equal to a first confidence threshold, and the maximum value of the first intersection-over-union (IoU) ratio of the candidate target is less than or equal to the first IoU threshold, the candidate target is determined as a first candidate target at a first matching level. If the confidence level of a candidate target is less than the first confidence threshold but greater than or equal to a second confidence threshold, and the maximum value of the first IoU ratio of the candidate target is less than or equal to the first IoU threshold, the candidate target is determined as a second candidate target at a second matching level. If the confidence level of a candidate target is less than the first confidence threshold but greater than or equal to the second confidence threshold, and the maximum value of the first IoU ratio of the candidate target is greater than the first IoU threshold but less than or equal to the second IoU threshold, the candidate target is determined as a third candidate target at a third matching level. Wherein, the first confidence threshold is greater than the second confidence threshold, and the first IoU threshold is less than the second IoU threshold.

[0231] In some embodiments, the tracker set in the video stream at state t-1 includes at least one of: a first tracker of a first historical target that was successfully matched in the (t-1)th video frame, and a second tracker of a second historical target that was not matched in the (t-1)th video frame, wherein the tracker includes the identifier, location, feature information, single-target prediction method, and prediction parameters of the tracked target.

[0232] The target matching module is used for:

[0233] Feature extraction is performed on the first candidate target to obtain the first feature information of the first candidate target; based on the first tracker of the first historical target, a corresponding single-target prediction method is adopted to determine the first prediction box of the first historical target in the t-th video frame; based on the first feature information of the first candidate target, the first candidate box of the first candidate target, and the first prediction box of the first historical target in the t-th video frame, a first loss matrix between the first historical target and the first candidate target is determined; based on the first loss matrix, a first matching result between the first historical target and the first candidate target is determined, the first matching result including at least one of the first candidate target that is successfully matched in the first matching level, the first candidate target that is not matched, and the first historical target that is not matched; for the first candidate target that is successfully matched in the first matching result, the first candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the matched first historical target, and the position of the target is the position of the first candidate box of the target.

[0234] In some embodiments, determining the first loss matrix for matching the first historical target and the first candidate target based on the first feature information of the first candidate target, the first candidate bounding box of the first candidate target, and the first predicted bounding box of the first historical target in the t-th video frame includes: determining a first feature distance matrix based on the first feature information of the first candidate target and the feature information of the first historical target; determining a first intersection-union-ratio (IUU) distance matrix based on the first candidate bounding box of the first candidate target and the first predicted bounding box of the first historical target in the t-th video frame; and determining the first loss matrix based on the first feature distance matrix and the first IUU distance matrix.

[0235] In some embodiments, the prediction method of the first historical target includes at least one single-target prediction method, and the first cross-union distance matrix correspondingly includes at least one cross-union distance matrix.

[0236] In some embodiments, the target matching module is further configured to:

[0237] Based on the prediction parameters of the third historical target, a Kalman prediction model is used to predict and determine the second prediction box of the third historical target in the t-th video frame. The third historical target is the first historical target that failed to match in the first matching result. Based on the first candidate box of the fourth candidate target and the second prediction box of the third historical target, a second intersection-union distance matrix is ​​determined. The fourth candidate target is the first candidate target that failed to match in the first matching result. Based on the first candidate box of the fourth candidate target and the first prediction box of the third historical target in the t-th video frame, a third intersection-union distance matrix is ​​determined. Based on the second intersection-union distance matrix and the third intersection-union distance matrix, a second loss matrix is ​​determined. Based on the second loss matrix, a second matching result between the fourth candidate target and the third historical target is determined. The second matching result includes at least one of a successfully matched fourth candidate target, a failed matched fourth candidate target, and a failed matched third historical target. For the successfully matched fourth candidate target in the second matching result, the fourth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the matched third historical target, and the position of the target is the position of the first candidate box of the target.

[0238] In some embodiments, the target matching module is further configured to:

[0239] Based on the first feature information of the fifth candidate target and the feature information of the second historical target, a second feature distance matrix is ​​determined, wherein the fifth candidate target is the fourth candidate target that failed to match in the second matching result; based on the second feature distance matrix, a third matching result is determined between the fifth candidate target and the second historical target, wherein the third matching result includes at least one of the fifth candidate target that matched successfully, the fifth candidate target that failed to match, and the second historical target that failed to match; for the fifth candidate target that matched successfully in the third matching result, the fifth candidate target is determined as the target in the t-th video frame; wherein the identifier of the target is the identifier of the matched second historical target, and the position of the target is the position of the first candidate box of the target.

[0240] In some embodiments, the target matching module is further configured to:

[0241] Based on the prediction parameters of the fourth historical target, a Kalman prediction model is used to predict and determine the third prediction box of the fourth historical target in the t-th video frame. The fourth historical target includes the third historical target that failed to match in the second matching result and the second historical target that failed to match in the third matching result. Based on the single-target prediction method of the fourth historical target, the fourth prediction box of the fourth historical target in the t-th video frame is determined. Based on the first candidate box of the sixth candidate target and the third and fourth prediction boxes of the fourth historical target, a third loss matrix is ​​determined. The sixth candidate target includes the second candidate target and the third candidate target. Based on the third loss matrix, a fourth matching result between the sixth candidate target and the fourth historical target is determined. The fourth matching result includes at least one of the sixth candidate target that matched successfully, the sixth candidate target that failed to match, and the fourth historical target that failed to match. For the sixth candidate target that matched successfully in the fourth matching result, the sixth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the matched fourth historical target, and the position of the target is the position of the first candidate box of the target.

[0242] In some embodiments, the target matching module is further configured to:

[0243] For the fifth candidate target that fails to match in the third matching result, the fifth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is a newly created identifier, and the position of the target is the position of the first candidate box of the target;

[0244] After determining the target tracking result of the t-th video frame, the device further includes: a creation module for creating a first tracker for the target in the tracker set, wherein the first tracker for the target includes the target's identifier, location, feature information, single-target prediction method, and prediction parameters.

[0245] In some embodiments, after determining the target tracking result of the t-th video frame, the apparatus further includes: a determining module, configured to determine a seventh candidate target, the seventh candidate target including: a first candidate target successfully matched in the first matching result, a fourth candidate target successfully matched in the second matching result, a fifth candidate target successfully matched in the third matching result, and a sixth candidate target successfully matched in the fourth matching result; and an updating module, configured to update the tracker of the seventh candidate target according to the position of the seventh candidate target.

[0246] In some embodiments, the update module is configured to:

[0247] Based on the location of the seventh candidate target, update the prediction parameters of the Kalman prediction model in the tracker of the seventh candidate target; for the single-target prediction method of the seventh candidate target, determine the second cross-union ratio (CUP) between the candidate bounding box of the seventh candidate target and the prediction bounding box determined by the single-target prediction method; if the second CUP is greater than the third CUP threshold, initialize the prediction parameters corresponding to the single-target prediction method; if the second CUP is less than or equal to the third CUP threshold, update the prediction parameters corresponding to the single-target prediction method based on the location of the seventh candidate target.

[0248] In some embodiments, the updating module is further configured to:

[0249] For the first candidate target among the seventh candidate targets, the feature information in the tracker of the first candidate target is updated according to the first feature information of the first candidate target; for the second candidate target among the seventh candidate targets, the second candidate target is subjected to feature extraction to obtain the second feature information of the second candidate target; and the feature information in the tracker of the second candidate target is updated according to the second feature information.

[0250] In some embodiments, after determining the target tracking result of the t-th video frame, the apparatus further includes: a parameter update module, configured to update the prediction parameters of the Kalman prediction model in the tracker of the fourth historical target based on the position of the fourth prediction box of the fourth historical target for the fourth historical target that failed to match in the fourth matching result.

[0251] In some embodiments, the video stream includes video streams from multiple acquisition areas, and after determining the tracking trajectory of the target in the video stream, the apparatus further includes:

[0252] The spatial grouping module is used to group targets based on the geographical locations of the multiple acquisition areas and the tracking trajectories of targets in the video streams of the multiple acquisition areas, by means of the spatial correlation between the tracking trajectories, to obtain multiple spatial groups. Each spatial group includes at least one target and its tracking trajectory. The temporal grouping module is used to group targets in any spatial group based on the start and end times of the tracking trajectories of targets in the spatial group and the passage time interval between the acquisition areas where the target is located, to obtain multiple temporal subgroups. The clustering module is used to cluster the tracking trajectories of targets in each temporal subgroup to obtain the trajectory matching results of the spatial groups. The trajectory matching results include tracking trajectories belonging to the same target in the video streams of different acquisition areas. The trajectory matching module is used to determine the trajectory matching results of the video streams of the multiple acquisition areas based on the trajectory matching results of the multiple spatial groups.

[0253] In some embodiments, the spatial grouping module is configured to:

[0254] Based on the geographical locations of the first and second acquisition areas, and the tracking trajectories of targets in the video streams of the first and second acquisition areas, a first spatial group and a second spatial group are determined; wherein the first and second acquisition areas are any two adjacent acquisition areas among the plurality of acquisition areas, the first spatial group includes targets entering the second acquisition area from the first acquisition area, and the second spatial group includes targets entering the first acquisition area from the second acquisition area.

[0255] In some embodiments, before grouping the targets in the spatial grouping to obtain multiple time subgroups, the device further includes: a duration determination module, configured to determine the passage time interval between the first collection area and the second collection area based on the distance between the first collection area and the second collection area, the maximum speed limit, the minimum speed limit and the average vehicle speed;

[0256] The time grouping module is configured to: for a first spatial group corresponding to the first acquisition area and the second acquisition area, if the end time of the target's tracking trajectory in the first acquisition area is before the start time of the target's tracking trajectory in the second acquisition area, and the time difference between the end time and the start time is within the passage time interval, group the corresponding targets into the same time subgroup.

[0257] In some embodiments, the clustering module is used to: cluster the tracking trajectories of the target in each time subgroup using hierarchical clustering to obtain the trajectory matching results of the spatial group.

[0258] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Figure 6 As shown, this disclosure provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the target tracking methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0259] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, enabling information exchange between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0260] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0261] According to embodiments of this disclosure, a non-transient computer-readable medium is also provided. This non-transient computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the target tracking methods described in the above embodiments.

[0262] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined above in the system of this disclosure.

[0263] It should be noted that the non-transient computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any non-transient computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the non-transient computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0264] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0265] The circuits or sub-circuits described in the embodiments of this disclosure can be implemented in software or hardware. The described circuits or sub-circuits can also be housed in a processor; for example, it can be described as: a processor including: a receiving circuit and a processing circuit, the processing module including a writing sub-circuit and a reading sub-circuit. The names of these circuits or sub-circuits do not necessarily constitute a limitation on the circuit or sub-circuit itself; for example, a receiving circuit can also be described as "receiving video signals".

[0266] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. A target tracking method, comprising: Obtaining a video stream of a preset acquisition area; Performing target detection on the t-th video frame of the video stream to determine a first candidate box of at least one candidate target in the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream; Classifying the candidate targets according to the confidence of the candidate targets and the first intersection over union between the first candidate boxes of the candidate targets to determine at least one candidate target of a matching level; Performing matching on the at least one candidate target of a matching level respectively according to the tracker set in the (t - 1)-th state of the video stream and a preset matching strategy to determine the target tracking result of the t-th video frame; wherein, the target tracking result includes the identifier and position of the target in the t-th video frame; Determining the tracking trajectory of the target in the video stream according to the target tracking results of the T video frames of the video stream, where the tracking trajectory includes the video frames corresponding to the target and the position in the corresponding video frames; The video stream includes video streams of multiple acquisition areas. After determining the tracking trajectory of the target in the video stream, the method further includes: Grouping the targets according to the geographical locations of the multiple acquisition areas and the tracking trajectories of the targets in the video streams of the multiple acquisition areas through the spatial correlation between the tracking trajectories to obtain multiple spatial groups, and each spatial group includes at least one target and the tracking trajectory of the target; For any one of the spatial groups, grouping the targets in the spatial group according to the start time and end time of the tracking trajectories of the targets in the spatial group and the passing time interval between the acquisition areas where the targets are located to obtain multiple time sub-groups; Clustering the tracking trajectories of the targets in each time sub-group respectively to obtain the trajectory matching result of the spatial group, where the trajectory matching result includes the tracking trajectories belonging to the same target in the video streams of different acquisition areas; Determining the trajectory matching result of the video streams of the multiple acquisition areas according to the trajectory matching results of the multiple spatial groups.

2. The method according to claim 1, wherein, For any one of the candidate targets, the classifying the candidate targets according to the confidence of the candidate targets and the first intersection over union between the first candidate boxes of the candidate targets to determine at least one candidate target of a matching level includes: When the confidence of the candidate target is greater than or equal to a first confidence threshold and the maximum value of the first intersection over union of the candidate target is less than or equal to a first intersection over union threshold, determining the candidate target as a first candidate target of a first matching level; When the confidence of the candidate target is less than the first confidence threshold and greater than or equal to a second confidence threshold, and the maximum value of the first intersection over union of the candidate target is less than or equal to a first intersection over union threshold, determining the candidate target as a second candidate target of a second matching level; If the confidence level of a candidate target is less than the first confidence threshold and greater than or equal to the second confidence threshold, and the maximum value of the first crossover ratio (CUP) of the candidate target is greater than the first CUP threshold and less than or equal to the second CUP threshold, then the candidate target is determined as the third candidate target of the third matching level. Wherein, the first confidence threshold is greater than the second confidence threshold, and the first cross-union ratio threshold is less than the second cross-union ratio threshold.

3. The method according to claim 2, wherein, The tracker set in the video stream at state t-1 includes at least one of the following: a first tracker of a first historical target that was successfully matched in the (t-1)th video frame, and a second tracker of a second historical target that was not matched in the (t-1)th video frame. Each tracker includes the target's identifier, location, feature information, single-target prediction method, and prediction parameters. The step of matching candidate targets at at least one matching level according to the tracker set of the video stream in state t-1 and a preset matching strategy to determine the target tracking result of the video frame t includes: Feature extraction is performed on the first candidate target to obtain the first feature information of the first candidate target; Based on the first tracker of the first historical target, the first prediction box of the first historical target in the t-th video frame is determined by adopting the corresponding single target prediction method. Based on the first feature information of the first candidate target, the first candidate bounding box of the first candidate target, and the first predicted bounding box of the first historical target in the t-th video frame, a first loss matrix between the first historical target and the first candidate target is determined. Based on the first loss matrix, a first matching result is determined between the first historical target and the first candidate target. The first matching result includes at least one of the first candidate target that is successfully matched, the first candidate target that fails to match, and the first historical target that fails to match in the first matching level. For the first candidate target that is successfully matched in the first matching result, the first candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the first historical target that is matched, and the position of the target is the position of the first candidate box of the target.

4. The method according to claim 3, wherein, The step of determining the first loss matrix for matching the first historical target and the first candidate target based on the first feature information of the first candidate target, the first candidate bounding box of the first candidate target, and the first predicted bounding box of the first historical target in the t-th video frame includes: Based on the first feature information of the first candidate target and the feature information of the first historical target, a first feature distance matrix is ​​determined; Based on the first candidate bounding box of the first candidate target and the first predicted bounding box of the first historical target in the t-th video frame, determine the first intersection-union distance matrix; The first loss matrix is ​​determined based on the first feature distance matrix and the first intersection-union distance matrix.

5. The method according to claim 4, wherein, The prediction method for the first historical target includes at least one single-target prediction method, and the first cross-union distance matrix correspondingly includes at least one cross-union distance matrix.

6. The method according to claim 3, wherein, The step of matching candidate targets at at least one matching level according to the tracker set of the video stream in state t-1 and a preset matching strategy to determine the target tracking result of the video frame t further includes: Based on the prediction parameters of the third historical target, a Kalman prediction model is used to predict and determine the second prediction box of the third historical target in the t-th video frame. The third historical target is the first historical target that failed to match in the first matching result. Based on the first candidate bounding box of the fourth candidate target and the second predicted bounding box of the third historical target, a second intersection-union distance matrix is ​​determined, wherein the fourth candidate target is the first candidate target that failed to match in the first matching result; The third intersection-union distance matrix is ​​determined based on the first candidate bounding box of the fourth candidate target and the first predicted bounding box of the third historical target in the t-th video frame. The second loss matrix is ​​determined based on the second cross-union distance matrix and the third cross-union distance matrix; Based on the second loss matrix, a second matching result is determined between the fourth candidate target and the third historical target. The second matching result includes at least one of the following: a successfully matched fourth candidate target, a failed matched fourth candidate target, and a failed matched third historical target. For the fourth candidate target that is successfully matched in the second matching result, the fourth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the matched third historical target, and the position of the target is the position of the first candidate box of the target.

7. The method according to claim 6, wherein, The step of matching candidate targets at at least one matching level according to the tracker set of the video stream in state t-1 and a preset matching strategy to determine the target tracking result of the video frame t further includes: Based on the first feature information of the fifth candidate target and the feature information of the second historical target, a second feature distance matrix is ​​determined, wherein the fifth candidate target is the fourth candidate target that failed to match in the second matching result; Based on the second feature distance matrix, a third matching result is determined between the fifth candidate target and the second historical target. The third matching result includes at least one of the successfully matched fifth candidate target, the unmatched fifth candidate target, and the unmatched second historical target. For the fifth candidate target that is successfully matched in the third matching result, the fifth candidate target is determined as the target in the t-th video frame; The identifier of the target is the identifier of the matching second historical target, and the position of the target is the position of the first candidate box of the target.

8. The method according to claim 7, wherein, The step of matching candidate targets at at least one matching level according to the tracker set of the video stream in state t-1 and a preset matching strategy to determine the target tracking result of the video frame t further includes: Based on the prediction parameters of the fourth historical target, a Kalman prediction model is used to predict and determine the third prediction box of the fourth historical target in the t-th video frame. The fourth historical target includes the third historical target that failed to match in the second matching result and the second historical target that failed to match in the third matching result. Based on the single-target prediction method of the fourth historical target, the fourth prediction box of the fourth historical target in the t-th video frame is determined; A third loss matrix is ​​determined based on the first candidate box of the sixth candidate target and the third and fourth predicted boxes of the fourth historical target. The sixth candidate target includes the second candidate target and the third candidate target. Based on the third loss matrix, a fourth matching result is determined between the sixth candidate target and the fourth historical target. The fourth matching result includes at least one of the following: a successfully matched sixth candidate target, a failed matched sixth candidate target, and a failed matched fourth historical target. For the sixth candidate target that is successfully matched in the fourth matching result, the sixth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is the identifier of the fourth historical target that is matched, and the position of the target is the position of the first candidate box of the target.

9. The method according to claim 7, wherein, The step of matching candidate targets at at least one matching level according to the tracker set of the video stream in state t-1 and a preset matching strategy to determine the target tracking result of the video frame t further includes: For the fifth candidate target that fails to match in the third matching result, the fifth candidate target is determined as the target in the t-th video frame, wherein the identifier of the target is a newly created identifier, and the position of the target is the position of the first candidate box of the target; After determining the target tracking result of the t-th video frame, the method further includes: A first tracker for the target is created in the tracker set. The first tracker for the target includes the target's identifier, location, feature information, single-target prediction method, and prediction parameters.

10. The method according to claim 8, wherein, After determining the target tracking result of the t-th video frame, the method further includes: A seventh candidate target is determined, which includes: the first candidate target that was successfully matched in the first matching result, the fourth candidate target that was successfully matched in the second matching result, the fifth candidate target that was successfully matched in the third matching result, and the sixth candidate target that was successfully matched in the fourth matching result; Update the tracker of the seventh candidate target based on its location.

11. The method according to claim 10, wherein, The update of the tracker for the seventh candidate target includes: Based on the location of the seventh candidate target, update the prediction parameters of the Kalman prediction model in the tracker of the seventh candidate target; For the single-target prediction method of the seventh candidate target, a second intersection-union ratio is determined between the candidate bounding box of the seventh candidate target and the prediction bounding box determined by the single-target prediction method; If the second cross-union ratio is greater than the third cross-union ratio threshold, initialize the prediction parameters corresponding to the single-target prediction method; If the second cross-union ratio is less than or equal to the third cross-union ratio threshold, the prediction parameters corresponding to the single-target prediction method are updated according to the position of the seventh candidate target.

12. The method according to claim 11, wherein, The tracker for updating the seventh candidate target further includes: For the first candidate target among the seventh candidate targets, the feature information in the tracker of the first candidate target is updated according to the first feature information of the first candidate target; For the second candidate target among the seventh candidate targets, feature extraction is performed on the second candidate target to obtain the second feature information of the second candidate target; Based on the second feature information, update the feature information in the tracker for the second candidate target.

13. The method according to claim 8, wherein, After determining the target tracking result of the t-th video frame, the method further includes: For the fourth historical target that failed to match in the fourth matching result, the prediction parameters of the Kalman prediction model in the tracker of the fourth historical target are updated according to the position of the fourth prediction box of the fourth historical target.

14. The method according to claim 1, wherein, The method involves grouping targets based on the geographical locations of the multiple acquisition areas and the target tracking trajectories in the video streams of the multiple acquisition areas, through the spatial correlation between the tracking trajectories, resulting in multiple spatial groups, including: Based on the geographical locations of the first and second acquisition areas, and the tracking trajectories of targets in the video streams of the first and second acquisition areas, the first spatial group and the second spatial group are determined. Wherein, the first acquisition area and the second acquisition area are any two adjacent acquisition areas among the plurality of acquisition areas. The first spatial group includes targets that enter the second acquisition area from the first acquisition area, and the second spatial group includes targets that enter the first acquisition area from the second acquisition area.

15. The method according to claim 14, wherein, Before grouping the targets in the spatial grouping to obtain multiple temporal subgroups, the method further includes: Based on the distance between the first collection area and the second collection area, the maximum speed limit, the minimum speed limit, and the average vehicle speed, the passage time interval between the first collection area and the second collection area is determined; Specifically, the targets in the spatial group are grouped according to the start and end times of the tracking trajectory of the target in the spatial group, and the transit time interval between the acquisition areas where the target is located, to obtain multiple time sub-groups, including: For the first spatial group corresponding to the first acquisition area and the second acquisition area, if the end time of the target's tracking trajectory in the first acquisition area is before the start time of the target's tracking trajectory in the second acquisition area, and the time difference between the end time and the start time is within the passage time interval, the corresponding targets are grouped into the same time subgroup.

16. The method according to claim 1, wherein, The process of clustering the tracking trajectories of the targets in each temporal subgroup to obtain the trajectory matching results of the spatial group includes: Hierarchical clustering is used to cluster the tracking trajectories of the target in each time subgroup to obtain the trajectory matching results of the spatial group.

17. A target tracking device, comprising: A video stream acquisition module, configured to acquire a video stream of a preset acquisition area; A target detection module, configured to perform target detection on the t-th video frame of the video stream to determine a first candidate box of at least one candidate target in the t-th video frame, where t is an integer and 1 < t ≤ T, and T is the number of video frames of the video stream; A target grading module, configured to grade the candidate targets according to the confidence levels of the candidate targets and the first intersection over union between the first candidate boxes of the candidate targets, and determine at least one candidate target at a matching level; A target matching module, configured to respectively match the at least one candidate target at a matching level according to the tracker set in the (t - 1)-th state of the video stream and a preset matching strategy, and determine the target tracking result of the t-th video frame; wherein, the target tracking result includes the identifier and position of the target in the t-th video frame; A trajectory determination module, configured to determine the tracking trajectory of the target in the video stream according to the target tracking results of the T video frames of the video stream, and the tracking trajectory includes the video frames corresponding to the target and the positions in the corresponding video frames; The video stream includes video streams of multiple acquisition areas. After determining the tracking trajectory of the target in the video stream, the apparatus further includes: A spatial grouping module, configured to group the targets according to the geographical locations of the multiple acquisition areas and the tracking trajectories of the targets in the video streams of the multiple acquisition areas, and obtain multiple spatial groupings through the spatial correlation between the tracking trajectories. Each spatial grouping includes at least one target and the tracking trajectory of the target; For any one of the spatial groupings, group the targets in the spatial grouping according to the start time and end time of the tracking trajectories of the targets in the spatial grouping and the passing time interval between the acquisition areas where the targets are located, and obtain multiple time sub-groupings; Cluster the tracking trajectories of the targets in each time sub-grouping respectively to obtain the trajectory matching result of the spatial grouping, and the trajectory matching result includes the tracking trajectories belonging to the same target in the video streams of different acquisition areas; Determine the trajectory matching result of the video streams of the multiple acquisition areas according to the trajectory matching results of the multiple spatial groupings.

18. An electronic device, comprising: One or more processors; A memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the target tracking method according to any one of claims 1 to 16.

19. A non-transient computer-readable medium having a computer program stored thereon, wherein, The computer program implements the steps in the target tracking method according to any one of claims 1 to 16 when executed by a processor.

Images