Multi-target tracking method, device, computer equipment and storage medium

Abstract

The present application relates to the field of target tracking technology, and in particular to a multi-target tracking method, device, computer equipment and storage medium. The multi-target tracking method includes: predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; using the predicted position of each target object and the forward appearance feature as the forward feature; wherein the forward appearance feature is the appearance feature of the target object in the forward video frame; obtaining the current position and current appearance feature of several detection objects in the current video frame as the current feature; inputting the forward feature and the current feature into a multi-level matching model, matching the detection object with the target object, so as to track each target object. The use of this method can introduce the trajectory of temporal memory and combine it with the appearance feature, which can reduce the risk of abnormalities and errors in the multi-target tracking process, thereby helping to improve the accuracy and reliability of the multi-target tracking method.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to a multi-target tracking method, a multi-target tracking device, a computer device, and a computer-readable storage medium. Background Art

[0002] In scenes such as conferences, sports games, and road monitoring, there are often multi-target tracking tasks. However, multi-target tracking technology often directly discards some of the detected objects in the video frame, which can easily lead to a large number of missed tracking and tracking anomalies. Alternatively, all detected objects in the video frame are tracked and matched in an unreasonable way, which can easily consume a lot of computing power and introduce a large number of mistracking situations. Summary of the invention

[0003] Based on this, it is necessary to provide a multi-target tracking method, a multi-target tracking device, a computer device and a computer-readable storage medium that can introduce a trajectory of temporal memory and combine appearance features to reduce the risk of anomalies and errors in the multi-target tracking process, thereby helping to improve the accuracy and reliability of the multi-target tracking method.

[0004] On the one hand, a multi-target tracking method is provided, and the multi-target tracking method includes: predicting the predicted positions of multiple target objects in a current video frame based on the trajectory of the target object; wherein the trajectory is a historical real trajectory obtained by historically tracking the target object; using the predicted position of each target object and a forward appearance feature as a forward feature; wherein the forward appearance feature is an appearance feature of the target object in a forward video frame; obtaining the current position and current appearance features of several detected objects in the current video frame as current features; inputting the forward features and the current features into a multi-level matching model, matching the detected objects with the target objects, so as to track each target object.

[0005] In one embodiment of the present application, a multi-level matching model includes a primary matching unit and a secondary matching unit; forward features and current features are input into the multi-level matching model, and matching of a detection object with a target object includes: taking a detection object having a confidence score higher than a confidence threshold as a first object; inputting the first object and the forward features into the primary matching unit for first feature matching; wherein the feature matching dimensions include motion features and appearance features; in response to the trajectory of the target object not matching the detection object of the current video frame, taking it as a secondary trajectory; taking a detection object having a confidence score lower than the confidence threshold as a second object; and inputting the second object, the secondary trajectory, and the corresponding forward appearance features of the target object into the secondary matching unit for second feature matching.

[0006] In one embodiment of the present application, the first object and the forward feature are input into a first-level matching unit, and after the first feature matching is performed, it also includes: in response to the first object not matching the target object, taking it as a new target object; creating a new trajectory of the new target object and assigning a new trajectory identifier, recording the current position of the new target object, and storing the new trajectory identifier, the newly created trajectory and the current position of the new target object in a trajectory storage module; storing the current appearance features of the new target object in an appearance matching module.

[0007] In one embodiment of the present application, the forward appearance features of the second object, the secondary trajectory and the corresponding target object are input into the secondary matching unit, and after the second feature matching is performed, it also includes: in response to the second object not matching the target object, discarding it and not creating a new trajectory; in response to the target object not matching the detection object through the multi-level matching model, increasing its trajectory age by one; until the trajectory age of the target object reaches the age threshold, the trajectory of the target object is deleted.

[0008] In one embodiment of the present application, feature matching includes first feature matching and / or second feature matching; feature matching includes: obtaining a target feature vector of each target object; wherein the target feature vector includes the predicted position and forward appearance features of the spliced ​​target object; obtaining a detection feature vector of each detection object; wherein the detection feature vector includes the current position and current appearance features of the spliced ​​detection object; respectively calculating the similarity between each target feature vector and each detection feature vector; matching the target object and the detection object based on the similarity using the Hungarian algorithm or the greedy algorithm.

[0009] In one embodiment of the present application, after matching the detection object with the target object, it also includes: in response to the existence of a target object that matches the detection object, inputting the current position into a trajectory storage module; the trajectory storage module retains the original trajectory identifier of the matched target object and refreshes its trajectory age to one; inputting the current appearance feature into the appearance matching module, and the appearance matching module fuses the current appearance feature with the appearance feature of the matched target object.

[0010] In one embodiment of the present application, inputting forward features and current features into a multi-level matching model includes: calculating a confidence score for each detection object; screening the confidence score and the confidence threshold to divide the detection object into a first object and a second object; constructing a first detection list input to a first-level matching unit based on a detection feature vector of the first object; constructing a second detection list input to a second-level matching unit based on a detection feature vector of the second object; constructing a target list of the target object based on a target feature vector of the target object; and inputting the first detection list, the second detection list, and the target list into the multi-level matching model.

[0011] In one embodiment of the present application, calculating the confidence score of each detected object includes: inputting the current video frame into the detector, the detector detecting whether there is a target object, obtaining the existence probability of each detected object and the category probability belonging to each target object; calculating the product of the existence probability of the detected object and the highest category probability as the confidence score.

[0012] In one embodiment of the present application, the detector detects whether a target object exists, including: dividing the current video frame into multiple grids; predicting at least one set of bounding boxes and category probabilities corresponding to the detection object in each grid; predicting the probability of the existence of the detection object in each bounding box based on the center point of each bounding box as a first prediction probability; predicting the probability of the existence of the detection object in each bounding box based on the size of each bounding box as a second prediction probability; and fusing the first prediction probability and the second prediction probability to obtain the existence probability.

[0013] In one embodiment of the present application, predicting the predicted positions of multiple target objects in the current video frame based on the trajectories of the target objects includes: in response to the target object matching the detection object in the forward video frame, obtaining the forward trajectory of the forward video frame; and predicting the center point position of the target object in the current video frame based on the forward trajectory to obtain the predicted position.

[0014] In one embodiment of the present application, the center point position of the target object in the current video frame is predicted based on the forward trajectory, and obtaining the predicted position includes: obtaining the displacement of the target object in the first video frame and the second video frame; wherein the first video frame is an adjacent forward video frame of the current video frame, and the second video frame is an adjacent forward video frame of the first video frame; calculating the moving speed of the target object based on the displacement; obtaining the center point position of the target object in the first video frame; and calculating the center point position of the target object in the current video frame using the center point position and the moving speed of the first video frame.

[0015] In one embodiment of the present application, obtaining current appearance features of several detection objects in the current video frame includes: inputting the current video frame into a feature extractor, and the feature extractor detects the detection objects existing in the current video frame; performing feature extraction on the detected several detection objects to obtain current appearance features belonging to each detection object.

[0016] In one embodiment of the present application, after matching the detection object with the target object, it also includes: in response to a detection object matching with a target object, storing a current appearance feature of a detection object to an appearance matching module; in the appearance matching module, fusing the current appearance feature of a detection object with a forward appearance feature of a target object; or, using the current appearance feature of a detection object to cover the forward appearance feature of a target object.

[0017] On the other hand, a multi-target tracking device is provided, which includes: a prediction module, a storage module, an extraction module and a multi-level matching model; the prediction module is used to predict the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; wherein the trajectory is the historical real trajectory obtained by historically tracking the target object; the storage module is used to store the trajectory of the target object and the forward appearance feature as the forward feature; wherein the forward appearance feature is the appearance feature of the target object in the forward video frame; the extraction module is used to obtain the current position and current appearance feature of several detection objects in the current video frame as the current feature; the multi-level matching model is used to input the forward feature and the current feature, match the detection object with the target object, so as to track each target object.

[0018] In one embodiment of the present application, the multi-level matching model includes a primary matching unit and a secondary matching unit; the primary matching unit is used to input a first object and a forward feature to perform a first feature matching; the first object is a detection object whose confidence score is higher than a confidence threshold; the secondary matching unit is used to input a second object, a secondary trajectory and a corresponding forward appearance feature of a target object to perform a second feature matching; the second object is a detection object whose confidence score is lower than the confidence threshold; the secondary trajectory is the trajectory of the target object of the detection object that is not matched to the current video frame.

[0019] In one embodiment of the present application, the storage module includes a trajectory storage module and an appearance matching module; the extraction module includes a detector and a feature extractor; the trajectory storage module is used to store the trajectory, trajectory identification and trajectory age of the target object; the appearance matching module is used to store the appearance features of the target object; the detector is used to detect whether the target object exists, and obtain the existence probability of each detection object and the category probability of belonging to each target object; the feature extractor is used to extract features of the detection objects existing in the current video frame to obtain the current appearance features belonging to each detection object.

[0020] On the other hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: predicting the predicted positions of multiple target objects in a current video frame based on the trajectory of the target object; using the predicted position of each target object and the forward appearance feature as the forward feature; wherein the forward appearance feature is the appearance feature of the target object in the forward video frame; obtaining the current positions and current appearance features of several detected objects in the current video frame as the current features; and inputting the forward features and the current features into a multi-level matching model to match the detected objects with the target objects so as to track each target object.

[0021] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; using the predicted position of each target object and the forward appearance feature as the forward feature; wherein the forward appearance feature is the appearance feature of the target object in the forward video frame; obtaining the current position and current appearance feature of several detected objects in the current video frame as the current feature; inputting the forward feature and the current feature into a multi-level matching model to match the detected object with the target object to track each target object.

[0022] The above-mentioned multi-target tracking method, multi-target tracking device, computer equipment and computer-readable storage medium introduce the existing trajectory of the target object for position prediction, that is, based on the existing trajectory of the target object, predict its possible position in the current video frame to obtain the predicted position of the target object. At the same time, the present application combines the position information and appearance information to match the target object and the detection object appearing in the current video frame. In other words, the present application can introduce the trajectory of temporal memory and combine it with the appearance features to reduce the risk of anomalies and errors in the multi-target tracking process, which is conducive to improving the accuracy and reliability of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of an application scenario of an embodiment of the multi-target tracking method of the present application;

[0024] Figure 2 It is a structural schematic diagram of an embodiment of a multi-target tracking device of the present application;

[0025] Figure 3 It is a structural schematic diagram of another embodiment of the multi-target tracking device of the present application;

[0026] Figure 4 It is a flowchart of an embodiment of a multi-target tracking method of the present application;

[0027] Figure 5 is a flowchart of another embodiment of the multi-target tracking method of the present application;

[0028] Figure 6 It is a flowchart of an embodiment of a feature matching method of the present application;

[0029] Figure 7 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] Object tracking is a technology that uses the contextual information of a video or image sequence to model the appearance and motion information of the target, thereby predicting the target's motion state and calibrating the target's position. It is an important basic problem in computer vision and has important theoretical research significance and application value. Object tracking estimates the position, shape or area occupied by the tracked target in a continuous video image sequence, determines the target's motion information such as speed, direction and trajectory, and realizes the analysis and understanding of the moving target's behavior in order to complete more advanced user tasks.

[0032] Target tracking is currently widely used in the fields of sports event broadcasting, security monitoring, drones, unmanned vehicles, robots, etc. However, it is found in this application that there are still many pain points and difficulties that affect the tracking effect. Specifically, posture change is a common interference problem in target tracking. When the posture of a moving target changes, its features and appearance model will change, which may easily lead to tracking failure. For example, athletes in sports competitions and pedestrians on the road. Scale adaptation is also a key issue in target tracking. When the target scale is reduced, it may be that the tracking frame cannot be adaptively tracked, and a lot of background information will be included, which may easily lead to update errors of the target model; when the target scale increases, it may be that the tracking frame cannot completely include the target, and it is easy to have incomplete target information in the tracking frame, which is also easy to cause update errors of the target model. At the same time, occlusion and disappearance may also occur, and the target may be occluded or disappear temporarily during the movement. When this happens, the tracking frame is likely to include the occluder and background information in the tracking frame, which may cause the tracking target in subsequent frames to drift onto the occluder. If the target is completely blocked, tracking may fail because the corresponding model of the target cannot be found. In addition, image blur may occur easily due to changes in light intensity, rapid target movement, low resolution, etc., especially when the moving target is similar to the background.

[0033] In multi-target tracking, a crucial link is the re-identification (Re-Identification, ReID) of the tracking target. The purpose is to identify and associate the same tracking target in multiple consecutive frames of video / images. In most tracking algorithms, the Tracking-by-Detection (TBD) method is used, that is, the detection results and a certain association matching strategy are used to match the detection targets of adjacent frames to achieve ReID. In most TBD multi-target tracking methods, a confidence threshold is set to filter out the detection frames in the detection results that are lower than the confidence threshold. This method is actually not reasonable, because when the target is blocked, partially disappeared, or blurred in the current frame, a low-confidence detection frame will be generated. After direct filtering, if the target reappears in the next frame, a new ID will be assigned, which causes the ID switch (ID Switch, IDS) of the tracking target, reducing the accuracy and efficiency of tracking.

[0034] For example, in the daily operation of a company, meetings of varying sizes are held every day, and the statistics of the number of participants in each meeting, the management of participants, and the analysis of the personnel composition during the meeting are important parts of conference affairs. At present, in order to address this problem, the method of participants signing in and conference staff counting is basically adopted, which takes up manpower and material resources.

[0035] In order to solve the technical problem that the target tracking accuracy and reliability are not well balanced in the related art, the present application provides a multi-target tracking method, a multi-target tracking device, a computer device and a computer-readable storage medium. Taking the application of the present application in a meeting as an example, the number of participants can be counted, and the company's face database can be connected to manage and analyze the participants. The multi-target tracking method includes: predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; using the predicted position of each target object and the forward appearance feature as the forward feature; wherein the forward appearance feature is the appearance feature of the target object in the forward video frame; obtaining the current position and current appearance features of several detection objects in the current video frame as the current feature; inputting the forward feature and the current feature into a multi-level matching model, matching the detection object with the target object, so as to track each target object. The specific device structure and working principle of the present application are described in detail below.

[0036] See also Figure 1 , Figure 1 It is a schematic diagram of an application scenario of an embodiment of the multi-target tracking method of the present application.

[0037] In one embodiment, the multi-target tracking method provided by the present application can be applied to Figure 1 In the application environment shown in the example.

[0038] Specifically, Figure 1 The application scenario shown as an example may include a terminal 102 and a server 104.

[0039] The terminal 102 can be used to shoot a video, and the video shot by it can be sent to the server 104 in the form of streaming media, etc. The video can be sent to the server 104 in real time when it is shot, or it can be sent to the server 104 later or in response to a task instruction of the user, which is not limited here.

[0040] Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices, or surveillance cameras, etc., depending on the application scenarios of the multi-target tracking method, such as online meetings, venues, sports games, road monitoring, etc., which are not limited here.

[0041] The terminal 102 can communicate with the server 104 through the network to transmit the video to the server 104.

[0042] The server 104 may be implemented as an independent server or a server cluster composed of multiple servers, so that the server 104 can perform multi-target tracking tasks on the received video.

[0043] The specific structure of the multi-target tracking device in this application is described in detail below.

[0044] See also Figure 2 , Figure 2 It is a structural diagram of an embodiment of a multi-target tracking device of the present application.

[0045] In one embodiment, the multi-target tracking device includes: a prediction module 21 , a storage module 22 , an extraction module 23 and a multi-level matching model 24 .

[0046] The prediction module 21 is used to predict the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object, wherein the trajectory is the historical real trajectory obtained by historically tracking the target object.

[0047] The storage module 22 is used to store the trajectory and forward appearance features of the target object as forward features; wherein the forward appearance features are appearance features of the target object in the forward video frame.

[0048] The extraction module 23 is used to obtain the current position and current appearance features of several detection objects in the current video frame as current features; the multi-level matching model 24 is used to input forward features and current features, match the detection objects with the target objects, and track each target object.

[0049] It can be seen that the multi-target tracking device in this embodiment introduces the existing trajectory of the target object to predict its position, that is, based on the existing trajectory of the target object, predicts its possible position in the current video frame to obtain the predicted position of the target object. At the same time, the present application combines the position information and appearance information to match the target object and the detection object appearing in the current video frame. In other words, the present application can introduce the trajectory of temporal memory and combine it with the appearance features to reduce the risk of anomalies and errors in the multi-target tracking process, which is conducive to improving the accuracy and reliability of the multi-target tracking device.

[0050] See also Figure 3 , Figure 3 It is a structural schematic diagram of another embodiment of the multi-target tracking device of the present application.

[0051] In one embodiment, the multi-stage matching model 24 may include a primary matching unit and a secondary matching unit.

[0052] Specifically, the first-level matching unit is used to input a first object and a forward feature to perform first feature matching; the first object is a detection object whose confidence score is higher than a confidence threshold.

[0053] The secondary matching unit is used to input the second object, the secondary trajectory and the forward appearance features of the corresponding target object to perform second feature matching; the second object is a detection object whose confidence score is lower than the confidence threshold; the secondary trajectory is the trajectory of the target object of the detection object that is not matched to the current video frame.

[0054] Please continue reading Figure 3 In one embodiment, the storage module 22 may include a track storage module 22 and an appearance matching module. The track storage module 22 is used to store the track, track identifier and track age of the target object; the appearance matching module is used to store the appearance features of the target object.

[0055] Optionally, the extraction module 23 may include a detector and a feature extractor. The detector is used to detect whether a target object exists, and obtain the existence probability of each detection object and the category probability of each target object; the feature extractor is used to extract features of the detection objects existing in the current video frame, and obtain the current appearance features belonging to each detection object.

[0056] Among them, the detector and the feature extractor can be integrated into the same model as an extractor, or the two can be independent models, which is not limited here.

[0057] For the specific definition of the multi-target tracking device, please refer to the definition of the multi-target tracking method below, which will not be repeated here. Each module in the above-mentioned multi-target tracking device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0058] The tracking principle of the multi-target tracking method of the present application is described in detail below.

[0059] See also Figure 4 , Figure 4 It is a flowchart of an embodiment of the multi-target tracking method of the present application.

[0060] S101: predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; wherein the trajectory is a historical real trajectory obtained by historically tracking the target object.

[0061] In this embodiment, the target object is a target that has been tracked before the current video frame. The trajectory of the target object can be considered as the trajectory of the target object moving in several forward video frames. The forward video frame is a video frame that occurs before the current video frame relative to the current video frame.

[0062] Optionally, in this embodiment, the forward video frame may be an adjacent forward video frame, in layman's terms, a previous video frame.

[0063] Predicting the current position of the target object based on its real historical trajectory can combine the motion trend information in the temporal memory, which is beneficial to improving the reliability of the predicted position.

[0064] S102: taking the predicted position and forward appearance features of each target object as forward features; wherein the forward appearance features are appearance features of the target object in the forward video frame.

[0065] In this embodiment, the predicted position of the target object and the forward appearance features are combined to perform target tracking by combining both position information and appearance information, which is beneficial to improving the target tracking accuracy and reliability.

[0066] Optionally, the predicted position may be formed into a position feature vector, and the forward appearance feature may be formed into a forward appearance feature vector to form a forward feature vector. Alternatively, the predicted position may be expressed in the form of coordinates, etc., and both may be used as forward features in the form of features, which is not limited here.

[0067] S103: Obtain current positions and current appearance features of several detection objects in the current video frame as current features.

[0068] In this embodiment, the target recognition can be performed on the current video frame through the pre-trained neural network model and algorithm, and the detected target is used as the detection object. In other words, the detection object is the target detected in the current video frame, which may be a target object that has been tracked at a historical moment, or a target object that appears for the first time since the current video frame, or even a non-target object that is captured by mistake, which will not be described in detail here.

[0069] Target recognition may be recognition of a target of a specified target type, and the target type may include one or more types, such as pedestrians, athletes, license plates, animals, skateboards, etc., which are not limited here.

[0070] When several detection objects are identified in the current video frame, the appearance features of the detection objects can be extracted to obtain the current appearance features of each detection object. The current position of the detection object can also be calculated, and the current position can be obtained through a pre-established coordinate system or a position algorithm, etc., which is not limited here.

[0071] S104: Input the forward feature and the current feature into a multi-level matching model to match the detected object with the target object to track each target object.

[0072] In this embodiment, the multi-level matching model can screen the detection objects based on preset conditions, and participate in the matching tasks of matching units at different levels respectively. The matching model with a relative level of one can reduce the redundant matching calculations, which is conducive to improving the matching efficiency and model stability.

[0073] Optionally, the multi-level matching model can be two-level, three-level, four-level, etc., and the current features input to the matching units at each level are the current features of part of the detection object.

[0074] Optionally, when the detection object is divided to adapt to the input of matching units at different levels, it can rely on the confidence score, the clarity of the detection object, the edge contour clarity of the detection object, and the richness of the appearance features, which are not limited here.

[0075] It can be seen that in this embodiment, the existing trajectory of the target object is introduced for position prediction, that is, based on the existing trajectory of the target object, its possible position in the current video frame is predicted to obtain the predicted position of the target object. At the same time, the target object and the detection object appearing in the current video frame are matched in this application in combination with the position information and the appearance information. In other words, this application can introduce the trajectory of temporal memory and combine it with the appearance features to reduce the risk of anomalies and errors in the multi-target tracking process, which is conducive to improving the accuracy and reliability of multi-target tracking.

[0076] See also Figure 5 , Figure 5 It is a flowchart of another embodiment of the multi-target tracking method of the present application.

[0077] In this embodiment, in response to the target object matching the detection object in the forward video frame, the forward trajectory of the forward video frame is obtained; based on the forward trajectory, the center point position of the target object in the current video frame is predicted to obtain the predicted position. Predicting the current position of the target object through the real historical trajectory can improve the reliability of the predicted position. In addition, the multi-target tracking method in this embodiment introduces position information, which can improve the detection efficiency of the detection object and is also conducive to improving the matching efficiency of the detection object and the target object.

[0078] Furthermore, the displacement of the target object in the first video frame and the second video frame can be obtained; wherein the first video frame is an adjacent forward video frame of the current video frame, and the second video frame is an adjacent forward video frame of the first video frame; the moving speed of the target object is calculated based on the displacement; the center point position of the target object in the first video frame is obtained; and the center point position of the target object in the current video frame is calculated using the center point position and the moving speed of the first video frame. As such, the detection and position prediction method based on the center point position in this embodiment has good computational efficiency, which is conducive to improving the lightweight deployment of edge terminals (such as monitoring and vehicle computers).

[0079] The predicted position and forward appearance features of each target object are used as forward features; wherein the forward appearance features are appearance features of the target object in the forward video frame.

[0080] The current video frame is input into the feature extractor, and the feature extractor detects the detection objects existing in the current video frame; features are extracted from the detected detection objects to obtain the current appearance features belonging to each detection object.

[0081] The confidence score of each detected object is calculated. Specifically, the current video frame can be input into the detector, and the detector detects whether there is a target object.

[0082] Optionally, the current video frame can be divided into multiple grids. At least one set of bounding boxes and the category probability corresponding to the detection object are predicted in each grid; the probability of the detection object existing in each bounding box is predicted based on the center point of each bounding box as the first prediction probability; the probability of the detection object existing in each bounding box is predicted based on the size of each bounding box as the second prediction probability; the first prediction probability and the second prediction probability are fused to obtain the existence probability.

[0083] Furthermore, the existence probability of each detection object and the category probability of each target object can be obtained; the product of the existence probability of the detection object and the highest category probability is calculated as the confidence score.

[0084] The confidence score and the confidence threshold are screened to divide the detected object into a first object and a second object.

[0085] Optionally, the detected objects whose confidence scores are higher than the confidence threshold may be taken as the first objects; and the detected objects whose confidence scores are lower than the confidence threshold may be taken as the second objects.

[0086] Based on the detection feature vector of the first object, a first detection list input to the first-level matching unit is constructed; based on the detection feature vector of the second object, a second detection list input to the second-level matching unit is constructed. Based on the target feature vector of the target object, a target list of the target object is constructed. The first detection list, the second detection list, and the target list are input to the multi-level matching model.

[0087] Optionally, the first detection list and the target list can be input into the primary matching unit, and the second detection list and the target list for deleting the target objects that have been matched to the detection object can be input into the secondary matching unit. Alternatively, the primary matching unit and the secondary matching unit are cascaded, and no additional manual / extra instructions are required to delete the target objects in the target list that have been matched to the detection object.

[0088] In this embodiment, the detection object with a confidence score higher than the confidence threshold can also be used as the first object; the first object and the forward feature are input into the first-level matching unit to perform first feature matching; wherein the feature matching dimensions include motion features and appearance features.

[0089] In response to the first object not matching the target object, the first object is used as a new target object, thereby reducing the possibility of missing the newly appeared target object, and then being able to reliably capture the target object that appears dynamically in the video, thereby improving the reliability and adaptability of the target tracking of this embodiment. For the new target object, a new trajectory of the target object can be created and a new trajectory identifier can be assigned, the current position of the new target object can be recorded, and the new trajectory identifier, the newly created trajectory and the current position of the new target object can be stored in the trajectory storage module. The current appearance features of the new target object are stored in the appearance matching module.

[0090] Optionally, in response to a detection object matching a target object, the current appearance features of the detection object can be stored in the appearance matching module; in the appearance matching module, the current appearance features of the detection object and the forward appearance features of the target object are fused, thereby improving the richness of the appearance features of the target object in the appearance matching module, so as to improve the reliability of the matching between the target object and the detection object, thereby improving the accuracy and reliability of multi-target tracking. Alternatively, the current appearance features of a detection object can be used to cover the forward appearance features of a target object, so as to ensure the timeliness of the current appearance features while saving the storage space of the appearance matching module.

[0091] Optionally, the appearance matching module may use a CNN (Convolutional Neural Network) with low computational complexity, such as ResNet-18 (the network architecture is ResNet, i.e., a deep residual network, and the network depth is 18 layers).

[0092] The appearance matching module can be used to extract the appearance features of the detection object in the detection frame. The current appearance features can also be represented by low-dimensional embedding (the process of mapping high-dimensional data to low-dimensional space). After each frame of detection + tracking, the target appearance features are extracted and saved.

[0093] In response to the track of the target object not being matched to the detected object of the current video frame, it is taken as a secondary track.

[0094] The detection object with a confidence score lower than the confidence threshold is taken as the second object; the second object, the secondary trajectory and the corresponding forward appearance feature of the target object are input into the secondary matching unit to perform second feature matching. That is to say, in this embodiment, the detection target and the target object can be matched through a two-level matching model, which can reduce the number of comparisons between the detection object with a low confidence and the target object. This can significantly reduce the number of comparisons in the multi-target tracking process, which is beneficial to reducing the amount of calculation in the matching process, thereby improving the matching efficiency between the detection object and the target object, and then improving the target tracking efficiency, that is, it is beneficial to improve the immediacy of multi-target tracking. At the same time, in the two-level matching process, the position information and the appearance information can be combined for matching, which increases the richness of the matching features, which is beneficial to reducing the matching complexity and improving the reliability of the matching results.

[0095] In response to the second object not matching the target object, it is discarded and no new track is created. This is because the second object is a detection object with a low confidence level, and it can be considered that the second object is relatively untrustworthy or a detection object that cannot be reliably identified due to other factors. Appropriately discarding untrustworthy data can reduce the occurrence of invalidation of previous track data, and not creating a new track can also save storage space of the track storage module.

[0096] In response to the target object not being matched to the detection object through the multi-level matching model, its trajectory age is increased by one; until the trajectory age of the target object reaches the age threshold, the trajectory of the target object is deleted, so that the older target object data in the storage module can be cleaned up in time, the storage space in the storage module can be released, the validity of the target object can be improved, and the invalid matching of the subsequent matching process can be reduced. While reducing the cumbersome invalid matching process, the matching efficiency can also be improved, thereby improving the efficiency of multi-target tracking.

[0097] For example, the age threshold can be set to 5-10. By setting a reasonable age threshold, the risk of excessive memory usage caused by a high setting can be reduced, and too many FNs (False Negatives) caused by a low setting can also be reduced.

[0098] Feature matching includes first feature matching and / or second feature matching. The specific principle of feature matching is described below.

[0099] In this embodiment, a target feature vector of each target object may be obtained; wherein the target feature vector includes a predicted position and a forward appearance feature of the spliced ​​target object.

[0100] A detection feature vector of each detection object is obtained; wherein the detection feature vector includes the current position and current appearance features of the spliced ​​detection object.

[0101] Calculate the similarity between each target feature vector and each detection feature vector respectively.

[0102] Based on similarity, the Hungarian algorithm or greedy algorithm is used to match the target object and the detection object.

[0103] It can be seen that in this embodiment, when matching the target object with the detection object, the appearance feature information and the position information are simultaneously integrated for matching. Feature fusion can combine the appearance features and motion features of the target object, which is conducive to more accurate matching. By expanding the feature matching dimension, the accuracy of the matching process can be enhanced, which in turn is conducive to improving the richness of the information provided during target recognition and matching.

[0104] After the first feature matching and the second feature matching are completed, the age track of the target object matched to the detection object can be updated. Specifically, in response to the existence of a target object matching the detection object, the current position can be input into the track storage module; the track storage module retains the original track identification of the matched target object and updates its track age to one, so that the track state can be updated in a timely manner, improving the flexibility of the multi-target tracking method of this embodiment, and facilitating the judgment of the effectiveness of the target object according to the track age. For example, in this embodiment, the target object that has lost its track for a period of time can also be discarded according to the track age to save the storage space of the storage module, and at the same time, the efficiency of the target object can be improved.

[0105] The current appearance feature is input into the appearance matching module, and the appearance matching module fuses the current appearance feature with the appearance feature of the matched target object.

[0106] The principle of feature matching in this application is described in a simple manner as follows.

[0107] Please refer to Figure 5 Also see Figure 6 , Figure 6 It is a flowchart of an embodiment of the feature matching method of the present application.

[0108] In one embodiment, feature matching may include, in sequence, screening of detection objects, building a list of detection objects to be matched, building a list of target objects to be matched, calculating the distance between the detection objects and the target objects, matching using the Hungarian algorithm or the greedy algorithm, and the like.

[0109] For the detection results of the detection object in the current frame, the detection object boxes with values ​​above the set threshold will be screened out in the first matching step, and the detection object boxes with values ​​below the set threshold will be screened out in the second matching step. The screened detection objects will be used for matching and some preprocessing operations will be performed to obtain the center position, speed information and category prediction of the detection objects.

[0110] According to the filtered detection object frames, a list of detection objects to be matched is constructed, and the dimension of the detection object list is [m, n]; where m is the number of detection objects detected in the tth frame, and n is the concatenation of position information and appearance features.

[0111] The position information and appearance features of the detected object in the t-1th frame are obtained from the trajectory storage module and the appearance feature auxiliary matching module respectively. The position information in the t-1th frame is predicted by the position information and speed information, and then concatenated with the appearance features. The dimension of the target object list to be matched is consistent with that of the detection object list to be matched.

[0112] According to the list of detection objects to be matched and the list of target objects to be matched, the Euclidean distance between each detection object in the list of detection objects to be matched and each target object in the list of target objects to be matched is calculated. The calculation formula of the Euclidean distance between vector a and vector b can be shown as follows:

[0113] Formula 1-1

[0114] Among them, n represents the vector dimension, ai and bi represent the elements of vector a and vector b in the i-th dimension respectively; vector a represents the detection feature vector; vector b represents the target feature vector.

[0115] The Hungarian algorithm or the greedy algorithm (which can be determined by the tracker parameter hungarian) is used to match the comprehensive similarity between the detected object and the target object. After matching, the target object and appearance features are stored and updated.

[0116] In summary, the multi-detection object tracking method of this application performs step-by-step ReID (Person re-identification) on the high and low confidence detection results, and introduces a trajectory preservation module with temporal memory and an appearance feature auxiliary matching module that uses the detection target features to assist in the matching of the detection target and the trajectory, which can reduce the mistracking, missed tracking, and logo switching during the tracking process, and improve the tracking accuracy and tracking efficiency. By applying it in conference scenarios, it can realize the statistics, management, and analysis of the participants, thereby saving related human, material, and computing resources.

[0117] In other words, the present application can use a step-by-step matching method for high- and low-confidence detection frames, first matching the high-confidence detection frame with the trajectory, and then matching the low-confidence detection frame with the trajectory. At the same time, a temporal memory module is introduced to save the trajectory and detection features for several forward frame videos to prevent the target in the current video frame from being completely blocked and having no detection frame, and then being re-detected in the next frame to increase IDS (ID Switch, switching of the tracking target object ID), and can use the similarity of the detection features to assist ReID, solving the problem of low-confidence detection frames being directly filtered out during the tracking process. In the above manner, the present application can improve the precision and accuracy of multi-target tracking.

[0118] It should be understood that although Figure 4-Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 4-Figure 6At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0119] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of a computer device of the present application.

[0120] In one embodiment, a computer device is provided. The computer device may be a device with computing capability such as a server or a terminal.

[0121] For example, the computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown in the example. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-target tracking method is implemented.

[0122] Those skilled in the art will understand that Figure 7 The structure shown as an example is merely a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the computer device to which the present application solution is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps may be implemented:

[0124] S101: predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; wherein the trajectory is a historical real trajectory obtained by historically tracking the target object.

[0125] S102: taking the predicted position and forward appearance features of each target object as forward features; wherein the forward appearance features are appearance features of the target object in the forward video frame.

[0126] S103: Obtain current positions and current appearance features of several detection objects in the current video frame as current features.

[0127] S104: Input the forward feature and the current feature into a multi-level matching model to match the detected object with the target object to track each target object.

[0128] In one embodiment, the multi-level matching model includes a primary matching unit and a secondary matching unit; when executing step S104, the processor may further implement the following steps when executing the computer program:

[0129] The detection object with a confidence score higher than the confidence threshold is taken as the first object; the first object and the forward feature are input into the primary matching unit to perform the first feature matching; wherein the feature matching dimension includes motion features and appearance features; in response to the trajectory of the target object not matching the detection object of the current video frame, it is taken as the secondary trajectory; the detection object with a confidence score lower than the confidence threshold is taken as the second object; the second object, the secondary trajectory and the corresponding forward appearance feature of the target object are input into the secondary matching unit to perform the second feature matching.

[0130] In one embodiment, after the first object and the forward feature are input into a primary matching unit and the first feature matching is performed, the processor can also implement the following steps when executing the computer program: in response to the first object not being matched to the target object, taking it as a new target object; creating a trajectory of the new target object and assigning a new trajectory identifier, recording the current position of the new target object, and storing the new trajectory identifier, the newly created trajectory and the current position of the new target object in a trajectory storage module; storing the current appearance features of the new target object in an appearance matching module.

[0131] In one embodiment, the second object, the secondary trajectory and the corresponding forward appearance feature of the target object are input into the secondary matching unit, and after the second feature matching is performed, the processor may further implement the following steps when executing the computer program:

[0132] In response to the second object not matching the target object, discarding it and not creating a new track; in response to the target object not matching the detection object through the multi-level matching model, increasing its track age by one; until the track age of the target object reaches an age threshold, deleting the track of the target object.

[0133] In one embodiment, the feature matching includes the first feature matching and / or the second feature matching; when performing the feature matching, the processor may further implement the following steps when executing the computer program:

[0134] Obtain a target feature vector of each of the target objects; wherein the target feature vector includes the predicted position and forward appearance features of the spliced ​​target objects; obtain a detection feature vector of each of the detection objects; wherein the detection feature vector includes the current position and current appearance features of the spliced ​​detection objects; respectively calculate the similarity between each of the target feature vectors and each detection feature vector; match the target object and the detection object based on the similarity using a Hungarian algorithm or a greedy algorithm.

[0135] In one embodiment, after matching the detection object with the target object, the processor may further implement the following steps when executing the computer program:

[0136] In response to the existence of the target object matching the detection object, the current position is input into a trajectory storage module; the trajectory storage module retains the original trajectory identification of the matched target object and updates its trajectory age to one; the current appearance feature is input into an appearance matching module, and the appearance matching module fuses the current appearance feature with the appearance feature of the matched target object.

[0137] In one embodiment, when executing step S104, the processor may further implement the following steps when executing the computer program:

[0138] Calculate the confidence score of each of the detected objects; screen the confidence score and the confidence threshold to divide the detected objects into a first object and a second object; construct a first detection list input to a first-level matching unit based on the detection feature vector of the first object; construct a second detection list input to a second-level matching unit based on the detection feature vector of the second object; construct a target list of the target object based on the target feature vector of the target object; input the first detection list, the second detection list and the target list into the multi-level matching model.

[0139] In one embodiment, when calculating the confidence score of each of the detected objects, the processor may further implement the following steps when executing the computer program:

[0140] The current video frame is input into the detector, and the detector detects whether there is a target object, obtains the existence probability of each detected object and the category probability belonging to each target object; calculates the product of the existence probability of the detected object and the highest category probability as the confidence score.

[0141] In one embodiment, when the detector detects whether a target object exists, the processor may further implement the following steps when executing the computer program:

[0142] The current video frame is divided into a plurality of grids; at least one set of bounding boxes and the category probabilities corresponding to the detection objects are predicted in each of the grids; the probability of the detection objects existing in each of the bounding boxes is predicted based on the center point of each of the bounding boxes as a first prediction probability; the probability of the detection objects existing in each of the bounding boxes is predicted based on the size of each of the bounding boxes as a second prediction probability; and the first prediction probability and the second prediction probability are combined to obtain the existence probability.

[0143] In one embodiment, when executing step S101, the processor may further implement the following steps when executing the computer program:

[0144] In response to the target object matching the detection object in the forward video frame, a forward trajectory of the forward video frame is acquired; and based on the forward trajectory, a center point position of the target object in the current video frame is predicted to obtain the predicted position.

[0145] In one embodiment, when predicting the center point position of the target object in the current video frame based on the forward trajectory to obtain the predicted position, the processor may further implement the following steps when executing the computer program:

[0146] Obtain the displacement of a target object in a first video frame and a second video frame; wherein the first video frame is an adjacent forward video frame of a current video frame, and the second video frame is an adjacent forward video frame of the first video frame; calculate the moving speed of the target object based on the displacement; obtain the center point position of the target object in the first video frame; and calculate the center point position of the target object in the current video frame using the center point position and the moving speed of the first video frame.

[0147] In one embodiment, when obtaining the current appearance features of the plurality of detection objects in the current video frame in step S103, the processor may further implement the following steps when executing the computer program:

[0148] The current video frame is input into a feature extractor, and the feature extractor detects detection objects in the current video frame; features are extracted from a plurality of the detected detection objects to obtain current appearance features belonging to each of the detection objects.

[0149] In one embodiment, the processor may further implement the following steps when executing the computer program:

[0150] After matching the detection object with the target object, the method also includes: in response to a detection object matching with a target object, storing a current appearance feature of a detection object in an appearance matching module; in the appearance matching module, fusing the current appearance feature of a detection object with a forward appearance feature of a target object; or, using the current appearance feature of a detection object to cover the forward appearance feature of a target object.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0152] S101: predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; wherein the trajectory is a historical real trajectory obtained by historically tracking the target object.

[0153] S102: taking the predicted position and forward appearance features of each target object as forward features; wherein the forward appearance features are appearance features of the target object in the forward video frame.

[0154] S103: Obtain current positions and current appearance features of several detection objects in the current video frame as current features.

[0155] S104: Input the forward feature and the current feature into a multi-level matching model to match the detected object with the target object to track each target object.

[0156] In one embodiment, the multi-level matching model includes a primary matching unit and a secondary matching unit; when executing step S104, the computer program can also implement the following steps when executed by the processor:

[0157] The detection object with a confidence score higher than the confidence threshold is taken as the first object; the first object and the forward feature are input into the primary matching unit to perform the first feature matching; wherein the feature matching dimension includes motion features and appearance features; in response to the trajectory of the target object not matching the detection object of the current video frame, it is taken as the secondary trajectory; the detection object with a confidence score lower than the confidence threshold is taken as the second object; the second object, the secondary trajectory and the corresponding forward appearance feature of the target object are input into the secondary matching unit to perform the second feature matching.

[0158] In one embodiment, after the first object and the forward feature are input into a primary matching unit and the first feature matching is performed, the computer program can also implement the following steps when executed by the processor: in response to the first object not being matched to the target object, taking it as a new target object; creating a trajectory of the new target object and assigning a new trajectory identifier, recording the current position of the new target object, and storing the new trajectory identifier, the newly created trajectory and the current position of the new target object in a trajectory storage module; and storing the current appearance features of the new target object in an appearance matching module.

[0159] In one embodiment, the second object, the secondary trajectory and the corresponding forward appearance feature of the target object are input into the secondary matching unit, and after the second feature matching is performed, the computer program can also implement the following steps when executed by the processor:

[0160] In response to the second object not matching the target object, discarding it and not creating a new track; in response to the target object not matching the detection object through the multi-level matching model, increasing its track age by one; until the track age of the target object reaches an age threshold, deleting the track of the target object.

[0161] In one embodiment, the feature matching includes the first feature matching and / or the second feature matching; when performing the feature matching, the computer program may further implement the following steps when executed by the processor:

[0162] Obtain a target feature vector of each of the target objects; wherein the target feature vector includes the predicted position and forward appearance features of the spliced ​​target objects; obtain a detection feature vector of each of the detection objects; wherein the detection feature vector includes the current position and current appearance features of the spliced ​​detection objects; respectively calculate the similarity between each of the target feature vectors and each detection feature vector; match the target object and the detection object based on the similarity using a Hungarian algorithm or a greedy algorithm.

[0163] In one embodiment, after matching the detection object with the target object, the computer program may further implement the following steps when executed by the processor:

[0164] In response to the existence of the target object matching the detection object, the current position is input into a trajectory storage module; the trajectory storage module retains the original trajectory identification of the matched target object and updates its trajectory age to one; the current appearance feature is input into an appearance matching module, and the appearance matching module fuses the current appearance feature with the appearance feature of the matched target object.

[0165] In one embodiment, when executing step S104, the computer program may further implement the following steps when executed by the processor:

[0166] Calculate the confidence score of each of the detected objects; screen the confidence score and the confidence threshold to divide the detected objects into a first object and a second object; construct a first detection list input to a first-level matching unit based on the detection feature vector of the first object; construct a second detection list input to a second-level matching unit based on the detection feature vector of the second object; construct a target list of the target object based on the target feature vector of the target object; input the first detection list, the second detection list and the target list into the multi-level matching model.

[0167] In one embodiment, when calculating the confidence score of each of the detected objects, the computer program may further implement the following steps when executed by the processor:

[0168] The current video frame is input into the detector, and the detector detects whether there is a target object, obtains the existence probability of each detected object and the category probability belonging to each target object; calculates the product of the existence probability of the detected object and the highest category probability as the confidence score.

[0169] In one embodiment, when the detector detects whether a target object exists, the computer program may further implement the following steps when executed by the processor:

[0170] The current video frame is divided into a plurality of grids; at least one set of bounding boxes and the category probabilities corresponding to the detection objects are predicted in each of the grids; the probability of the detection objects existing in each of the bounding boxes is predicted based on the center point of each of the bounding boxes as a first prediction probability; the probability of the detection objects existing in each of the bounding boxes is predicted based on the size of each of the bounding boxes as a second prediction probability; and the first prediction probability and the second prediction probability are combined to obtain the existence probability.

[0171] In one embodiment, when executing step S101, the computer program may further implement the following steps when executed by the processor:

[0172] In response to the target object matching the detection object in the forward video frame, a forward trajectory of the forward video frame is acquired; and based on the forward trajectory, a center point position of the target object in the current video frame is predicted to obtain the predicted position.

[0173] In one embodiment, when the center point position of the target object in the current video frame is predicted based on the forward trajectory to obtain the predicted position, the computer program may further implement the following steps when executed by the processor:

[0174] Obtain the displacement of a target object in a first video frame and a second video frame; wherein the first video frame is an adjacent forward video frame of a current video frame, and the second video frame is an adjacent forward video frame of the first video frame; calculate the moving speed of the target object based on the displacement; obtain the center point position of the target object in the first video frame; and calculate the center point position of the target object in the current video frame using the center point position and the moving speed of the first video frame.

[0175] In one embodiment, when obtaining the current appearance features of the plurality of detection objects in the current video frame in step S103, the computer program may further implement the following steps when executed by the processor:

[0176] The current video frame is input into a feature extractor, and the feature extractor detects detection objects in the current video frame; features are extracted from a plurality of the detected detection objects to obtain current appearance features belonging to each of the detection objects.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps may also be implemented:

[0178] After matching the detection object with the target object, the method also includes: in response to a detection object matching with a target object, storing a current appearance feature of a detection object in an appearance matching module; in the appearance matching module, fusing the current appearance feature of a detection object with a forward appearance feature of a target object; or, using the current appearance feature of a detection object to cover the forward appearance feature of a target object.

[0179] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A multi-target tracking method, characterized in that: The multi-target tracking method comprises: Predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; wherein the trajectory is a historical real trajectory obtained by historically tracking the target object; The predicted position and forward appearance feature of each target object are used as forward features; wherein the forward appearance feature is the appearance feature of the target object in the forward video frame; Obtaining current positions and current appearance features of a plurality of detection objects in the current video frame as current features; Inputting the forward feature and the current feature into a multi-level matching model, matching the detected object with the target object, so as to track each of the target objects; Dividing the detection object based on the confidence score to input the current features of the detection object into matching units at different levels of the multi-level matching model; The multi-level matching model includes a primary matching unit and a secondary matching unit; Inputting the forward feature and the current feature into a multi-level matching model to match the detection object with the target object comprises: The detected object with a confidence score higher than the confidence threshold is taken as the first object; the first object and the forward feature are input into the primary matching unit to perform first feature matching; wherein the feature matching dimension includes motion features and appearance features; In response to the trajectory of the target object not matching the detected object of the current video frame, treating it as a secondary trajectory; The detected object with a confidence score lower than the confidence threshold is taken as the second object; the second object, the secondary track and the corresponding forward appearance feature of the target object are input into the secondary matching unit to perform second feature matching.

2. The multi-target tracking method according to claim 1, characterized in that: After inputting the first object and the forward feature into a primary matching unit and performing first feature matching, the method further includes: In response to the first object not being matched to the target object, taking the first object as a new target object; Creating a new trajectory of the new target object and assigning a new trajectory identifier, recording the current position of the new target object, and storing the new trajectory identifier, the created trajectory, and the current position of the new target object in a trajectory storage module; The current appearance features of the new target object are stored in the appearance matching module.

3. The multi-target tracking method according to claim 1, characterized in that: After inputting the second object, the secondary track and the corresponding forward appearance feature of the target object into a secondary matching unit and performing second feature matching, the following further comprises: In response to the second object not being matched to the target object, discarding it and not creating a new track; In response to the target object not being matched to the detection object through the multi-level matching model, the trajectory age of the target object is increased by one; until the trajectory age of the target object reaches an age threshold, the trajectory of the target object is deleted.

4. The multi-target tracking method according to claim 1, characterized in that: Feature matching includes the first feature matching and / or the second feature matching; The feature matching includes: Obtaining a target feature vector of each target object; wherein the target feature vector includes the predicted position and forward appearance features of the spliced ​​target object; Acquire a detection feature vector of each of the detection objects; wherein the detection feature vector includes the current position and current appearance features of the spliced ​​detection objects; Calculating the similarity between each target feature vector and each detection feature vector respectively; The target object and the detection object are matched based on the similarity using the Hungarian algorithm or the greedy algorithm.

5. The multi-target tracking method according to claim 1, characterized in that: After matching the detection object with the target object, the following step further includes: In response to the existence of the target object matching the detection object, the current position is input into a track storage module; the track storage module retains the original track identifier of the matching target object and updates its track age to one; The current appearance feature is input into an appearance matching module, and the appearance matching module fuses the current appearance feature with the appearance feature of the matched target object.

6. The multi-target tracking method according to claim 1, characterized in that: The inputting the forward feature and the current feature into a multi-level matching model comprises: Calculating a confidence score for each of the detection objects; Screening the confidence score and the confidence threshold to divide the detected object into a first object and a second object; Based on the detection feature vector of the first object, a first detection list input to the primary matching unit is constructed; based on the detection feature vector of the second object, a second detection list input to the secondary matching unit is constructed; based on the target feature vector of the target object, a target list of the target object is constructed; The first detection list, the second detection list, and the target list are input into the multi-level matching model.

7. The multi-target tracking method according to claim 6, characterized in that: The calculating the confidence score of each of the detected objects comprises: Input the current video frame into a detector, the detector detects whether there is a target object, and obtains the existence probability of each detected object and the category probability of each target object; The product of the existence probability of the detected object and the highest category probability is calculated as the confidence score.

8. The multi-target tracking method according to claim 7, characterized in that: The detector detecting whether a target object exists includes: Dividing the current video frame into a plurality of grids; Predicting at least one set of bounding boxes and class probabilities corresponding to the detected objects in each of the grids; Predicting the probability of the detection object existing in each of the bounding boxes based on the center point of each of the bounding boxes as a first prediction probability; predicting the probability of the detection object existing in each of the bounding boxes based on the size of each of the bounding boxes as a second prediction probability; The first predicted probability and the second predicted probability are integrated to obtain the existence probability.

9. The multi-target tracking method according to claim 1, characterized in that: The predicting of the predicted positions of multiple target objects in the current video frame based on the trajectories of the target objects includes: In response to the target object matching the detection object in the forward video frame, acquiring a forward trajectory of the forward video frame; The center point position of the target object in the current video frame is predicted based on the forward trajectory to obtain the predicted position.

10. The multi-target tracking method according to claim 9, characterized in that: The predicting the center point position of the target object in the current video frame based on the forward trajectory to obtain the predicted position includes: Acquire the displacement of the target object in a first video frame and a second video frame; wherein the first video frame is an adjacent forward video frame of the current video frame, and the second video frame is an adjacent forward video frame of the first video frame; Calculating a moving speed of the target object based on the displacement; Obtaining a center point position of the target object in the first video frame; The center point position of the target object in the current video frame is calculated by using the center point position of the first video frame and the moving speed.

11. The multi-target tracking method according to claim 1, characterized in that: Acquiring current appearance features of several detected objects in the current video frame includes: Inputting the current video frame into a feature extractor, wherein the feature extractor detects a detection object existing in the current video frame; Feature extraction is performed on the detected multiple detection objects to obtain current appearance features belonging to each of the detection objects.

12. The multi-target tracking method according to claim 1 or 11, characterized in that: After matching the detection object with the target object, the following step further includes: In response to a detection object matching a target object, storing a current appearance feature of the detection object in an appearance matching module; In the appearance matching module, the current appearance feature of the detection object is fused with the forward appearance feature of the target object; or the forward appearance feature of the target object is covered by the current appearance feature of the detection object.

13. A multi-target tracking device, characterized in that: The multi-target tracking device comprises: A prediction module, used for predicting the predicted positions of multiple target objects in the current video frame based on the trajectory of the target object; wherein the trajectory is a historical real trajectory obtained by historically tracking the target object; A storage module, used to store the trajectory and forward appearance features of the target object as forward features; wherein the forward appearance features are appearance features of the target object in the forward video frame; An extraction module, used for obtaining the current positions and current appearance features of a plurality of detection objects in the current video frame as current features; A multi-level matching model, used for inputting the forward feature and the current feature, matching the detection object with the target object, so as to track each of the target objects; Dividing the detection object based on the confidence score to input the current features of the detection object into matching units at different levels of the multi-level matching model; The multi-level matching model includes: A primary matching unit, used for inputting a first object and the forward feature to perform first feature matching; the first object is a detection object whose confidence score is higher than a confidence threshold; The secondary matching unit is used to input a second object, a secondary trajectory and the corresponding forward appearance feature of the target object to perform a second feature matching; the second object is a detection object whose confidence score is lower than a confidence threshold; the secondary trajectory is the trajectory of the target object that is not matched to the detection object of the current video frame.

14. The multi-target tracking device according to claim 13, characterized in that: The storage module comprises: A trajectory storage module, used to store the trajectory, trajectory identifier and trajectory age of the target object; An appearance matching module, used to store the appearance features of the target object; The extraction module comprises: The detector is used to detect whether a target object exists, and obtain the existence probability of each detected object and the category probability of each target object; The feature extractor is used to extract features of the detection objects existing in the current video frame to obtain current appearance features belonging to each of the detection objects.

15. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-target tracking method according to any one of claims 1 to 12 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-target tracking method according to any one of claims 1 to 12 are implemented.

Images