Multi-target tracking method and computer program product

Abstract

This application discloses a multi-target tracking method and computer program product, relating to the field of computer vision technology. The method comprises: inputting a target image sequence into a target detector to obtain target detection results output by the target detector; obtaining an appearance feature vector and a motion feature vector of the detection object in the current image frame; obtaining a first cost matrix between the appearance feature vector and the target trajectory to be matched, and obtaining a second cost matrix between the motion feature vector and the target trajectory; determining a first detection object corresponding to the target trajectory based on the first cost matrix, and determining a second detection object corresponding to the target trajectory based on the second cost matrix; and, if the first detection object and the second detection object are different objects, determining the target detection object corresponding to the target trajectory in the current image frame based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector, and the target trajectory. This method can improve the robustness and accuracy of multi-target tracking.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a multi-target tracking method and a computer program product. Background Art

[0002] Multiple object tracking (MOT) is a key technology in computer vision, widely used in autonomous driving, video surveillance, behavior recognition, and other fields. In multi-object tracking, factors such as the scale, deformation, occlusion, rapid motion, and lighting changes of the tracked objects can affect the tracking results.

[0003] When matching the detected target with the trajectory target by constructing a cost matrix, target conflicts are likely to occur. In related technologies, most rely on the velocity information of the trajectory to resolve conflicts. However, this method is susceptible to tracking noise and may lead to matching errors, thus affecting the stability and accuracy of multi-target tracking. Summary of the Invention

[0004] The present application provides a multi-target tracking method and a computer program product to at least solve the problems in the related art such as being susceptible to tracking noise and causing matching errors.

[0005] This application provides a multi-target tracking method, comprising the following steps:

[0006] Inputting a target image sequence into a target detector to obtain a target detection result output by the target detector, wherein the target detection result includes a detection object of each image frame in the target image sequence;

[0007] Obtain the appearance feature vector and motion feature vector of the detected object in the current image frame;

[0008] Obtaining a first cost matrix between the appearance feature vector and the target trajectory to be matched, and obtaining a second cost matrix between the motion feature vector and the target trajectory;

[0009] Determining a first detection object corresponding to the target trajectory based on the first cost matrix, and determining a second detection object corresponding to the target trajectory based on the second cost matrix;

[0010] In the case that the first detection object and the second detection object are different objects, the target detection object corresponding to the target trajectory in the current image frame is determined based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector and the target trajectory.

[0011] The present application also provides a computer program product, comprising:

[0012] a first processing module, configured to input a target image sequence into a target detector, and obtain a target detection result output by the target detector, wherein the target detection result includes a detection object of each image frame in the target image sequence;

[0013] The second processing module is used to obtain the appearance feature vector and motion feature vector of the detected object in the current image frame;

[0014] a third processing module, configured to obtain a first cost matrix between the appearance feature vector and the target trajectory to be matched, and to obtain a second cost matrix between the motion feature vector and the target trajectory;

[0015] a fourth processing module, configured to determine a first detection object corresponding to the target trajectory based on the first cost matrix, and determine a second detection object corresponding to the target trajectory based on the second cost matrix;

[0016] a fifth processing module, for determining, when the first detection object and the second detection object are different objects, the target detection object corresponding to the target trajectory in the current image frame based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector and the target trajectory.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned multi-target tracking methods when executing the computer program.

[0018] The present application also provides a non-volatile computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned multi-target tracking methods are implemented.

[0019] Through the present application, for multi-target tracking, a parallel mode tracking is adopted to calculate the cost matrix of the appearance feature vector and the motion feature vector respectively. When the first detection object and the second detection object are different objects, that is, a tracking conflict occurs, the first detection object, the appearance feature vector, the second detection object, the motion feature vector and the target trajectory and other information are integrated. Starting from the essence of the problem, that is, tracking matching is the matching of the detection target and the tracking trajectory, the target trajectory is used as a benchmark, and a conflict judgment is performed based on the detection results and the tracking trajectory. The conflict degree (or matching degree) between the first detection object corresponding to the appearance feature vector and the target trajectory, and the conflict degree (or matching degree) between the second detection object corresponding to the motion feature vector and the target trajectory are judged. The appearance feature vector and the motion feature vector are integrated and compared from the two aspects of vision and trajectory speed. The complementarity between different features is fully utilized, and the additional noise caused by the mechanical combination of different features in traditional serial tracking is reduced. The final tracking result of the target trajectory is determined, which can achieve the technical effect of improving the robustness and accuracy of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 One of the flow charts of the multi-target tracking method provided in the embodiment of the present application;

[0022] Figure 2 The second flowchart of the multi-target tracking method provided in the embodiment of the present application;

[0023] Figure 3 The third flowchart of the multi-target tracking method provided in the embodiment of the present application;

[0024] Figure 4 A schematic diagram of the structure of a computer program product provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0028] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] An embodiment of the present application provides a multi-target tracking method to at least solve the problem of matching errors caused by tracking noise in related technologies. It performs parallel tracking association of multiple targets based on appearance features and motion features, fully utilizes the complementarity between different features, and reduces the additional noise caused by the mechanical combination of different features in traditional serial tracking.

[0030] The multi-target tracking method provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the multi-target tracking method. The electronic devices mentioned in the embodiments of the present application include but are not limited to computers, servers, etc.

[0031] like Figure 1 As shown, the multi-target tracking method provided in the embodiment of the present application includes steps 110 to 150.

[0032] Step 110: Input the target image sequence into the target detector to obtain the target detection result output by the target detector.

[0033] The target detection result includes the detected objects of each image frame in the target image sequence.

[0034] It can be understood that the target image sequence is a dynamic visual data stream consisting of a plurality of consecutive image frames arranged in time sequence.

[0035] In this step, a target detector is used to perform target detection, and the target image sequence is input into the target detector frame by frame. The target detection results output by the target detector include one or more detection objects in each image frame. The target detection results can also include a detection box for locating the detection object in the image frame, a label corresponding to the detection object, and confidence information.

[0036] Step 120: Obtain the appearance feature vector and motion feature vector of the detected object in the current image frame.

[0037] Perform target detection on the image frame, perform feature extraction on the detection object obtained by target detection, and obtain the appearance feature vector and motion feature vector of the detection object. Among them, the appearance feature vector is a feature representation used to characterize the visual characteristics of the detection object such as color, texture, shape, etc. The motion feature vector is used to characterize the spatiotemporal motion law of the detection object, including the position information of the detection object and the position information obtained by tracking trajectory prediction.

[0038] In this embodiment, the detected object in the current image frame may be input into the appearance feature extraction module and the motion feature extraction module respectively for feature extraction to obtain the appearance feature vector and the motion feature vector of the detected object in the current image frame.

[0039] In actual implementation, the target detector can be a detector in the Tracking-by-Detection (TBD) tracking mode, such as the YOLO series detector, which inputs the detection object into the feature extraction module for feature extraction; the target detector can also be a detector in the Joint-Detection-and-Tracking (JDT) tracking mode, which can simultaneously obtain the detection object and its corresponding appearance feature vector. The specific method can be determined according to actual needs, and the embodiments of the present application are not limited to this.

[0040] For example, the TBD tracking mode is used for tracking, and the detected objects are input into the appearance feature extraction module and the motion feature extraction module respectively for feature extraction.

[0041] The appearance feature extraction module may be any Re-Identification (ReID) feature extraction module, such as using a fast-ReID model to extract appearance features.

[0042] Step 130: Obtain a first cost matrix between the appearance feature vector and the target trajectory to be matched, and obtain a second cost matrix between the motion feature vector and the target trajectory.

[0043] In this step, the respective cost matrices are calculated based on the appearance feature vector and the motion feature vector, so as to facilitate the subsequent trajectory matching according to the cost matrix.

[0044] Among them, the cost matrix is ​​used to quantify the matching degree between different detection objects and trajectories, and can be expressed as a two-dimensional matrix.

[0045] It can be understood that an image frame may include multiple detection objects, and the corresponding appearance feature vector and motion feature vector can be extracted for each detection object. The first cost matrix between the appearance feature vector of a detection object and the target trajectory to be matched, as well as the second cost matrix between the motion feature vector of the detection object and the target trajectory, are calculated to measure the degree of mismatch between the detection object and the target trajectory.

[0046] It should be noted that the cost matrix quantifies the degree of mismatch between two targets, and the feature similarity measures the degree of match between two targets. The cost matrix and feature similarity can be associated through mathematical transformation.

[0047] In this embodiment, the feature similarity distance may be calculated based on the appearance feature vector and the motion feature vector respectively.

[0048] In actual implementation, the appearance feature similarity distance corresponding to the appearance feature vector can be calculated using the cosine distance, and the calculation formula is as follows:

[0049]

[0050] in, Indicates the detection object i In the t Appearance feature vector of each image frame and tracking trajectory j (i.e. the target trajectory to be matched) corresponding appearance feature template The feature similarity distance between them.

[0051] It is understandable that The larger the distance value, the closer the detection object is. i and tracking trajectories j The more dissimilar (i.e. mismatch) The smaller the distance value, the closer the detection object is. i and tracking trajectories j The more similar (i.e., matching) they are.

[0052] The motion feature similarity distance corresponding to the motion feature vector can be calculated using the Intersection over Union (IOU) distance, and the calculation formula is as follows:

[0053]

[0054] in, Indicates the detection object i In the t Motion feature vector and tracking trajectory of each image frame j The feature similarity distance between the motion feature templates corresponding to the target trajectory to be matched, Indicates the t Detecting objects in a frame i The detection frame position, Represents tracking trajectory j The corresponding detection frame position (i.e. motion feature template).

[0055] In actual implementation, for the current image frame, the similarity distance between all detected objects and each target track can be traversed and calculated (the appearance feature vector can use the cosine distance, and the motion feature vector can use the IOU distance) to obtain the first cost matrix and the second cost matrix, which is convenient for the subsequent use of methods such as the Hungarian matching method to perform tracking and matching association, and the tracking results are judged based on the association results.

[0056] Step 140: Determine a first detection object corresponding to the target trajectory based on the first cost matrix, and determine a second detection object corresponding to the target trajectory based on the second cost matrix.

[0057] In this step, tracking matching is performed based on the first cost matrix corresponding to the appearance feature vector to obtain a first detection object corresponding to the target trajectory, wherein the first detection object belongs to the appearance tracking result; tracking matching is performed based on the second cost matrix corresponding to the motion feature vector to obtain a second detection object corresponding to the target trajectory, wherein the second detection object belongs to the motion tracking result.

[0058] In this embodiment, based on the first cost matrix and the second cost matrix, a method such as the Hungarian matching method may be used to perform tracking and matching to obtain an appearance tracking result and a motion tracking result.

[0059] It can be understood that after tracking and matching the target trajectory and obtaining the first detection object and the second detection object corresponding to the target trajectory, it can be determined whether the first detection object and the second detection object are the same object in the image frame, that is, whether the appearance tracking result and the motion tracking result conflict.

[0060] Step 150: When the first detection object and the second detection object are different objects, determine the target detection object corresponding to the target trajectory in the current image frame based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector and the target trajectory.

[0061] In this step, the first detection object and the second detection object are different objects, the appearance tracking result and the motion tracking result conflict, and two detection objects appear in the current image frame and are highly matched with the unified target trajectory, resulting in the system being unable to clearly assign a corresponding relationship (that is, which detection object corresponds to the target trajectory in the current image frame). At this time, conflict resolution is performed based on the first detection object, appearance feature vector, second detection object, motion feature vector and target trajectory.

[0062] In this embodiment, in the tracking conflict processing link, based on the first detection object, the appearance feature vector and the target trajectory, the conflict degree (or matching degree) between the first detection object and the target trajectory is quantified; based on the second detection object, the motion feature vector and the target trajectory, the conflict degree (or matching degree) between the second detection object and the target trajectory is quantified, and it is identified which of the first detection object and the second detection object is more matched with the target trajectory and which one is more conflicting with the target trajectory, so as to eliminate tracking ambiguity and obtain a unique and reasonable matching result, thereby maintaining the continuity and accuracy of tracking.

[0063] It should be noted that in the detection-based TBD tracking mode, due to the significant improvement in detector performance, the research focus has shifted to the optimization of data association algorithms. For example, classic tracking methods include Sort and Deep-Sort. The core of the sort algorithm lies in Kalman filtering and Hungarian matching, and a single feature is used to calculate the cost matrix. The Deep-Sort algorithm adds cascade matching (Matching Cascade) and new trajectory confirmation on the basis of the Sort algorithm. The similarity cost matrix of cascade matching uses hyperparameters to combine two different features into one. For this type of algorithm, noise from either of the two different features will affect the matrix cost value, thereby affecting the final trajectory matching result.

[0064] In related technologies, some tracking methods calculate the cost matrix for two different features separately. When there is a conflict between the targets associated with the two, the idea of ​​motion estimation is used. When the distance between two adjacent frames is close, the speed is considered to be small, and the matching results of the motion features are trusted. When the distance between two adjacent frames is far, the speed is considered to be large, and the matching results of features other than the motion features are trusted more. This type of method ignores the fact that the actual cost matrix is ​​the matching of the detection target and the trajectory target. The conflict problem is judged solely based on the trajectory speed, which is easily affected by tracking noise, resulting in tracking matching errors, thereby affecting the stability and accuracy of multi-target tracking.

[0065] In an embodiment of the present application, for multi-target tracking, a parallel mode tracking is adopted in which the cost matrices of the appearance feature vector and the motion feature vector are calculated respectively. When the first detection object and the second detection object are different objects, that is, a tracking conflict occurs, information such as the first detection object, the appearance feature vector, the second detection object, the motion feature vector and the target trajectory are integrated. Starting from the essence of the problem, that is, tracking matching is the matching of the detection target and the tracking trajectory, the target trajectory is used as a benchmark, and a conflict judgment is performed based on the detection results and the tracking trajectory. The degree of conflict (or matching degree) between the first detection object corresponding to the appearance feature vector and the target trajectory, as well as the degree of conflict (or matching degree) between the second detection object corresponding to the motion feature vector and the target trajectory are judged. The appearance feature vector and the motion feature vector are integrated and compared from the aspects of vision and trajectory speed. The complementarity between different features is fully utilized, the additional noise caused by the mechanical combination of different features in traditional serial tracking is reduced, the final tracking result of the target trajectory is determined, and the robustness and accuracy of multi-target tracking are improved.

[0066] According to the multi-target tracking method provided in the embodiment of the present application, the cost matrices of the appearance feature vector and the motion feature vector are calculated respectively, and tracking matching is performed to determine the first detection object and the second detection object. When a tracking conflict occurs, the target trajectory is used as a reference, and a conflict judgment is performed based on the detection results and the tracking trajectory. The first detection object, the appearance feature vector, the second detection object, the motion feature vector and the target trajectory are comprehensively considered to determine the target detection object corresponding to the target trajectory in the current image frame. The complementarity between different features is fully utilized, the tracking noise interference is suppressed, and the robustness and accuracy of multi-target tracking are improved.

[0067] In some embodiments, determining a target detection object corresponding to the target trajectory in the current image frame based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector, and the target trajectory includes:

[0068] Determining a first collision prediction value based on the first detection object, the appearance feature vector, and the target trajectory;

[0069] determining a second collision prediction value based on the second detection object, the motion feature vector, and the target trajectory;

[0070] A target detection object is determined based on the first conflict prediction value and the second conflict prediction value.

[0071] Among them, the conflict prediction value is an indicator used to quantify the degree of matching ambiguity between the detected object and the trajectory, reflecting the competition intensity between two or more potential matching objects in the current image frame.

[0072] In this embodiment, a first conflict prediction value between the first detection object and the target trajectory is calculated through the first detection object, the appearance feature vector and the target trajectory. A second conflict prediction value between the second detection object and the target trajectory is calculated through the second detection object, the motion feature vector and the target trajectory. The competition intensity between the first detection object and the second detection object is determined based on the first conflict prediction value and the second conflict prediction value, and the target detection object is accurately determined.

[0073] In some embodiments, determining the first collision prediction value based on the first detected object, the appearance feature vector, and the target trajectory may include:

[0074] Determining a first conflict prediction value based on the confidence level of the first detection object, the confidence level of the target trajectory, and a feature similarity distance between the target trajectory and the appearance feature vector of the first detection object;

[0075] Determining a second collision prediction value based on the second detection object, the motion feature vector, and the target trajectory may include:

[0076] A second conflict prediction value is determined based on the confidence level of the second detection object, the confidence level of the target trajectory, and a feature similarity distance between the target trajectory and the motion feature vector of the second detection object.

[0077] The confidence of the detected object may refer to the confidence information corresponding to the detection box of the detected object in the current image frame, and the target detection result output by the target detector may include the confidence of each detected object in the image frame.

[0078] It should be noted that in the tracking trajectory matching process, whether a trajectory can be matched successfully is not only related to the trajectory itself, but also to the confidence of the detected target and the correlation results between the corresponding features of the detected target and the trajectory.

[0079] In this embodiment, the conflict prediction value is determined based on the confidence of the detection object, the confidence of the target trajectory, and the feature similarity distance between the target trajectory and the appearance feature vector or motion feature vector of the matched detection object. This can accurately quantify the degree of matching ambiguity between the trajectory and the detection object, which helps to improve the robustness and accuracy of multi-target tracking.

[0080] In actual implementation, the calculation formula for the first conflict prediction value is as follows:

[0081]

[0082] In formula (3), represents the first conflict prediction value, Represents the target trajectory t The confidence level, Represents the target trajectoryt Based on the confidence of the detection target (i.e., the first detection object) matched by the appearance feature vector, Represents the target trajectory t The similarity distance with the appearance feature vector of the first detected object matched, is the weighted value, which can be 0.5.

[0083] The calculation formula of the second conflict prediction value is as follows:

[0084]

[0085] In formula (4), represents the second conflict prediction value, Represents the target trajectory t The confidence level, Represents the target trajectory t Based on the confidence of the detection target (i.e., the second detection object) matched with the motion feature vector, Represents the target trajectory t The similarity distance with the motion feature vector of the second detection object matched, is the weighted value, which can be 0.5.

[0086] In some embodiments, determining the target detection object based on the first conflict prediction value and the second conflict prediction value may include:

[0087] When the difference between the first conflict prediction value and the second conflict prediction value is greater than the target difference threshold, the first detection object is used as the target detection object;

[0088] Alternatively, when the difference between the first conflict prediction value and the second conflict prediction value is less than or equal to the target difference threshold, the second detection object is used as the target detection object.

[0089] In this embodiment, the difference between the first conflict prediction value and the second conflict prediction value is obtained, and the difference between the first conflict prediction value and the second conflict prediction value is compared with a preset target difference threshold. If the difference between the first conflict prediction value and the second conflict prediction value is greater than the target difference threshold, the detection target tracked based on the appearance features is trusted, that is, the first detection object is taken as the target detection object corresponding to the target trajectory in the current image frame.

[0090] If the difference between the first conflict prediction value and the second conflict prediction value is less than or equal to the target difference threshold, the detection target tracked based on the motion feature is trusted, that is, the second detection object is taken as the target detection object corresponding to the target trajectory in the current image frame.

[0091] For example, after calculating the first conflict prediction value and the second conflict prediction value After that, the tracking conflict judgment is performed. When , the detection target tracked based on the appearance feature vector is trusted, and the first detection object is taken as the target detection object. When , the detection target tracked based on the motion feature vector is trusted, and the second detection object is taken as the target detection object.

[0092] in, It is the preset target difference threshold, which can be adjusted according to actual needs. Can be greater than or equal to 0.

[0093] In some embodiments, after determining the first detection object corresponding to the target trajectory based on the first cost matrix and determining the second detection object corresponding to the target trajectory based on the second cost matrix, the multi-target tracking method may further include:

[0094] In the case that the first detection object and the second detection object are the same object, the target detection object is determined to be the first detection object and the second detection object.

[0095] In this embodiment, the first detection object tracked based on the appearance feature vector and the second detection object tracked based on the motion feature vector are the same object. The tracking result is trusted, and the first detection object and the second detection object (the same object) are used as the target detection objects corresponding to the target trajectory in the current image frame.

[0096] For example, Figure 3 As shown, for the target trajectory j to be matched, the appearance tracking result corresponding to the target trajectory j is obtained, that is, according to the appearance feature vector, the first cost matrix is ​​calculated, and tracking matching is performed to obtain the first detection object, that is, Figure 3 The detection object A is shown.

[0097] Obtain the appearance tracking result corresponding to the tracking trajectory j, that is, calculate the second cost matrix according to the motion feature vector, perform tracking matching, and obtain the second detection object, that is, Figure 3 The detection object A' is shown.

[0098] Determine whether the detected objects A and A' are the same object. If A and A' are the same object, it means that the target trajectory j is tracked successfully. Trust the tracking result and continue to determine the next target trajectory.

[0099] If A and A' are not the same object, track j is marked as a pending track and the detection object A, detection object A', and track j are sent to the conflict resolution module for conflict resolution. The conflict prediction value is calculated to determine the target detection object of track j.

[0100] In some embodiments, after determining the target detection object corresponding to the target trajectory in the current image frame, the multi-target tracking method may further include:

[0101] When it is determined that the template update condition is met, the tracking template of the target trajectory is updated based on the target detection object.

[0102] The template update condition is a pre-set rule for determining whether to update the tracking template of the target trajectory using the information of the target detection object in the current image frame.

[0103] In this embodiment, before updating the tracking template of the target trajectory, it is first determined whether the tracking result of the current image frame meets the template update conditions. After determining that the template update conditions are met, the template is updated. This allows the template to adapt to dynamic characteristics such as target deformation and lighting changes, and avoid drift problems introduced by noise.

[0104] For example, the template update condition may be to determine whether the target detection object associated with the target trajectory in the current image frame is occluded. If occluded, the template update condition is not satisfied.

[0105] In some embodiments, determining that a template update condition is satisfied includes:

[0106] When the confidence of the target detection object is greater than the target confidence threshold and the target detection frame of the target detection object does not overlap with the detection frames of other detection objects in the current image frame, it is determined that the template update condition is met.

[0107] In this embodiment, the confidence of the target detection object can be judged first. When the confidence of the target detection object is greater than the target confidence threshold, the occlusion of the target detection object can be judged. When the target detection frame of the target detection object does not overlap with the detection frames of other detection objects in the current image frame, that is, the target detection frame of the target detection object is not occluded, it is judged that the template update condition is met.

[0108] In actual execution, when it is determined that the template update condition is met, the appearance feature template of the target trajectory can be updated using the information of the target detection object.

[0109] For example, Represents the target trajectory j In the i The appearance feature template of the frame is based on the i -1 Historical Appearance Feature Template and i The appearance detection features of the target detection object newly associated with the frame are calculated, and the calculation formula is as follows:

[0110]

[0111] in, Represents the target trajectory j No. i -1 historical appearance feature template, Indicates the preset scale factor, Indicates the i The appearance detection features (i.e., appearance feature vector) of the newly associated target detection object in the frame.

[0112] In actual implementation, the corresponding motion feature template can be updated according to the target detection frame of the target detection object in the current image frame, and the target trajectory can also be calculated. j The average confidence of the latest K frames is used as the target trajectory j The trajectory confidence.

[0113] It is understandable that when the confidence of the target detection object is less than or equal to the target confidence threshold, it indicates that the information of the target detection object cannot be used for template update, that is, it is determined that the template update condition is not met.

[0114] In some embodiments, determining that a template update condition is satisfied includes:

[0115] When the confidence of the target detection object is greater than a target confidence threshold and the target detection frame of the target detection object overlaps with the detection frames of other detection objects in the current image frame, determining a first detection frame overlapping with the target detection frame and obtaining a confidence set, where the confidence set includes the confidence of the first detection frame;

[0116] When the confidence of the target detection box is greater than the maximum value in the confidence set, it is determined that the template update condition is met.

[0117] It is understandable that the target detection frame of the target detection object overlaps with the detection frames of other detection objects in the current image frame, that is, the target detection frame of the target detection object may be at least partially obscured.

[0118] In this embodiment, the first detection frame that overlaps with the target detection frame is first screened out from the detection frames of other detection objects in the current image frame, and the confidence of each first detection frame is obtained to construct a confidence set. When the confidence of the target detection frame is greater than the maximum value in the confidence set, it is determined that the template update condition is met.

[0119] For example, A represents the target detection frame of the target detection object. The n first detection frames overlapping with the target detection frame A are screened out to form a confidence set C = [c1, c2, ..., cn]. When the confidence corresponding to A is greater than all the confidences in the confidence set C, it is judged that the template update condition is met, and the appearance feature template of the target trajectory is updated using the information of the target detection object.

[0120] In some embodiments, determining that a template update condition is satisfied includes:

[0121] When the confidence of the target detection object is greater than the target confidence threshold and the target detection frame of the target detection object overlaps with the detection frames of other detection objects in the current image frame, determine a first detection frame overlapping with the target detection frame, and obtain a confidence set and an overlap ratio set, where the confidence set includes the confidence of the first detection frame, and the overlap ratio set includes the overlap ratio between the target detection frame and the first detection frame;

[0122] Determine a second detection frame based on the confidence of the target detection frame and the confidence set, where the second detection frame is a first detection frame in the confidence set whose confidence is greater than or equal to the confidence of the target detection frame;

[0123] Determining a sum of the overlap ratios of the second detection frame based on the overlap ratio set;

[0124] When the sum of the overlap ratios is less than the target overlap ratio threshold, it is determined that the template update condition is met.

[0125] In this embodiment, the first detection frames that overlap with the target detection frame are first screened out from the detection frames of other detection objects in the current image frame, the confidence of each first detection frame is obtained, a confidence set is constructed, and the overlap ratio between each first detection frame and the target detection frame is obtained, and an overlap ratio set is constructed.

[0126] The confidence of the target detection frame is compared with the confidence in the confidence set, and the first detection frame in the confidence set whose confidence is greater than the confidence of the target detection frame is selected, that is, the second detection frame.

[0127] The overlap ratios corresponding to these second detection frames are searched in the overlap ratio set, the overlap ratios of all second detection frames are obtained, and the sum of the overlap ratios is calculated. If the sum of the overlap ratios is less than the target overlap ratio threshold, it is determined that the template update condition is met.

[0128] For example, A represents the target detection frame of the target detection object, and n first detection frames overlapping with the target detection frame A are screened out to form a confidence set C=[c1,c2,…,cn], and the overlap ratio of each first detection frame with the target detection frame A is obtained to construct an overlap ratio set R=[r1,r2,…,rn].

[0129] If there is a confidence level greater than or equal to the confidence level corresponding to A in the confidence level set C, the second detection frames are selected and the sum of the overlap ratios corresponding to these second detection frames, rtotal, is calculated using the overlap ratio set. For example, if the confidence levels of the first and second first detection frames are greater than or equal to the confidence level corresponding to A, rtotal = r1 + r2.

[0130] The sum of the overlap ratios is compared with the target overlap ratio threshold. If the sum of the overlap ratios is less than the target overlap ratio threshold, the template update condition is met, and the appearance feature template of the target trajectory is updated using the information of the target detection object.

[0131] In some embodiments, when the sum of the overlap ratio values ​​is greater than or equal to the target overlap ratio threshold, it is determined that the template update condition is not satisfied.

[0132] In this embodiment, if the sum of the overlap ratios is greater than or equal to the target overlap ratio threshold, it indicates that the target detection object is partially occluded, and the appearance feature template of the target trajectory is not updated.

[0133] It can be understood that the target overlap ratio threshold is a critical value that reflects the proportion of the occluded part of the target detection object to the overall object. For example, the target overlap ratio threshold can be set to 0.4. The sum of the overlap ratios is greater than or equal to the target overlap ratio threshold, indicating that the target detection object is largely occluded and the appearance feature template is not updated.

[0134] A specific embodiment is described below.

[0135] like Figure 2 As shown, the target image sequence is input into the target detector frame by frame to obtain the target detection result output by the target detector. The target detection result may include the detection objects in the image frame, such as detection object A1 and detection object A2.

[0136] The detection object is input into the appearance feature extraction module and the motion feature extraction module respectively for feature extraction, and the cost matrices corresponding to the appearance feature vector and the motion feature vector are calculated respectively. Based on their respective cost matrices, the Hungarian matching method is used for trajectory matching and tracking judgment.

[0137] If a tracking conflict occurs, conflict resolution is performed. Combined with the confidence of the current detection target, the confidence of the tracking trajectory, and the similarity of the appearance features and motion features between the current detection target and the trajectory, a dynamic judgment is made as to whether the current trajectory tracking result is the motion information tracking result or the appearance information tracking result. The complementarity between different features is fully utilized to suppress tracking noise interference, determine the final tracking result, and update the template based on the final tracking result.

[0138] In the template update phase, the occlusion of the matched target is first judged. When the occlusion threshold is less than the preset value (such as the sum of the overlap ratios is less than the target overlap ratio threshold), the template is updated with the current image frame to reduce the impact of occlusion on the appearance feature template and improve the robustness and accuracy of multi-target tracking.

[0139] An embodiment of the present application also provides a computer program product.

[0140] like Figure 4 As shown, the computer program product includes:

[0141] A first processing module 410 is configured to input the target image sequence into a target detector and obtain a target detection result output by the target detector, wherein the target detection result includes a detection object in each image frame in the target image sequence;

[0142] The second processing module 420 is used to obtain the appearance feature vector and motion feature vector of the detected object in the current image frame;

[0143] The third processing module 430 is used to obtain a first cost matrix between the appearance feature vector and the target trajectory to be matched, and obtain a second cost matrix between the motion feature vector and the target trajectory;

[0144] A fourth processing module 440 is configured to determine a first detection object corresponding to the target trajectory based on the first cost matrix, and to determine a second detection object corresponding to the target trajectory based on the second cost matrix;

[0145] The fifth processing module 450 is used to determine the target detection object corresponding to the target trajectory in the current image frame based on the first detection object, appearance feature vector, second detection object, motion feature vector and target trajectory when the first detection object and the second detection object are different objects.

[0146] According to the computer program product provided in the embodiment of the present application, the cost matrices of the appearance feature vector and the motion feature vector are calculated respectively, and tracking matching is performed to determine the first detection object and the second detection object. When a tracking conflict occurs, the target trajectory is used as a reference, and a conflict judgment is performed based on the detection results and the tracking trajectory. The first detection object, the appearance feature vector, the second detection object, the motion feature vector and the target trajectory are comprehensively considered to determine the target detection object corresponding to the target trajectory in the current image frame. The complementarity between different features is fully utilized, the tracking noise interference is suppressed, and the robustness and accuracy of multi-target tracking are improved.

[0147] In some embodiments, the fifth processing module 450 is configured to determine the target detection object corresponding to the target trajectory in the current image frame based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector, and the target trajectory, including:

[0148] Determining a first collision prediction value based on the first detection object, the appearance feature vector, and the target trajectory;

[0149] determining a second collision prediction value based on the second detection object, the motion feature vector, and the target trajectory;

[0150] A target detection object is determined based on the first conflict prediction value and the second conflict prediction value.

[0151] In some embodiments, the fifth processing module 450 is configured to determine a first collision prediction value based on the first detection object, the appearance feature vector, and the target trajectory, including:

[0152] Determining a first conflict prediction value based on the confidence level of the first detection object, the confidence level of the target trajectory, and a feature similarity distance between the target trajectory and the appearance feature vector of the first detection object;

[0153] Determining a second collision prediction value based on the second detection object, the motion feature vector, and the target trajectory includes:

[0154] A second conflict prediction value is determined based on the confidence level of the second detection object, the confidence level of the target trajectory, and a feature similarity distance between the target trajectory and the motion feature vector of the second detection object.

[0155] In some embodiments, the fifth processing module 450 is configured to determine the target detection object based on the first conflict prediction value and the second conflict prediction value, including:

[0156] When the difference between the first conflict prediction value and the second conflict prediction value is greater than the target difference threshold, the first detection object is used as the target detection object;

[0157] Alternatively, when the difference between the first conflict prediction value and the second conflict prediction value is less than or equal to the target difference threshold, the second detection object is used as the target detection object.

[0158] In some embodiments, after determining the first detection object corresponding to the target trajectory based on the first cost matrix and determining the second detection object corresponding to the target trajectory based on the second cost matrix, the fifth processing module 450 is also used to determine that the target detection object is the first detection object and the second detection object when the first detection object and the second detection object are the same object.

[0159] In some embodiments, after determining the target detection object corresponding to the target track in the current image frame, the fifth processing module 450 is further configured to update the tracking template of the target track based on the target detection object if it is determined that the template update condition is met.

[0160] In some embodiments, determining that a template update condition is satisfied includes:

[0161] When the confidence of the target detection object is greater than the target confidence threshold and the target detection frame of the target detection object does not overlap with the detection frames of other detection objects in the current image frame, it is determined that the target detection object meets the template update condition.

[0162] In some embodiments, determining that a template update condition is satisfied includes:

[0163] When the confidence of the target detection object is greater than a target confidence threshold and the target detection frame of the target detection object overlaps with the detection frames of other detection objects in the current image frame, determining a first detection frame overlapping with the target detection frame and obtaining a confidence set, where the confidence set includes the confidence of the first detection frame;

[0164] When the confidence of the target detection box is greater than the maximum value in the confidence set, it is determined that the target detection object meets the template update condition.

[0165] In some embodiments, determining that a template update condition is satisfied includes:

[0166] When the confidence of the target detection object is greater than the target confidence threshold and the target detection frame of the target detection object overlaps with the detection frames of other detection objects in the current image frame, determine a first detection frame overlapping with the target detection frame, and obtain a confidence set and an overlap ratio set, where the confidence set includes the confidence of the first detection frame, and the overlap ratio set includes the overlap ratio between the target detection frame and the first detection frame;

[0167] Determine a second detection frame based on the confidence of the target detection frame and the confidence set, where the second detection frame is a first detection frame in the confidence set whose confidence is greater than or equal to the confidence of the target detection frame;

[0168] Determining a sum of the overlap ratios of the second detection frame based on the overlap ratio set;

[0169] When the sum of the overlap ratios is less than the target overlap ratio threshold, it is determined that the template update condition is met.

[0170] The computer program product provided in the embodiments of the present application can implement each process implemented in the above-mentioned multi-target tracking method embodiment. To avoid repetition, they will not be described here.

[0171] like Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, the various processes of the above-mentioned multi-target tracking method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0172] An embodiment of the present application further provides a non-volatile computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned multi-target tracking method embodiments when running.

[0173] In an exemplary embodiment, the non-volatile computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0174] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned multi-target tracking method embodiments are implemented.

[0175] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned multi-target tracking method embodiments.

[0176] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] The above is a detailed introduction to a multi-target tracking method and program product provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A multi-target tracking method, characterized in that: The following steps are involved: Inputting a target image sequence into a target detector to obtain a target detection result output by the target detector, wherein the target detection result includes a detection object of each image frame in the target image sequence; Obtain the appearance feature vector and motion feature vector of the detected object in the current image frame; Obtaining a first cost matrix between the appearance feature vector and the target trajectory to be matched, and obtaining a second cost matrix between the motion feature vector and the target trajectory; Determining a first detection object corresponding to the target trajectory based on the first cost matrix, and determining a second detection object corresponding to the target trajectory based on the second cost matrix; In a case where the first detection object and the second detection object are different objects, determining a target detection object corresponding to the target trajectory in the current image frame based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector, and the target trajectory; The determining, based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector, and the target trajectory, of the target detection object corresponding to the target trajectory in the current image frame includes: determining a first collision prediction value based on the first detection object, the appearance feature vector, and the target trajectory; determining a second conflict prediction value based on the second detection object, the motion feature vector, and the target trajectory; Determining the target detection object based on the first conflict prediction value and the second conflict prediction value; The determining a first collision prediction value based on the first detection object, the appearance feature vector, and the target trajectory includes: determining the first conflict prediction value based on the confidence of the first detection object, the confidence of the target trajectory, and a feature similarity distance between the target trajectory and the appearance feature vector of the first detection object; The determining of a second conflict prediction value based on the second detection object, the motion feature vector, and the target trajectory includes: determining the second conflict prediction value based on the confidence of the second detection object, the confidence of the target trajectory, and a feature similarity distance between the target trajectory and the motion feature vector of the second detection object; The image frame includes multiple detection objects, and the first cost matrix and the second cost matrix are used to measure the degree of mismatch between the detection object and the target trajectory; the first cost matrix is ​​determined based on the feature similarity distance of the appearance feature vector, and the feature similarity distance of the appearance feature vector is determined based on the appearance feature template corresponding to the appearance feature vector of the detection object in the image frame and the target trajectory; the second cost matrix is ​​determined based on the feature similarity distance of the motion feature vector, and the feature similarity distance of the motion feature vector is determined based on the motion feature template corresponding to the motion feature vector of the detection object in the image frame and the target trajectory.

2. The multi-target tracking method according to claim 1, characterized in that: The determining the target detection object based on the first conflict prediction value and the second conflict prediction value includes: When the difference between the first conflict prediction value and the second conflict prediction value is greater than a target difference threshold, taking the first detection object as the target detection object; Alternatively, when the difference between the first conflict prediction value and the second conflict prediction value is less than or equal to the target difference threshold, the second detection object is used as the target detection object.

3. The multi-target tracking method according to claim 1, wherein: After determining a first detection object corresponding to the target trajectory based on the first cost matrix and determining a second detection object corresponding to the target trajectory based on the second cost matrix, the method further includes: In the case that the first detection object and the second detection object are the same object, the target detection object is determined to be the first detection object and the second detection object.

4. The multi-target tracking method according to any one of claims 1 to 3, characterized in that: After determining the target detection object corresponding to the target trajectory in the current image frame, the method further includes: When it is determined that the template update condition is met, the tracking template of the target trajectory is updated based on the target detection object.

5. The multi-target tracking method according to claim 4, characterized in that: The determining that the template update condition is met includes: When the confidence of the target detection object is greater than the target confidence threshold and the target detection frame of the target detection object does not overlap with the detection frames of other detection objects in the current image frame, it is determined that the target detection object meets the template update condition.

6. The multi-target tracking method according to claim 4, characterized in that: The determining that the template update condition is met includes: If the confidence of the target detection object is greater than a target confidence threshold and a target detection frame of the target detection object overlaps with a detection frame of another detection object in the current image frame, determining a first detection frame overlapping with the target detection frame, and obtaining a confidence set, where the confidence set includes the confidence of the first detection frame; In a case where the confidence of the target detection frame is greater than a maximum value in the confidence set, it is determined that the target detection object meets the template update condition.

7. The multi-target tracking method according to claim 4, characterized in that: The determining that the template update condition is met includes: When the confidence of the target detection object is greater than a target confidence threshold and a target detection frame of the target detection object overlaps with a detection frame of another detection object in the current image frame, determining a first detection frame overlapping with the target detection frame, and obtaining a confidence set and an overlap ratio set, the confidence set including the confidence of the first detection frame, and the overlap ratio set including the overlap ratio between the target detection frame and the first detection frame; Determining a second detection frame based on the confidence of the target detection frame and the confidence set, where the second detection frame is the first detection frame in the confidence set whose confidence is greater than or equal to the confidence of the target detection frame; Determining a sum of the overlap ratios of the second detection frames based on the overlap ratio set; When the sum of the overlap ratios is less than the target overlap ratio threshold, it is determined that the template update condition is met.

8. A computer program product, characterized in that include: a first processing module, configured to input a target image sequence into a target detector, and obtain a target detection result output by the target detector, wherein the target detection result includes a detection object of each image frame in the target image sequence; The second processing module is used to obtain the appearance feature vector and motion feature vector of the detected object in the current image frame; a third processing module, configured to obtain a first cost matrix between the appearance feature vector and the target trajectory to be matched, and to obtain a second cost matrix between the motion feature vector and the target trajectory; a fourth processing module, configured to determine a first detection object corresponding to the target trajectory based on the first cost matrix, and determine a second detection object corresponding to the target trajectory based on the second cost matrix; a fifth processing module, configured to, when the first detection object and the second detection object are different objects, determine, based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector, and the target trajectory, a target detection object corresponding to the target trajectory in the current image frame; The fifth processing module, configured to determine the target detection object corresponding to the target trajectory in the current image frame based on the first detection object, the appearance feature vector, the second detection object, the motion feature vector, and the target trajectory, includes: determining a first collision prediction value based on the first detection object, the appearance feature vector, and the target trajectory; determining a second conflict prediction value based on the second detection object, the motion feature vector, and the target trajectory; Determining the target detection object based on the first conflict prediction value and the second conflict prediction value; The fifth processing module, configured to determine a first collision prediction value based on the first detection object, the appearance feature vector, and the target trajectory, includes: determining the first conflict prediction value based on the confidence of the first detection object, the confidence of the target trajectory, and a feature similarity distance between the target trajectory and the appearance feature vector of the first detection object; The determining of a second conflict prediction value based on the second detection object, the motion feature vector, and the target trajectory includes: determining the second conflict prediction value based on the confidence of the second detection object, the confidence of the target trajectory, and a feature similarity distance between the target trajectory and the motion feature vector of the second detection object; The image frame includes multiple detection objects, and the first cost matrix and the second cost matrix are used to measure the degree of mismatch between the detection object and the target trajectory; the first cost matrix is ​​determined based on the feature similarity distance of the appearance feature vector, and the feature similarity distance of the appearance feature vector is determined based on the appearance feature template corresponding to the appearance feature vector of the detection object in the image frame and the target trajectory; the second cost matrix is ​​determined based on the feature similarity distance of the motion feature vector, and the feature similarity distance of the motion feature vector is determined based on the motion feature template corresponding to the motion feature vector of the detection object in the image frame and the target trajectory.

Images