An improved visual multi-target tracking method based on positioning reliability

Abstract

The present invention relates to an improved visual multi-target tracking method based on fixed position reliability, which belongs to the technical field of environmental perception and comprises the following steps: S1: detecting information of target objects at each moment; S2: predicting the position coordinates of the trajectory in the current frame by using an adaptive Kalman filter based on the trajectory tracking result of the previous frame, i.e. obtaining a trajectory prediction frame of the current frame; S3: calculating a fusion cost function of the trajectory prediction frame and the detection frame of the current frame, and achieving optimal matching of the trajectory and the detection frame based on the fusion cost function; S4: using an adaptive Kalman filter, based on the detection frame successfully matched to the trajectory, updating the trajectory tracking result of the current frame, and outputting the tracking result; S5: for the trajectory successfully matched to the detection frame, adaptively updating the appearance features of the trajectory by using a dynamic appearance update strategy.

Description

Technical Field

[0001] The invention belongs to the technical field of environmental perception and relates to an improved visual multi-target tracking method based on positioning reliability. Background Art

[0002] As an important technology in the field of computer vision, multi-object tracking (MOT) has been widely used in the fields of autonomous driving, security monitoring, and robotics. At present, the tracking-by-detection (TBD) paradigm is the mainstream paradigm of multi-object tracking, which mainly improves tracking performance by focusing on data association.

[0003] At present, there are still the following challenges in multi-target tracking technology. First, for object motion modeling, most methods use linear Kalman filtering algorithms. These methods set a uniform observation noise for all targets and do not consider the impact of the quality of the detection frame on the update state. For detection frames with poor quality, the position error of the tracking frame will increase during Kalman update. Secondly, in the data association process, since both motion and appearance information are valuable, some trackers fuse the two types of information and construct a combined cost function for matching detection frames and trajectories. It is worth noting that these methods do not adaptively determine the weight distribution of motion and appearance information for each detection frame, but simply select a fixed weighting factor through experiments to allocate the fusion weight of the two types of information, seeking to improve the overall association effect of the data set. Finally, for the successfully matched trajectory, its appearance feature state needs to be updated, and the existing appearance update strategy does not distinguish the degree of influence of the positioning accuracy and appearance clarity of the detection frame on the appearance features, and the improvement of association accuracy is limited. Summary of the invention

[0004] In view of this, the object of the present invention is to provide an improved visual multi-target tracking method based on positioning confidence, namely a new adaptive Kalman filter, an adaptive fusion cost function and a more accurate and reasonable dynamic appearance updating strategy.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] An improved visual multi-target tracking method based on positioning reliability comprises the following steps:

[0007] S1: Detect the information of the target object at each moment;

[0008] S2: Based on the trajectory tracking result of the previous frame, the adaptive Kalman filter is used to predict the position coordinates of the trajectory in the current frame, that is, to obtain the trajectory prediction box of the current frame;

[0009] S3: Calculate the fusion cost function of the trajectory prediction box and the detection box of the current frame, and achieve the optimal matching of the trajectory and the detection box based on the fusion cost function;

[0010] S4: using an adaptive Kalman filter, based on the detection frame successfully matched by the trajectory, updating the tracking result (trajectory) of the current frame, and outputting the tracking result;

[0011] S5: For the track that successfully matches the detection box, the dynamic appearance update strategy is used to adaptively update the appearance features of the track.

[0012] Further, in step S1, a 2D detector is used to detect the information of the target object at each moment, including position coordinates, detection confidence, positioning confidence and classification confidence; the target object information is represented as

[0013] 2D bounding box=(x1,y1,x2,y2,s det ,s loc ,s cls )

[0014] Where (x1, y1, x2, y2) is the coordinates of the lower left corner (x1, y1) and the upper right corner (x2, y2) of the detection box, s det is the detection confidence of the detection box, s loc is the positioning confidence of the detection box, s cls is the classification confidence of the detection box.

[0015] Further, in step S2, based on the trajectory of the previous frame, an adaptive Kalman filter is used to predict the position coordinates (x1, y1, x2, y2) of the trajectory in the current frame, where (x1, y1, x2, y2) are the coordinates of the lower left corner and the upper right corner of the trajectory prediction box.

[0016] Further, step S3 specifically includes the following steps:

[0017] S31: Calculate the motion information-based cost function of the trajectory prediction box and the detection box of the current frame, that is, the Intersection over Union (IoU). The specific calculation formula is as follows:

[0018]

[0019] Among them, d i represents the i-th detection box of the current frame, t j represents the jth track of the current frame;

[0020] S32: Calculate the cost function based on appearance information of the trajectory and the detection frame of the current frame, that is, the cosine distance between the trajectory and the appearance feature vector of the detection frame. The specific calculation formula is as follows:

[0021]

[0022] in, Represents the appearance feature vector of the i-th detection box in the current frame, Represents the appearance feature vector of the jth track in the current frame;

[0023] S33: Based on the positioning confidence and classification confidence of the detection frame, the fusion cost function of the current frame trajectory and the detection frame is calculated. The calculation formula is as follows:

[0024] C af (i,j)=s loc ×C iou (i,j)+s det ×C cos (i,j)

[0025] Among them, s loc represents the positioning confidence of the i-th detection box, s det Represents the detection confidence of the i-th detection box;

[0026] The positioning confidence of the detection frame can reflect the positioning accuracy of the detection frame to a certain extent. At the same time, the detection confidence can reflect the appearance clarity of the detection frame to a certain extent. Therefore, the positioning confidence and the detection confidence are multiplied by the cost functions based on motion and appearance information respectively to obtain the fusion cost function.

[0027] S34: Based on cost function C af , the Hungarian algorithm is used to optimally match the trajectory and the detection box;

[0028] S35: Set the matching threshold of the fusion cost matrix to thresh. If the C between the i-th detection box and the j-th track af The value of (i, j) is less than the threshold thresh, then the track and the detection box are matched successfully; if C af If the value of (i,j) is greater than the threshold thresh, the track and the detection box fail to match.

[0029] Further, step S4 specifically includes the following steps:

[0030] S41: For the trajectory that successfully matches the detection frame, the observation noise covariance R is adaptively adjusted according to the number of disappeared frames of the trajectory and the detection confidence of the detection frame. k , the specific calculation formula is as follows:

[0031]

[0032] Among them, R k is the preset observation noise covariance, s det is the detection confidence of the detection box matched by the trajectory, and σ is the threshold score for dividing the detection confidence of the detection box; for lose, the specific calculation formula is as follows:

[0033]

[0034] Among them, lost frame is the number of frames that the track has disappeared continuously. thresh is the maximum frame number threshold for trajectory retention;

[0035] S42: Using R k Calculate the Kalman gain, perform Kalman update, obtain the final position coordinates of the trajectory, and output it as the tracking result.

[0036] Further, in step S5, based on the positioning confidence and classification confidence of the detection frame matched by the trajectory in the current frame, the influence of the positioning accuracy and appearance clarity on the appearance features is considered respectively, and an adaptive weighting factor is determined. The appearance features of the matched detection frame are added to the appearance feature state of the trajectory, and the appearance feature state of the trajectory is updated. The calculation formula is as follows:

[0037] e t =α PDA e t-1 +(1-α PDA )f t

[0038] Among them, e t-1 is the appearance feature state of the trajectory before updating; f t is the appearance feature of the detection box that the trajectory successfully matches in the current frame; for the adaptive weight factor α PDA , the calculation formula is as follows:

[0039]

[0040] Among them, α f is a fixed threshold; s cls is the classification confidence of the detection box matched by the trajectory; s loc is the positioning confidence of the detection frame matched by the trajectory; σ det The threshold score for classifying the detection box with high or low confidence level; σ loc The threshold score for dividing the detection box into high and low position confidence.

[0041] The beneficial effects of the present invention are as follows: the present invention improves the data association accuracy of the MOT method based on the TBD paradigm, provides more accurate environmental perception for unmanned driving technology, and lays a foundation for path planning and control decisions.

[0042] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0044] Figure 1 The flowchart of the improved visual multi-target tracking method based on positioning reliability of the present invention is as follows;

[0045] Figure 2 It is a comparison chart of tracking visualization results, where (a) is the visualization result of the LG-Track algorithm in the MOT17-05 scenario, and (b) is the visualization result of the method of the present invention in the MOT17-05 scenario. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0047] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0048] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0049] like Figure 1 As shown, the present invention provides an improved visual multi-target tracking method based on positioning reliability, and the method specifically comprises the following steps:

[0050] S1: Use the 2D detector to obtain the information of the target object at each moment, including position coordinates, detection confidence, location confidence and classification confidence; the object information obtained by the 2D detector is calculated using 2D boundingbox = (x1, y1, x2, y2, s det ,s loc ,s cls ), where (x1, y1, x2, y2) are the coordinates of the lower left and upper right corners of the detection box, and s det is the detection confidence of the detection box, s loc is the positioning confidence of the detection box, s cls is the classification confidence of the detection box.

[0051] S2: Based on the tracking result (trajectory) of the previous frame, the adaptive Kalman filter is used to predict the position coordinates of the trajectory in the current frame, that is, the trajectory prediction frame of the current frame is obtained; specifically, based on the trajectory of the previous frame, the adaptive Kalman filter is used to predict the position coordinates of the trajectory in the current frame (x1, y1, x2, y2), (x1, y1, x2, y2) are the coordinates of the lower left corner and the upper right corner of the trajectory prediction frame, and the specific formula of Kalman prediction is as follows:

[0052] x′ k =Ax k-1 +Bu k

[0053] P′ k =AP K-1 A T +Q

[0054] Among them, x′ k represents the predicted state of the trajectory of the current frame (the kth frame), x k-1is the state of the previous frame (k-1 frame) of the trajectory, A is the state transfer matrix, u k is the state control vector, B is the control variable matrix; P′ k is the state estimation covariance matrix of the current frame, P K-1 is the state covariance matrix of the previous frame, and Q is the system noise covariance matrix.

[0055] S3: Calculate the fusion cost function of the trajectory prediction box and the detection box of the current frame, and achieve the optimal matching of the trajectory and the detection box based on the fusion cost function; specifically, the following steps are included:

[0056] S31: Calculate the motion information-based cost function of the trajectory prediction box and the detection box of the current frame, that is, the Intersection over Union (IoU). The specific calculation formula is as follows:

[0057]

[0058] Among them, d i represents the i-th detection box of the current frame, t j Represents the jth track of the current frame.

[0059] S32: Calculate the cost function based on appearance information of the trajectory and the detection frame of the current frame, that is, the cosine distance between the trajectory and the appearance feature vector of the detection frame. The specific calculation formula is as follows:

[0060]

[0061] in, Represents the appearance feature vector of the i-th detection box in the current frame, Represents the appearance feature vector of the jth track in the current frame.

[0062] S33: Based on the positioning confidence and classification confidence of the detection frame, the fusion cost function of the current frame trajectory and the detection frame is calculated. The specific calculation formula is as follows:

[0063] C af (i,j)=s loc ×C iou (i,j)+s det ×C cos (i,j)

[0064] Among them, s loc represents the positioning confidence of the i-th detection box, s det Represents the detection confidence of the i-th detection box.

[0065] The positioning confidence of the detection frame can reflect the positioning accuracy of the detection frame to a certain extent. At the same time, the detection confidence can reflect the appearance clarity of the detection frame to a certain extent. Therefore, the positioning confidence and the detection confidence are multiplied by the cost functions based on motion and appearance information respectively to obtain the fusion cost function.

[0066] S34: Based on cost function C af , the Hungarian algorithm is used to optimally match the trajectory and the detection box.

[0067] Furthermore, the process of the Hungarian algorithm is as follows:

[0068] S341: The cost function C af The cost matrix is ​​formed by subtracting the minimum value of each row from the row:

[0069] S342: Subtract the minimum value of each column from each column of the obtained new matrix;

[0070] S343: Use the minimum number of rows and columns to cover all zero elements in the new matrix.

[0071] If the line cannot cover all zero elements, then go to S344, otherwise go to S345;

[0072] S344: Find the minimum value among the elements that cannot be connected by row and column lines, and subtract the minimum value from the remaining elements.

[0073] Small value, add the minimum value to the element at the intersection of the row and column lines;

[0074] S345: Start matching from the row or column with the least zero elements until all rows and categories are matched, and then obtain the optimal matching result.

[0075] S35: Set the matching threshold of the fusion cost matrix to thresh. If the C between the i-th detection box and the j-th track af The value of (i, j) is less than the threshold thresh, then the track and the detection box are matched successfully; if C af If the value of (i,j) is greater than the threshold thresh, the track and the detection box fail to match.

[0076] S4: using an adaptive Kalman filter, based on the detection frame successfully matched by the trajectory, updating the tracking result (trajectory) of the current frame, and outputting the tracking result; specifically including the following steps:

[0077] S41: For the trajectory that successfully matches the detection frame, the observation noise covariance R is adaptively adjusted according to the number of disappeared frames of the trajectory and the detection confidence of the detection frame. k , the specific calculation formula is as follows:

[0078]

[0079] Among them, R k is the preset observation noise covariance, s det is the detection confidence of the detection box matched by the trajectory, σ is the threshold score for dividing the detection confidence of the detection box, and unless otherwise specified, 0.6 is generally taken as the threshold. For lost, the specific calculation formula is as follows:

[0080]

[0081] Among them, Lost frame is the number of frames that the track has disappeared continuously. thresh It is the maximum frame number threshold of the trajectory. Unless otherwise specified, it is usually 30 frames.

[0082] S42: The adaptively adjusted observation noise covariance R obtained using the above formula k , R k Substitute the formula, calculate the Kalman gain, perform Kalman update, and get the final state of the current frame trajectory. The specific calculation formula is as follows:

[0083] K=HP′ k H T (HP′ k H T +R k ) -1

[0084] x k = x′ k +K(Z k -Hx′ k )

[0085] P k =(1-KH)P′ k

[0086] Among them, K is the Kalman gain, x k is the trajectory state of the current frame, P k is the state covariance matrix of the current frame.

[0087] S5: For the trajectory that successfully matches the detection frame, the dynamic appearance update strategy is used to adaptively update the appearance features of the trajectory. That is, based on the positioning confidence and classification confidence of the detection frame matched by the trajectory in the current frame, the influence of its positioning accuracy and appearance clarity on the appearance features is considered respectively, and the adaptive weighting factor is determined. The appearance features of the matched detection frame are added to the appearance feature state of the trajectory, and the appearance feature state of the trajectory is updated. The specific calculation formula is as follows:

[0088] e t =α PDA e t-1 +(1-α PDA )f t

[0089] Among them, e t-1 is the appearance feature state of the trajectory before updating; f t is the appearance feature of the detection box that the trajectory successfully matches in the current frame. For the adaptive weight factor α PDA , the specific calculation formula is as follows:

[0090]

[0091] Among them, α f is a fixed threshold value, taking 0.95; s cls is the classification confidence of the detection box matched by the trajectory; s loc is the positioning confidence of the detection frame matched by the trajectory; σ det The threshold score for classifying the detection box with high or low confidence level; σ loc The threshold score for dividing the detection box into high and low position confidence.

[0092] like Figure 2 As shown in Figure 2, the proposed improved method is compared with the advanced tracker LG-Track. For the MOT17-05 scene, it can be seen that for the two pedestrians with IDs 2 and 56, LG-Track fails to track them in the 310th frame, and the two pedestrians have switched identities, while the proposed improved tracking method always tracks robustly without identity switching, and the IDs are always 2 and 54.

[0093] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, the steps of the method can be implemented. The storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An improved visual multi-target tracking method based on positioning reliability, characterized by: The following steps are involved: S1: Detect the information of the target object at each moment; In step S1, a 2D detector is used to detect the information of the target object at each moment, including position coordinates, detection confidence, positioning confidence and classification confidence; the target object information is represented as 2D bounding box=(x1,y1,x2,y2,s det ,s loc ,s cls ) Where (x1, y1, x2, y2) is the coordinates of the lower left corner (x1, y1) and the upper right corner (x2, y2) of the detection box, s det is the detection confidence of the detection box, s loc is the positioning confidence of the detection box, s cls is the classification confidence of the detection box; S2: Based on the trajectory tracking result of the previous frame, the adaptive Kalman filter is used to predict the position coordinates of the trajectory in the current frame, that is, to obtain the trajectory prediction box of the current frame; S3: Calculate the fusion cost function of the trajectory prediction frame and the detection frame of the current frame, and achieve the optimal matching of the trajectory and the detection frame based on the fusion cost function; step S3 specifically includes the following steps: S31: Calculate the motion information-based cost function of the trajectory prediction box and the detection box of the current frame, that is, the Intersection over Union (IoU). The specific calculation formula is as follows: Among them, d i represents the i-th detection box of the current frame, t j represents the jth track of the current frame; S32: Calculate the cost function based on appearance information of the trajectory and the detection frame of the current frame, that is, the cosine distance between the trajectory and the appearance feature vector of the detection frame. The specific calculation formula is as follows: in, Represents the appearance feature vector of the i-th detection box in the current frame, Represents the appearance feature vector of the jth track in the current frame; S33: Based on the positioning confidence and detection confidence of the detection frame, the fusion cost function of the current frame trajectory and the detection frame is calculated. The calculation formula is as follows: C af (i,j)=s loc ×C iou (i,j)+s det ×C cos (i,j) Among them, s loc represents the positioning confidence of the i-th detection box, s det Represents the detection confidence of the i-th detection box; S34: Based on cost function C af , the Hungarian algorithm is used to optimally match the trajectory and the detection box; S35: Set the matching threshold of the fusion cost matrix to thresh. If the C between the i-th detection box and the j-th track af The value of (i, j) is less than the threshold thresh, then the track and the detection box are matched successfully; if C af If the value of (i,j) is greater than the threshold thresh, the track and the detection box fail to match; S4: using an adaptive Kalman filter, based on the detection frame successfully matched by the trajectory, updating the trajectory tracking result of the current frame, and outputting the tracking result; S5: For the trajectory successfully matched to the detection frame, the appearance features of the trajectory are adaptively updated using the dynamic appearance update strategy; in step S5, based on the positioning confidence and classification confidence of the detection frame matched by the trajectory in the current frame, the influence of the positioning accuracy and appearance clarity on the appearance features is considered respectively, and the adaptive weighting factor is determined, and the appearance features of the matched detection frame are added to the appearance feature state of the trajectory, and the appearance feature state of the trajectory is updated. The calculation formula is as follows: e t =a PDA e t-1 +(1-a PDA )f t Among them, e t-1 is the appearance feature state of the trajectory before updating; f t is the appearance feature of the detection box that the trajectory successfully matches in the current frame; for the adaptive weight factor α PDA , the calculation formula is as follows: Among them, α f is a fixed threshold; s cls is the classification confidence of the detection box matched by the trajectory; s loc is the positioning confidence of the detection frame matched by the trajectory; σ cls The threshold score for classifying the detection box with high or low confidence level; σ loc It is the threshold value for dividing the positioning confidence of the detection frame.

2. The improved visual multi-target tracking method based on positioning reliability according to claim 1, characterized in that: In step S2, based on the trajectory of the previous frame, an adaptive Kalman filter is used to predict the position coordinates (x1, y1, x2, y2) of the trajectory in the current frame, where (x1, y1, x2, y2) are the coordinates of the lower left corner and the upper right corner of the trajectory prediction box.

3. The improved visual multi-target tracking method based on positioning reliability according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41: For the trajectory that successfully matches the detection frame, the observation noise covariance R is adaptively adjusted according to the number of disappeared frames of the trajectory and the detection confidence of the detection frame. k , the specific calculation formula is as follows: Among them, R k is the preset observation noise covariance, s det is the detection confidence of the detection box matched by the trajectory, and σ is the threshold score for dividing the detection confidence of the detection box; for lose, the specific calculation formula is as follows: Among them, lost frame is the number of frames that the track has disappeared continuously. thresh is the maximum frame number threshold for trajectory retention; S42: Using R k Calculate the Kalman gain, perform Kalman update, obtain the final position coordinates of the trajectory, and output it as the tracking result.

Images