Two-stage Multi-object Tracking Data Association Method and System Based on Discriminative Features

Abstract

The present invention relates to the field of multi-object tracking, which is one of the intelligent transportation and autonomous driving support technologies, and discloses a two-stage multi-object tracking data association method and system based on discriminative features. The method includes: Step 1: Obtaining discriminative features of detected vehicles; Step 2: One-stage data association based on discriminative features of detected vehicles; Step 3: Obtaining discriminative features of trajectory segments; Step 4: Two-stage data association based on discriminative features of trajectory segments. The present invention aims at low-quality vehicles caused by occlusion, forms discriminative detected vehicle features with topological properties through the feature fusion of the current detected vehicle and other detected vehicles, and combines a two-stage data association scheme to realize the continuous and effective association of interrupted driving trajectories caused by disappearing vehicles.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-object tracking, which is one of the intelligent transportation and autonomous driving support technologies, and particularly relates to a two-stage multi-object tracking data association method and system based on discriminative features. Background Technique

[0002] Multi-object tracking is one of the current hot technologies in computer vision and has been widely studied and used in practical applications such as video surveillance and driverless driving. Thanks to the rapid development of object detection technology, the detect-then-track scheme has become one of the mainstream paradigms for multi-object tracking, that is, first using a detector to independently generate a set of bounding boxes for each video frame, and then associating these detected vehicles with the trajectories uniquely identified by the object identification numbers according to the motion pattern or visual cues to form the running trajectories of each detected vehicle appearing in the video.

[0003] Among them, the tracking effect after detection depends mostly on the performance of the data association algorithm and the performance of obtaining motion patterns and visual cues. When facing a vehicle-dense scene, the frequent mutual occlusion between vehicles will cause the missing of the detected vehicle regions or the disappearance of the detected vehicles, thereby causing unreliable phenomena of the detected vehicles, interfering with the effectiveness of feature extraction based on the detected vehicles and the accuracy of data association relying on the extracted features. Therefore, the present invention proposes a two-stage data association scheme based on discriminative features to increase the effectiveness of the low-quality detected vehicle features caused by occlusion through the discriminative representation of the detected vehicle features; on this basis, combining the two-stage data association scheme to achieve the continuous and effective association of the interrupted driving trajectories caused by the disappeared vehicles. Summary of the Invention

[0004] The present invention proposes a two-stage multi-object tracking data association method and system based on discriminative features to increase the effectiveness of the low-quality detected vehicle features caused by occlusion through the discriminative representation of the detected vehicle features; on this basis, combining the two extreme data association schemes to achieve the continuous and effective association of the interrupted driving trajectories caused by the disappearance of vehicles due to occlusion.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] Aiming at the influence of vehicle mutual occlusion on multi-object tracking performance in a dense scene, the present invention proposes a two-stage multi-object tracking data association method based on discriminative features. For the low-quality detected vehicles caused by occlusion, by fusing the features of the current detected vehicle and other detected vehicles to form a discriminative feature of the detected vehicle with topological properties, and combining the two-stage data association scheme, for the disappeared vehicles caused by occlusion, to complete the effective association between the trajectory segment and the detected vehicle and between the trajectory segments, including:

[0007] Step 1: Form discriminative features of the detected vehicle with topological properties by fusing the features of the current detected vehicle and other detected vehicles.

[0008] Step 2: Perform one-stage data association based on the discriminative features of the detected vehicle to obtain a set of initial trajectory segments.

[0009] Step 3: Obtain discriminative features of the initial trajectory segments based on the set of initial trajectory segments.

[0010] Step 4: Perform two-stage data association based on the discriminative features of the initial trajectory segments to obtain the final multi-vehicle motion trajectories.

[0011] Furthermore, Step 1 includes:

[0012] Different from the method of forming discriminative features of the detected vehicle only by using appearance, position, etc., the present invention is based on the detected vehicle (specifically, the YoLox detector can be used for detection), and uses its appearance (specifically, it can be extracted through a residual network) and position information as initial features, that is, o i =(φ i , b i ), o i is the current detected vehicle, φ i is the obtained appearance feature, b i =(x i , y i , w i , h i ) is the position information calibrated by the tracking box;

[0013] Then, as shown in formula (1), by aggregating the feature information of other detected vehicles in the same video frame, the intra-frame topological relationship is considered to enhance the discriminability of the current detected vehicle features to cope with the occlusion problem:

[0014]

[0015] Among them, W V is the transformation matrix; w ij is the attention weight, which is obtained by simultaneously considering the appearance similarity and the relative position of other detected vehicles and the current detected vehicle o i .

[0016] Furthermore, Step 2 includes:

[0017] Based on the discriminative features of the detected vehicle, this part first performs data association between the detected vehicles to form initial trajectory segments. Among them, the present invention divides the trajectory segments into active and inactive trajectory segments; then, given the last frame of the detected vehicle of the active trajectory segment l j feature f l last ​The detected vehicle of the current video frame Feature It is planned to calculate the similarity between the two according to formula (2):

[0018]

[0019] In formula (2), D(·) represents the Euclidean distance. Based on the similarity, data association is established according to the threshold γ, that is: when the similarity is greater than the threshold γ, the detected vehicle is added to the current trajectory segment l j to form a new active trajectory. Furthermore, in the present invention, after the active trajectory segment has not been updated for consecutive β frames, it is converted into an inactive trajectory segment to reduce the interference of low-quality or disappearing vehicles caused by occlusion on the formation of the trajectory segment. After the data association between detected vehicles, the present invention calibrates the unassigned detected vehicles as new trajectory segments and outputs the initial trajectory segment set

[0020]

[0021] Furthermore, step 3 includes:

[0022] For the initial trajectory segments generated based on the data association of detected vehicles, in order to better realize the continuous tracking of low-quality or disappearing vehicles caused by occlusion, the present invention further extracts the discriminative features of the initial trajectory segments. Among them, in order to avoid the influence of the degradation problem of low-quality vehicles caused by occlusion on the extraction of the discriminative features of the trajectory segments, a feature transfer mechanism is designed to effectively aggregate the discriminative features of the detected vehicles included in the trajectory segments. That is, for the trajectory segment l j , it is planned to select the video frames in l j whose detected vehicle areas arranged in time sequence are greater than the threshold T ft as the frame set of the trajectory segment l j , and then use the feature transfer mechanism shown in formula (3) to output the discriminative features of the trajectory segment:

[0023]

[0024] In the formula, Q p-1 is the query matrix of the initial trajectory segment p-1, and f p-1 is the aggregated feature to the p-1 frame formed by using the feature transfer mechanism; K p , V p respectively represent the key matrix and value matrix generated based on the discriminative features of the detected vehicle p; is the time-sequence correlation attention weight, which is used to adjust V p to obtain the value matrix in the current detected vehicle that has a strong time-sequence correlation with f p-1 ; φ(·) is the fusion function.

[0025] Further, step 4 includes:

[0026] The present invention designs a data re - association mechanism for generating initial trajectory segments to further achieve continuous tracking of low - quality or disappearing vehicles caused by occlusion:

[0027] 1) Arrange the initial trajectory segments in chronological order and number them;

[0028] 2) Define all trajectory segments starting after the end of trajectory segment l i as the backward - associated trajectories of l i and establish a re - association model between l i and the backward trajectories;

[0029] 3) According to the feature similarity S(l i , l k ) between associated trajectory segments, select the l i with the highest similarity from the associated trajectory segments where S(l k ) > T ts as the backward trajectory segment of l j and merge them. Here, T i is the threshold; then, taking the trajectory feature of l ts as the pre - aggregation feature and the feature of l i as the current feature, form the aggregation feature V j of the merged trajectory segment l i-j according to formula (3); j ;

[0030] 4) Take the merged trajectory segment as the forward - trajectory segment node and repeat steps 2) and 3);

[0031] 5) Repeat step 4) until the similarity between associated trajectory segments is lower than T ts to terminate the continuous association of merged trajectory segments;

[0032] 6) Take the new trajectory segment as the forward node and repeat the above steps until the re - association of all trajectory segments is completed. Among them, if the trajectory segment l k is an associated trajectory segment of the forward node l i , but l i and l j are unassociated trajectories, then l k cannot be used as the backward trajectory segment of l j ;

[0033] 7) Take the result of the re - association of trajectory segments as the final multi - vehicle motion trajectory.

[0034] On the other hand, the present invention proposes a two-stage multi-object tracking data association system based on discriminative features, including:

[0035] A detection vehicle discriminative feature acquisition module, configured to form a detection vehicle discriminative feature with topological properties by fusing the features of the current detected vehicle and other detected vehicles;

[0036] A first-stage data association module, configured to perform first-stage data association based on the discriminative features of the detected vehicles to obtain an initial trajectory segment set;

[0037] A trajectory segment discriminative feature acquisition module, configured to obtain initial trajectory segment discriminative features based on the initial trajectory segment set;

[0038] A second-stage data association module, configured to perform second-stage data association based on the initial trajectory segment discriminative features to obtain the final multi-vehicle motion trajectories.

[0039] Further, the detection vehicle discriminative feature acquisition module is specifically configured to:

[0040] Use the YoLox detector to output each detected vehicle in each video frame;

[0041] Use a residual network to extract the appearance feature φ of each detected vehicle i , combined with the current detection box coordinates b i =(x i ,y i ,w i ,h i ) to form the current detected vehicle feature o i =(φ i ,b i );

[0042] According to formula (1), aggregate the feature information of other detected vehicles in the same video frame to consider the in-frame topological relationship:

[0043]

[0044] Among them, W V is the transformation matrix; w ij is the attention weight, obtained by simultaneously considering the appearance similarity and the relative position of other detected vehicles and the current detected vehicle o i .

[0045] Further, the first-stage data association module is specifically configured to:

[0046] Divide the trajectory segments into active and inactive trajectory segments, and set all the detected vehicles in the first video frame as active trajectory segments;

[0047] Subsequently, according to the active trajectory segment lj The detected vehicle in the last frame Feature f l last Complete the feature comparison with all detected vehicles in the current video frame, that is, calculate the similarity between the two according to formula (2):

[0048]

[0049] Among them, represents the i-th detected vehicle in the current video frame, and f t i represents its feature, represents and the similarity of; D(·) represents the Euclidean distance;

[0050] Based on the similarity, establish data association according to the threshold γ. When the similarity is greater than the threshold γ, add to the current track segment l j to form a new active track; if the active track segment has not been updated for consecutive β frames, it will be converted into an inactive track segment to reduce the interference of low-quality or disappearing vehicles caused by occlusion on the formation of the track segment; after the first-stage data association, the unassigned detected vehicles are calibrated as new track segments, and the initial track segment set is output

[0051]

[0052] Furthermore, the discriminative feature acquisition module of the track segment is specifically used for:

[0053] For the generated initial track segment set Output the discriminative features of the initial track segments by using the feature transfer mechanism shown in formula (3):

[0054]

[0055] Among them, Q p-1 is the query matrix of the initial track segment p-1, and f p-1 is the aggregated feature to the p-1 frame formed by using the feature transfer mechanism; K p , V p respectively represent the key matrix and value matrix generated based on the discriminative features of the detected vehicle p; is the temporal correlation attention weight, which is used to adjust V p to obtain the value matrix with strong temporal correlation with f p-1 in the current detected vehicle; φ(·) is the fusion function.

[0056] Furthermore, the second-stage data association module is specifically used for:

[0057] 1) Arrange the initial trajectory segments in chronological order and number them;

[0058] 2) Define all the trajectory segments starting after the end of trajectory segment l i as the backward associated trajectories of l i and establish the re - association model between l i and the backward trajectories;

[0059] 3) According to the feature similarity S(l i , l k ) between associated trajectory segments, select the l i with the highest similarity from the associated trajectory segments where S(l k ) > T ts as the backward trajectory segment of l j and merge them, where T i is the threshold; then, taking the trajectory feature of l ts as the pre - aggregation feature and the feature of l i as the current feature, form the aggregation feature V j of the merged trajectory segment l i-j according to formula (3); j ;

[0060] 4) Take the merged trajectory segment as the forward trajectory segment node and repeat steps 2) and 3);

[0061] 5) Repeat step 4) until the similarity between associated trajectory segments is lower than T ts to terminate the continuous association of merged trajectory segments;

[0062] 6) Take the new trajectory segment as the forward node and repeat the above steps until the re - association of all trajectory segments is completed. Among them, if the trajectory segment l k is the associated trajectory segment of the forward node l i , but l i and l j are non - associated trajectories, then l k cannot be used as the backward trajectory segment of l j ;

[0063] 7) Take the result of the re - association of trajectory segments as the final multi - vehicle motion trajectory.

[0064] Compared with the prior art, the beneficial effects of the present invention are:

[0065] The present invention aims at the low - quality vehicles caused by occlusion, and forms discriminative detection vehicle features with topological properties through the feature fusion of the current detected vehicle and other detected vehicles. Combining with the two - stage data association scheme, it realizes the continuous and effective association of the interrupted driving trajectories caused by disappearing vehicles.

[0066] The present invention discloses a two-stage multi-object tracking data association method and system based on discriminative features. The present invention forms discriminative features of detected vehicles with topological properties through feature fusion of the current detected vehicle and other detected vehicles; performs one-stage data association based on the discriminative features of the detected vehicles according to the detected vehicles and features; for the generated set of initial trajectory segments, obtains discriminative features of the trajectory segments; and performs two-stage data association based on the discriminative features of the trajectory segments based on the discriminative features of the initial trajectory segments to form multi-vehicle driving trajectories in an occlusion environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a basic flowchart of a two-stage multi-object tracking data association method based on discriminative features according to an embodiment of the present invention;

[0068] Figure 2 is a schematic diagram of obtaining discriminative features according to an embodiment of the present invention;

[0069] Figure 3 is a two-stage data association scheme diagram based on discriminative features of trajectory segments according to an embodiment of the present invention;

[0070] Figure 4 is an example of a vehicle trajectory diagram generated by a two-stage data association scheme based on discriminative features of trajectory segments according to an embodiment of the present invention;

[0071] Figure 5 is a schematic diagram of the architecture of a two-stage multi-object tracking data association system based on discriminative features according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] The following further explains and illustrates the present invention with reference to the accompanying drawings and specific embodiments:

[0073] As Figure 1 shown, in view of the influence of vehicle occlusion on multi-object tracking performance in a dense scene, the present invention proposes a two-stage multi-object tracking data association method based on discriminative features. For low-quality vehicles caused by occlusion, discriminative features of detected vehicles with topological properties are formed through feature fusion of the current detected vehicle and other detected vehicles. Combining with a two-stage data association scheme, for disappeared vehicles caused by occlusion, effective associations between trajectory segments and detected vehicles and between trajectory segments and trajectory segments are completed, including:

[0074] Step S101: Obtaining discriminative features of detected vehicles;

[0075] Step S102: One-stage data association based on discriminative features of detected vehicles;

[0076] Step S103: Obtaining discriminative features of trajectory segments;

[0077] Step S104: Two-stage data association based on discriminative features of trajectory segments.

[0078] Furthermore, in the step S101, the specific steps for obtaining the discriminative features of the detected vehicle are as follows:

[0079] Step S101.1, use the YoLox detector to output the detected vehicles in each video frame respectively;

[0080] Step S101.2, use the residual network to extract the appearance feature φ of the detected vehicle in each frame i , combined with the coordinates b of the square frame of the current detected vehicle i =(x i , y i , w i , h i ) to form the feature o of the current detected vehicle i =(φ i , b i );

[0081] Step S101.3, then, according to formula (1), aggregate the feature information of other detected vehicles in the same video frame to consider the intra-frame topological relationship;

[0082]

[0083] Furthermore, in the step S102, the specific steps for one-stage data association based on the discriminative features of the detected vehicle are as follows:

[0084] Step S102.1, according to the detected vehicle and its features, the present invention takes the lead in implementing a one-stage data association scheme based on the discriminative features of the detected vehicle. In this part, the present invention divides the trajectory segments into active and inactive trajectory segments. And all the detected vehicles in the first video frame are set as active trajectory segments.

[0085] Step S102.2, subsequently, according to the active trajectory segment l j the detected vehicle in the last frame feature f l last complete the feature comparison with all the detected vehicles in the current video frame, that is, calculate the similarity between the two according to formula (2).

[0086]

[0087] Among them, represents the i-th detected vehicle in the current video frame, f t i represents its feature, represents and Similarity; D(·) represents the Euclidean distance. Based on the similarity, data association is established according to the threshold γ, that is: when the similarity is greater than the threshold γ, the detected vehicle is added to the current trajectory segment l j to form a new active trajectory. Furthermore, in the present invention, after the active trajectory segment has not been updated for consecutive β frames, it is converted into an inactive trajectory segment to reduce the interference of low-quality or disappearing vehicles caused by occlusion on the formation of the trajectory segment. After the first-stage data association, the present invention calibrates the unassigned detected vehicles as new trajectory segments and outputs the initial trajectory segment set

[0088]

[0089] Furthermore, as Figure 2 shown, in the step S103, the specific steps for obtaining the discriminative features of the trajectory segment are as follows:

[0090]

[0091] where Q p-1 is the query matrix of the initial trajectory segment p-1, and f p-1 is the aggregated feature to the p-1 frame formed by using the feature transfer mechanism; K p , V p respectively represent the key matrix and the value matrix generated based on the discriminative features of the detected vehicle p; is the temporal correlation attention weight, which is used to adjust V p to obtain the value matrix in the current detected vehicle that has a strong temporal correlation with f p-1 ; φ(·) is a fusion function.

[0092] Furthermore, in the step S104, the specific steps for the first-stage data association based on the discriminative features of the trajectory segment are as follows:

[0093] Based on the discriminative features of the initial trajectory segment, the present invention executes a two-stage data association scheme based on the discriminative features of the trajectory segment, and further realizes the continuous tracking of low-quality or disappearing vehicles caused by occlusion by using the scheme as Figure 3 shown:

[0094] 1) Arrange the initial trajectory segments in chronological order and number them, as shown in Figure 3 A;

[0095] 2) Define all the trajectory segments that start after the end of the trajectory segment l i as the backward associated trajectories of l i , and establish a re-association model between l i and the backward trajectories, as shown in Figure 3 B.

[0096] 3) According to the feature similarity S(l i , l k ) between associated trajectory segments, select the l with the highest similarity from the associated trajectory segments where S(l i , l k ) > T ts as the backward trajectory segment of l j and merge them; then, taking the trajectory feature of l i as the pre-aggregation feature and the feature of l i as the current feature, form the aggregation feature V j of the merged trajectory segment l i-j according to formula (3). Taking the example shown by the red frame in B of j : l2 is the most matching trajectory segment of l1, so merge the trajectory segments l1 and l2 and generate the merged trajectory segment feature V2; Figure 3

[0097] 4) Take the merged trajectory segment as the forward trajectory segment node, as shown by the purple frame in B of Figure 3 , and repeat steps 2) and 3);

[0098] 5) Repeat step 4) until the similarity between associated trajectory segments is lower than T ts to terminate the continuous association of merged trajectory segments;

[0099] 6) Take the new trajectory segment as the forward node and repeat the above steps until the re-association of all trajectory segments is completed. Among them, if the trajectory segment l k is an associated trajectory segment of the forward node l i , but l i and l j are non-associated trajectories, then l k cannot be used as the backward trajectory segment of l j ; as shown by the blue frame in B of Figure 3 : In the re-associated trajectory model starting from ③, ⑤ is an associated trajectory segment of ③, while ④ has no association with ③; therefore, in the re-associated model starting from ④, although ⑤ is a backward trajectory segment of ④, it is not an associated trajectory segment of ④;

[0100] 7) As shown in C of Figure 3 and D of Figure 3 , take the result after re-associating the trajectory segments as the final multi-vehicle motion trajectory, and a trajectory example is shown in Figure 4 .

[0101] On the basis of the above embodiments, as shown in Figure 5 , the present invention also proposes a two-stage multi-object tracking data association system based on discriminative features, including:

[0102] ​The detection vehicle discriminative feature acquisition module is used to form the detection vehicle discriminative feature with topological properties through the feature fusion of the current detection vehicle and other detection vehicles;

[0103] The first-stage data association module is used to perform first-stage data association based on the detection vehicle discriminative feature to obtain the initial trajectory segment set;

[0104] The trajectory segment discriminative feature acquisition module is used to obtain the initial trajectory segment discriminative feature based on the initial trajectory segment set;

[0105] The second-stage data association module is used to perform second-stage data association based on the initial trajectory segment discriminative feature to obtain the final multi-vehicle motion trajectory.

[0106] Furthermore, the detection vehicle discriminative feature acquisition module is specifically used for:

[0107] Using the YoLox detector to output the detection vehicles in each video frame respectively;

[0108] Using the residual network to extract the appearance feature φ of the detection vehicle in each frame i , combined with the square frame coordinates b of the current detection vehicle i =(x i ,y i ,w i ,h i ) to form the current detection vehicle feature o i =(φ i ,b i );

[0109] According to formula (1), aggregate the feature information of other detection vehicles in the same video frame to consider the in-frame topological relationship:

[0110]

[0111] where, W V is the transformation matrix; w ij is the attention weight, which is obtained by simultaneously considering the appearance similarity and the relative position of other detection vehicles and the current detection vehicle o i .

[0112] Furthermore, the first-stage data association module is specifically used for:

[0113] Divide the trajectory segments into active and inactive trajectory segments, and set all the detection vehicles in the first video frame as active trajectory segments;

[0114] Subsequently, according to the active trajectory segment l j The detection vehicle in the last frame Feature f llast Complete the feature comparison with all detected vehicles in the current video frame, that is, calculate the similarity between the two according to formula (2):

[0115]

[0116] Among them, represents the i-th detected vehicle in the current video frame, and f t i represents its feature, represents and similarity; D(·) represents the Euclidean distance;

[0117] Based on the similarity, establish data association according to the threshold γ. When the similarity is greater than the threshold γ, add to the current trajectory segment l j to form a new active trajectory; if the active trajectory segment has not been updated for consecutive β frames, it will be converted into an inactive trajectory segment to reduce the interference of low-quality or disappearing vehicles caused by occlusion on the formation of the trajectory segment; after the first-stage data association, the unassigned detected vehicles are calibrated as new trajectory segments, and the initial trajectory segment set is output

[0118]

[0119] Furthermore, the trajectory segment discriminative feature acquisition module is specifically used for:

[0120]

[0121] Among them, Q p-1 is the query matrix of the initial trajectory segment p-1, and f p-1 is the aggregated feature to the p-1 frame formed by using the feature transfer mechanism; K p , V p respectively represent the key matrix and value matrix generated based on the discriminative features of the detected vehicle p; is the temporal correlation attention weight, which is used to adjust V p to obtain the value matrix in the current detected vehicle that has a strong temporal correlation with f p-1 ; φ(·) is the fusion function.

[0122] Furthermore, the second-stage data association module is specifically used for:

[0123] 1) Arrange the initial trajectory segments in chronological order and number them;

[0124] 2) Define all the trajectory segments that start after the end of the trajectory segment l i as the backward association trajectories of l i and establish l iRe - association model with backward trajectories;

[0125] 3) According to the feature similarity S(l i ,l k ) between associated trajectory segments, select the l i ,l k ) with the highest similarity from the associated trajectory segments where S(l ts >T j as the backward trajectory segment of l i , and perform merging, where T ts is the threshold; then, taking the trajectory feature of l i as the pre - aggregation feature, the feature of l j as the current feature, form the aggregation feature V i-j of the merged trajectory segment l j according to formula (3);

[0126] 4) Take the merged trajectory segment as the forward trajectory segment node, and repeat steps 2) and 3);

[0127] 5) Repeat step 4) until the similarity between associated trajectory segments is lower than T ts to terminate the continuous association of merged trajectory segments;

[0128] 6) Take the new trajectory segment as the forward node, and repeat the above steps until the re - association of all trajectory segments is completed. Among them, if the trajectory segment l k is the associated trajectory segment of the forward node l i , but l i and l j are unassociated trajectories, then l k cannot be used as the backward trajectory segment of l j ;

[0129] 7) Take the result of the re - association of trajectory segments as the final multi - vehicle motion trajectory.

[0130] In summary, the present invention first forms a discriminative feature of the detected vehicle with topological properties through the feature fusion of the current detected vehicle and other detected vehicles; secondly, performs one - stage data association based on the discriminative feature of the detected vehicle according to the detected vehicle and the feature; thirdly, obtains the discriminative feature of the trajectory segment for the generated initial trajectory segment set; finally, performs two - stage data association based on the discriminative feature of the initial trajectory segment to increase the effectiveness of the low - quality vehicle features caused by occlusion, and forms a multi - vehicle driving trajectory.

[0131] The above - shown is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A two-stage multi-object tracking data association method based on discriminative features, characterized in that, Including: Step 1: Form discriminative features of the detected vehicles with topological properties through the feature fusion of the current detected vehicles and other detected vehicles; The said Step 1 includes: Step 101: Use the YoLox detector to output the detected vehicles in each video frame respectively; Step 102: Use the residual network to extract the appearance feature φ of each detected vehicle i , and combine it with the coordinates b i =(x i , y i , w i , h i ) of the current detected vehicle's square frame to form its feature o i =(φ i , b i ); Step 103: According to formula (1), aggregate the feature information of other detected vehicles in the same video frame to consider the in-frame topological relationship: Among them, W V is the transformation matrix; w ij is the attention weight, which is obtained by simultaneously considering the appearance similarity and the relative position of other detected vehicles and the current detected vehicle o i ; Step 2: Perform first-stage data association based on the discriminative features of the detected vehicles to obtain an initial set of trajectory segments; The said Step 2 includes: Step 201: Divide the trajectory segments into active and inactive trajectory segments, and set all the detected vehicles in the first video frame as active trajectory segments; Step 202. Subsequently, according to the active trajectory segment l j the detected vehicle of the last frame feature complete the feature comparison with all the detected vehicles in the current video frame, that is, calculate the similarity between the two according to formula (2): Among them, represents the i-th detected vehicle in the current video frame, f t i represents its features, represents the similarity with ; D(·) represents the Euclidean distance; Based on the similarity, data association is established according to the threshold γ. When the similarity is greater than the threshold γ, is added to the current trajectory segment l j to form a new active trajectory; if the active trajectory segment has not been updated for consecutive β frames, it is converted into an inactive trajectory segment to reduce the interference of low-quality or disappearing vehicles caused by occlusion on the formation of the trajectory segment; after the first-stage data association, the unassigned detected vehicles are calibrated as new trajectory segments, and the initial trajectory segment set is output Step 3: Obtain discriminative features of the initial trajectory segments based on the initial set of trajectory segments; Step 4: Perform second-stage data association based on the discriminative features of the initial trajectory segments to obtain the final multi-vehicle motion trajectories; The said Step 4 includes: Step 401, arrange the initial trajectory segments in chronological order and number them; Step 402: Define all the trajectory segments starting after the end of the trajectory segment l i as the backward associated trajectories of l i and establish the re - association model between l i and the backward trajectories; Step 403: according to the associated trajectory segment l i ,l k The feature similarity S(l i ,l k ), from S(l i ,l k )>T ts Select the highest similarity l from the associated trajectory segments j As l i The backward trajectory segments are merged, where T ts As the threshold value; then l i The trajectory feature is the pre-aggregation feature, l j The feature is the current feature, forming a merged trajectory segment l i-j The aggregate feature V j ; Step 404: Take the merged trajectory segments as forward trajectory segment nodes, and repeat Step 402 and Step 403; Step 405: Repeat Step 404 until the similarity between associated trajectory segments is lower than T ts to terminate the continued association of merged trajectory segments; Step 406: Take the new trajectory segment as the forward node, and repeat the above steps until the reassociation of all trajectory segments is completed; wherein, if the trajectory segment l k is the associated trajectory segment of the forward node l i , but l i and l j are unassociated trajectories, then l k cannot be used as the backward trajectory segment of l j . Step 407: Take the result after re-associating the trajectory segments as the final multi-vehicle motion trajectories.

2. The two-stage multi-object tracking data association method based on discriminative features according to claim 1, characterized in that, The said Step 3 includes: For the generated set of initial trajectory segments Output discriminative features of the initial trajectory segments by using the feature transmission mechanism as shown in formula (3): Among them, Q p-1 is the query matrix of the initial trajectory segment p-1, and f p-1 is the aggregated feature to the p-1 frame formed by using the feature transfer mechanism; K p , V p respectively represent the key matrix and the value matrix generated based on the discriminative features of the detected vehicle p; is the temporal correlation attention weight, which is used to adjust V p to obtain the value matrix in the current detected vehicle that has a strong temporal correlation with f p-1 ; φ(·) is the fusion function.

3. A two-stage multi-object tracking data association system based on discriminative features, characterized in that, Including: A discriminative feature acquisition module for detected vehicles, which is used to form discriminative features of the detected vehicles with topological properties through the feature fusion of the current detected vehicles and other detected vehicles; The discriminative feature acquisition module for the detected vehicles is specifically used for: Step 101: Use the YoLox detector to output the detected vehicles in each video frame respectively; Step 102, use the residual network to extract the appearance feature φ of each frame of the detected vehicle i , and combine it with the coordinates b i = (x i , y i , w i , h i ) of the current detected vehicle to form the feature o i = (φ i , b i ); Step 103: According to formula (1), aggregate the feature information of other detected vehicles in the same video frame to consider the in-frame topological relationship: Among them, W V is the transformation matrix; w ij is the attention weight, which is obtained by simultaneously considering the appearance similarity and the relative position of other detected vehicles and the current detected vehicle o i ; A first-stage data association module, which is used to perform first-stage data association based on the discriminative features of the detected vehicles to obtain an initial set of trajectory segments; The first-stage data association module is specifically used for: Step 201: Divide the trajectory segments into active and inactive trajectory segments, and set all the detected vehicles in the first video frame as active trajectory segments; Step 202, subsequently, according to the active trajectory segment l j The last frame of the detected vehicle Feature f l last Complete the feature comparison with all the detected vehicles in the current video frame, that is, calculate the similarity between the two according to formula (2): Among them, represents the i-th detected vehicle in the current video frame, f t i represents its features, represents the similarity with ; D(·) represents the Euclidean distance; Based on the similarity, data association is established according to the threshold γ. When the similarity is greater than the threshold γ, is added to the current trajectory segment l j to form a new active trajectory; if the active trajectory segment has not been updated for β consecutive frames, it is converted into an inactive trajectory segment to reduce the interference caused by low-quality or disappearing vehicles due to occlusion to the formation of the trajectory segment; after the first-stage data association, the unassigned detected vehicles are calibrated as new trajectory segments, and the initial trajectory segment set is output A discriminative feature acquisition module for trajectory segments, which is used to obtain discriminative features of the initial trajectory segments based on the initial set of trajectory segments; A second-stage data association module, which is used to perform second-stage data association based on the discriminative features of the initial trajectory segments to obtain the final multi-vehicle motion trajectories; The second-stage data association module is specifically used for: Step 401, arrange the initial trajectory segments in chronological order and number them; Step 402, define all the trajectory segments starting after the trajectory segment l i as the backward associated trajectories of l i , and establish a re-association model between l i and the backward trajectories; Step 403, according to the feature similarity S(l i , l k ) between the associated trajectory segments l i , l k ), select the l i , l k with the highest similarity from the associated trajectory segments where S(l ts ) > T j as the backward trajectory segment of l i and merge them, where T ts is the threshold; then, using the trajectory feature of l i as the pre-aggregation feature and the feature of l j as the current feature, form the aggregation feature V i-j of the merged trajectory segment l j ; Step 404: Take the merged trajectory segments as forward trajectory segment nodes, and repeat Step 402 and Step 403; Step 405: Repeat Step 404 until the similarity between associated trajectory segments is lower than T ts to terminate the continued association of merged trajectory segments; Step 406: Take the new trajectory segment as the forward node, and repeat the above steps until the reassociation of all trajectory segments is completed. Among them, if trajectory segment l k is the associated trajectory segment of forward node l i , but l i and l j are unassociated trajectories, then l k cannot be used as the backward trajectory segment of l j . Step 407: Take the result after re-associating the trajectory segments as the final multi-vehicle motion trajectories.

4. The discriminative feature-based two-stage multi-object tracking data association system according to claim 3, characterized in that, The discriminative feature acquisition module for the trajectory segments is specifically used for: For the generated set of initial trajectory segments Output discriminative features of the initial trajectory segments by using the feature transmission mechanism as shown in formula (3): Among them, Q p-1 is the query matrix of the initial trajectory segment p-1, and f p-1 is the aggregated feature to the p-1 frame formed by using the feature transfer mechanism; K p , V p respectively represent the key matrix and value matrix generated based on the discriminative features of the detected vehicle p; is the temporal correlation attention weight, which is used to adjust V p to obtain the value matrix with strong temporal correlation between the current detected vehicle and f p-1 ; φ(·) is the fusion function.

Images