Road user trajectory extraction and behavior labeling method integrating detection and tracking functions

Abstract

The invention provides a road user track extraction and behavior labeling method integrating detection and tracking functions, and belongs to the technical field of automatic driving and intelligent traffic. The technical problems of discontinuous road user track recognition and low behavior marking efficiency in dense heterogeneous traffic scenes are solved. According to the technical scheme, multi-scale target detection is achieved through an improved YOLOv5 network architecture, continuous track tracking is completed in combination with a space-time correlation algorithm, two-dimensional index data is constructed to achieve frame-by-frame behavior labeling, and a detection-tracking-labeling integrated processing flow is formed. The method has the beneficial effects that three functional modules of target detection, trajectory tracking and behavior marking are deeply fused, and through an improved deep learning algorithm architecture, the technical problems of accurate identification and continuous tracking of road user trajectories in dense heterogeneous complex traffic scenes are effectively solved; and meanwhile, rapid, continuous and accurate marking of traffic behaviors is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving and intelligent transportation, and in particular to a road user trajectory extraction and behavior labeling method combining detection and tracking functions. BACKGROUND

[0002] With the iterative evolution of automatic driving and intelligent transportation systems, trajectory extraction and behavior labeling of road users (covering subjects such as motor vehicles, non-motor vehicles, and pedestrians) have become a key technical path for building high-quality training datasets. However, the effectiveness of this technical system is highly dependent on the completeness and accuracy of the data source, especially in dense and heterogeneous complex traffic scenes (such as urban road intersections, mixed motor and non-motor vehicle sections, and commercial street blocks). Due to the characteristics of heterogeneous road user types and dynamically nonlinearly coupled behaviors, the construction of large-scale high-precision datasets faces technical bottlenecks such as high acquisition costs and strict labeling accuracy requirements.

[0003] The current mainstream labeling method has the following shortcomings or deficiencies:

[0004] (1) In terms of trajectory extraction, existing labeling methods often have missed detection and false detection phenomena, and have insufficient recognition ability for small targets, and it is difficult to ensure the consistency of trajectories between consecutive frames, resulting in serious missing frame problems in the extraction results;

[0005] (2) Existing labeling methods are mainly based on sparse and homogeneous simple traffic scenes, with less data collection and labeling workload. However, most of China is a dense and heterogeneous complex traffic scene, with dense interaction of heterogeneous road users and complex and variable behavior patterns, making the data collection and labeling workload of traditional methods in this scene grow exponentially.

[0006] (3) In terms of behavior labeling, existing methods are inefficient, mainly in the following aspects: lack of correlation in cross-frame labeling of the same target; multiple targets need to be repeatedly labeled in a single frame image; and labeling results are inconsistent before and after, etc.

[0007] (4) The current dataset production process relies too much on manual operation, which not only consumes time and effort, but also makes it difficult to guarantee the labeling quality, seriously restricting the large-scale production of datasets. SUMMARY

[0008] The application aims to provide a road user trajectory extraction and behavior labeling method integrating detection and tracking functions, which deeply integrates three function modules of target detection, trajectory tracking and behavior labeling, effectively solves the problems of accurate identification and continuous tracking of road user trajectories in dense, heterogeneous and complex traffic scenes through an improved deep learning algorithm architecture, and realizes rapid, continuous and accurate labeling of traffic behaviors.

[0009] In order to achieve the above-mentioned application purposes, the technical scheme of the application is as follows: a road user trajectory extraction and behavior labeling method integrating detection and tracking functions, comprising the following steps:

[0010] S1, collecting road video image sequences in dense, heterogeneous and complex traffic environments;

[0011] S2, performing multi-class detection on each road user in the video image sequence frame based on an improved deep learning convolutional neural network (CNN), and outputting target detection boxes with confidence and class information;

[0012] S3, fusing the target detection boxes and video timestamp information through a space-time correlation algorithm, assigning a unique ID to each road user, and generating an initial trajectory set strictly arranged in time sequence according to the timestamp;

[0013] S4, performing data cleaning and preprocessing operations on the initial trajectory set to realize noise elimination and missing compensation, and correcting cross-frame continuity, specifically including:

[0014] adopting Kalman filtering algorithm to eliminate trajectory noise;

[0015] compensating for missing frame trajectory data through linear interpolation;

[0016] S5, constructing a double-dimension index trajectory database, wherein:

[0017] the primary index is a time sequence structure strictly sorted according to the timestamp from early to late;

[0018] the secondary index is a sorted storage with road user ID label as key value;

[0019] S6, scanning the complete trajectory of each road user in the database along the time axis in the forward direction, and realizing the behavior labeling from the starting frame to the ending frame on the continuous frame sequence of a single road user trajectory according to the preset traffic behavior classification system.

[0020] S7, exporting trajectory and behavior data results, and ending the current process.

[0021] Further, in step S1, the roadbed camera device is used to capture video image sequences at a fixed frequency f sampl The video image sequence is captured.

[0022] Further, the frequency f sampl = 25 Hz.

[0023] Step S2 further includes:

[0024] The video image sequence frame is imported into the modified YOLOv5 target detection network, the features are extracted through the Darknet-53 backbone network, and the regression analysis is performed after upsampling and feature fusion to obtain the road user target category and detection frame.

[0025] Further, the modified YOLOv5 target detection network includes:

[0026] The CoTAttention attention mechanism module is added in the backbone network to fully utilize the context information between the input keys to guide the learning of the dynamic attention matrix, thereby enhancing the ability of visual representation;

[0027] The feature pyramid structure is optimized, the GFPN weighted bidirectional feature pyramid network is adopted, the multi-scale feature fusion is performed, and the detection ability for small targets is enhanced.

[0028] The spatio-temporal correlation algorithm of step S3 specifically includes:

[0029] The Hungarian algorithm is used to solve the matching problem of the detection frame and the existing track;

[0030] For the newly appearing target that fails to match, the incremental ID is dynamically allocated;

[0031] The ID allocation error is corrected through motion consistency test.

[0032] The data cleaning and preprocessing operation of step S4 specifically includes:

[0033] The threshold value for track noise elimination is dynamically adjusted according to the target motion speed and scene density;

[0034] The missing frame compensation is interpolated by using the linear interpolation method according to the existing frames before and after.

[0035] The construction of the double-dimension index database of step S5 meets:

[0036] The timestamp accuracy of the primary index reaches the single frame level;

[0037] The secondary index supports fast retrieval of complete tracks according to ID;

[0038] The storage structure simultaneously stores original detection data and cleaned data.

[0039] The traffic behavior classification system in step S6 includes:

[0040] The traffic behavior is predefined as a four-class system of aggressive, reckless, safe, and conservative;

[0041] Different categories of road user types have differentiated definitions of the four-class traffic behavior.

[0042] The implementation of the behavior labeling in step S6 is:

[0043] When labeling the behavior of each frame of detection result, the behavior state of the previous frame is inherited, and the initial default behavior is safe;

[0044] Different colors are used to correspond to different traffic behavior labeling labels;

[0045] The labeled information is automatically saved at intervals of 5 frames.

[0046] Compared with the prior art, the beneficial effects of the present application are:

[0047] 1. The detection accuracy of the trajectory extraction is significantly improved: through the improved deep learning network architecture and multi-scale feature fusion technology, the target detection accuracy of the present application in complex traffic scenes is improved by more than 7% compared with traditional methods. The original YOLOv5 has an mAP@0.5 of 82.3% in complex traffic scenes, and the present method has an mAP@0.5 of 89.4%. In particular, the recognition ability for small size targets <50px is significantly enhanced. The original YOLOv5s small target detection rate is 65.5%, and the present method is 70.3%.

[0048] 2. Trajectory continuity optimization: the spatio-temporal correlation algorithm of the present application fuses data cleaning and preprocessing, effectively solving the common technical problems of trajectory breakage in traditional methods. Through frame interpolation of the extracted trajectory, the missing data is filled by using linear interpolation method, ensuring the integrity and reliability of the trajectory data.

[0049] 3. The scene adaptability and behavior labeling efficiency are greatly improved: the present method is specially optimized for the unique dense and heterogeneous complex traffic scene, and can effectively handle the mixed scene of dense interaction of motor vehicles, non-motor vehicles and pedestrians. In particular, the behavior labeling process adopts single-target continuous uninterrupted labeling, which reduces labeling errors from the source and greatly improves the labeling speed. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation of the present application.

[0051] Figure 1The application provides a road user trajectory extraction and behavior continuous labeling method fusing detection and tracking functions;

[0052] Figure 2 A flow chart of multi-target detection, recognition and tracking of dense heterogeneous complex road users in the application;

[0053] Figure 3 A main network architecture diagram of the improved YOLOv5 in the application;

[0054] Figure 4 For Figure 3 An improved YOLOv5 cross-layer connection and cross-scale connection schematic diagram in the application;

[0055] Figure 5 For A dense heterogeneous complex traffic environment road user trajectory extraction and rapid continuous labeling method integrating detection, tracking and labeling in the application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and examples. Of course, the specific examples described herein are only used to explain the application and not to limit the application.

[0057] Example 1

[0058] Referring to Figure 1 And Figure 5 The technical scheme of the embodiment is a road user trajectory extraction and behavior continuous labeling method fusing detection and tracking functions, comprising the following steps:

[0059] S1, collecting road video image sequences in a dense heterogeneous complex traffic environment;

[0060] S2, performing multi-class detection on each road user in the video image sequence frame based on an improved deep learning convolutional neural network (CNN), and outputting a target detection frame with confidence and class information;

[0061] S3, fusing the target detection frame and video timestamp information through a space-time correlation algorithm, assigning a unique ID to each road user, and generating an initial trajectory set strictly time-sequenced according to the timestamp;

[0062] S4, performing data cleaning and preprocessing operations on the above initial trajectory set to realize noise elimination and missing compensation, and performing cross-frame continuity correction, specifically including:

[0063] Using Kalman filtering algorithm to eliminate trajectory noise;

[0064] Compensating for missing frame trajectory data through linear interpolation;

[0065] S5, constructing a double-dimension indexed trajectory database, wherein:

[0066] the primary index is a time series structure strictly ordered by timestamps from start to end;

[0067] the secondary index is an ordered storage with road user ID labels as key values;

[0068] S6, scanning the complete trajectories of road users in the database along the time axis in a forward direction, and sequentially implementing the frame-by-frame behavior labeling from the start frame to the end frame on the continuous frame sequence of a single road user trajectory according to a preset traffic behavior classification system.

[0069] S7, exporting the trajectory and behavior data results, and ending the current process.

[0070] Referring to Figure 1 , the embodiment provides a road user trajectory extraction and behavior continuous labeling method fusing detection and tracking functions, comprising the following steps:

[0071] 1) acquiring video image sequences at a fixed frequency (such as f sampl = 25 Hz) through a roadbed camera device;

[0072] 2) performing multi-class detection on each road user in the video image sequence frames based on an improved deep learning convolutional neural network CNN, and outputting target detection boxes and class information with confidence to step 3) as input;

[0073] 3) fusing the target detection boxes and video timestamp information through a space-time correlation algorithm, assigning a unique ID label to each road user, and outputting an initial trajectory set arranged in strict time series according to timestamps to step 4) as input;

[0074] 4) performing data cleaning and preprocessing operations, eliminating trajectory noise by using a Kalman filter algorithm, and compensating for missing frame trajectory data by linear interpolation, and outputting noise-free and non-missing frame trajectories as input for step 5);

[0075] 5) constructing a double-dimension indexed trajectory database, wherein the primary index is ordered by timestamps from start to end, and the secondary index is ordered by road user ID labels as key values; the ordered trajectory data is transferred to step 6) as input;

[0076] 6) Scan the complete trajectories of each road user in the trajectory database forward along the time axis, and sequentially implement the per-frame behavior labeling from the start frame to the end frame on the continuous frame sequence of the single road user trajectory according to the preset traffic behavior classification system. The preset traffic behavior classification system includes “aggressive”, “reckless”, “safe”, and “conservative”. The characteristics of aggressive behavior are that the vehicle or pedestrian actively compresses the time and space interval with other road users, runs at a speed obviously higher than the average speed of the environment, and preferentially obtains the right of way through abrupt longitudinal acceleration or deceleration or lateral insertion action, and significantly reduces the safety redundancy. The characteristics of reckless behavior are that the vehicle or pedestrian selects the direction or path in a manner that violates the traffic regulations or signal control provisions, suddenly intrudes into an area that does not belong to its right of way, causes irreconcilable conflicts with the expected trajectories of other road users, and transfers the potential collision risk to the other party. The characteristics of safe behavior are that the vehicle or pedestrian always follows the traffic regulations and the right of way signal, and the speed, route, and acceleration and deceleration amplitude are kept within the expected range of the traffic flow consensus, and clear external signals are sent in advance to ensure the cooperation and predictability with other road users. The characteristics of conservative behavior are that the vehicle or pedestrian travels at a speed obviously lower than the acceptable speed of the environment, maintains a safety interval far exceeding the conventional demand, and delays the start or continues to decelerate for a long time even if there is sufficient right of way, and significantly sacrifices the traffic efficiency to ensure zero risk. Table 1 is the specific behavior classification characteristics.

[0077] Table 1 Road user behavior classification characteristics

[0078]

[0079] 7) Derive the trajectory and behavior data results, and end the current process.

[0080] Referring to Figure 2 , the present embodiment realizes the specific implementation of steps 2 and 3 above through the target detection and recognition module and the multi-target tracking module, i.e., multi-class detection of each road user in the video image sequence frame based on the improved YOLOv5, feature extraction through the Darknet-53 backbone network, upsampling and feature fusion, and then regression analysis to obtain the road user target class and detection frame; secondly, the spatiotemporal correlation algorithm DeepSORT fuses the target detection frame and the video timestamp information, assigns a unique ID identifier to each road user, and outputs the trajectory data of the target class, detection, and tracking results strictly time-sequenced according to the timestamp.

[0081] Figure 3The improvement point of YOLOv5 in step 2 is illustrated, and the CoTAttention self-attention mechanism module is added in the Darknet-53 backbone network, so that the context information between the input keys is fully utilized to guide the learning of the dynamic attention matrix, thereby enhancing the ability of visual representation. The module first encodes the input keys through 3x3 convolution to obtain the static context representation of the input. Then, the encoded keys are combined with the input query, and a dynamic multi-head attention matrix is learned through two consecutive 1x1 convolutions. The learned attention matrix is multiplied by the input value to realize the dynamic context representation of the input. Finally, the fusion of the static and dynamic context representations is taken as the output. Figure 4 The GFPN is a generalized feature pyramid network, and the original YOLOv5 uses PANet for feature fusion. The GFPN solves the problems of insufficient feature fusion between adjacent layers and loss of weak features. The GFPN fuses features from multiple low layers through cross-layer connection, fuses features of the same layer and adjacent layers above and below through cross-scale connection, enhances multi-scale information flow, and improves the recognition ability of small size targets. Figure 5 It is one embodiment of the present application.

[0082] Embodiment 2

[0083] Based on embodiment 1, the DAIR-V2X dataset subset is selected for testing, and the present application shows significant technical advantages. Based on the improved YOLOv5 architecture (fusing GFPN feature pyramid and CoTAttention mechanism), the target detection mAP@0.5 is improved from 68.7% of the benchmark to 73.4% at a real-time processing speed of 38.3ms, and the small target (<50px) detection accuracy is improved to 60.8%; in the behavior labeling link, 10 hours of video data only need 13.6 hours to complete the behavior labeling, and the behavior labeling label consistency rate is 95.7%.

[0084] Embodiment 3

[0085] Based on embodiment 2, the CISTR-AI dataset subset is selected for testing, and based on the improved YOLOv5 architecture (fusing GFPN feature pyramid and CoTAttention mechanism), the target detection mAP@0.5 is improved from 65.2% of the benchmark to 68.3% at a real-time processing speed of 41.6ms; in the behavior labeling link, 10 hours of video data only need 12.2 hours to complete the behavior labeling, and the behavior labeling label consistency rate is 93.7%.

[0086] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of road user trajectory extraction and behavior annotation that fuses detection and tracking functions, characterized in that, The method comprises the following steps: S1, collecting road video image sequences in a dense heterogeneous complex traffic environment; S2, performing multi-class detection on each road user in the video image sequence frames based on an improved deep learning convolutional neural network (CNN), and outputting target detection boxes with confidence and class information; S3, fusing the target detection boxes and video timestamp information by a space-time correlation algorithm, assigning a unique ID to each road user, and generating an initial trajectory set arranged in time sequence according to the timestamp; S4, performing data cleaning and preprocessing operations on the initial trajectory set to achieve noise elimination and missing compensation, and correcting cross-frame continuity, including: eliminating trajectory noise by using Kalman filtering algorithm; compensating for missing frame trajectory data by linear interpolation; S5, constructing a double-dimension index trajectory database, wherein: the main index is a time sequence structure strictly sorted according to the timestamp from the start to the end; the secondary index is a sorted storage with road user ID label as the key value; S6, scanning the complete trajectory of each road user in the database along the time axis, and sequentially performing frame-by-frame behavior labeling from the start frame to the end frame on the continuous frame sequence of a single road user trajectory according to a preset traffic behavior classification system; S7, exporting the trajectory and behavior data results, and ending the current process. 2.The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 1, wherein, In the step S1, the roadbed is photographed by the photographing device at a fixed frequency f sampl The acquisition of the video image sequence is performed. 3.The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 2, characterized in that, The frequency f sampl = 25 Hz. 4.The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 1, wherein, The step S2 comprises: importing the video image sequence frames into a modified YOLOv5 target detection network, extracting features through a Darknet-53 backbone network, performing upsampling and feature fusion, and then performing regression analysis to obtain the road user target category and detection box.

5. The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 4, characterized in that, The modified YOLOv5 target detection network comprises: adding a CoTAttention attention mechanism module in the backbone network to fully utilize the context information between the input keys to guide the learning of the dynamic attention matrix; optimizing the feature pyramid structure by using a GFPN weighted bidirectional feature pyramid network for multi-scale feature fusion. 6.The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 1, wherein, In the step S3, the space-time correlation algorithm comprises the following steps: S31, predicting the trajectory position by Kalman filtering: state prediction: covariance prediction: where: is the predicted target state vector; F k is the state transition matrix; x k-1 is the optimal state estimate at the previous time; is the predicted estimate covariance matrix; P k-1 is the covariance at the previous time; is the transpose of the state transition matrix; Q k is the process noise covariance matrix; S32, using the Hungarian algorithm to solve the matching problem of the detection box and the existing trajectory: matching cost matrix calculation: wherein: C ij is the matching cost; D i is the ith detection box; T j is the jth predicted box; IoU(D i , T j ) is the degree of overlap calculated between the detection box D i and the predicted box T j ; ||v i -v j ||2 is the geometric distance calculated between two velocity vectors v i and v j ; S33, dynamically assigning an incremental ID to a new target that fails to match successfully: where: ID new is a new target ID number; MAX ID is the maximum ID value currently allocated; B k is the 4 coordinate parameters (x, y, w, h) of the detection frame; p k is a prime number sequence (2, 3, 5, 7). 7.The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 1, wherein, In the step S4, the data cleaning and preprocessing operation comprises the following steps: S41, the threshold for trajectory noise elimination is dynamically adjusted according to the target motion speed and scene density: adaptive threshold: where: σ dynamic is the adaptive threshold; σ base is the base noise threshold; v is the target instantaneous speed; v max is the maximum allowed speed for the scene; p is the local target density; S42, the missing frame compensation uses linear interpolation method to interpolate the missing frame: For any frame missing in the interval t e (t a , t b ), two-dimensional planar coordinate interpolation: 8.The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 1, wherein, In the step S5, the construction of the double-dimension index database comprises the following steps: S51, the timestamp precision of the main index reaches the single frame level; S52, the secondary index supports fast retrieval of complete trajectories according to ID; S53, the storage structure simultaneously saves the original detection data and the cleaned data. 9.The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 1, wherein, The traffic behavior classification system of the step S6 comprises: S61, the traffic behavior is predefined as a four-class system of aggressive, reckless, safe and conservative; S62, different categories of road user types have differential definitions for the four classifications of traffic behavior.

10. The road user trajectory extraction and behavior labeling method fusing detection and tracking functions according to claim 1, characterized in that, The implementation of the behavior labeling of the step S6 includes the following steps: S6-1, when labeling the behavior of each frame detection result, the behavior state of the previous frame is inherited, and the initial default behavior is safe; S6-2, different colors correspond to different traffic behavior labels; S6-3, save the labeled information automatically every 5 frames.

Images