A vehicle trajectory extraction method based on the complementarity of lidar and video technologies

Abstract

This invention provides a vehicle trajectory extraction method based on the complementarity of LiDAR and video technologies, comprising: constructing an integrated radar device; identifying key targets based on video acquisition data collected by a high-definition video device, and constructing a basic image information database through image optimization and data filtering; converting radar data into readable road and vehicle information based on the LiDAR acquisition device, and constructing an object and behavior database for target identification; fusing the object and behavior database and the basic image information database, and constructing a driving behavior detection dataset to achieve road element calibration, vehicle element calibration, and behavior parameter calibration; identifying and tracking the calibrated objects, thereby outputting a trajectory database. This invention can provide a data foundation for road traffic operation and risk identification, and provide a scientific basis for traffic management departments to evaluate and manage road traffic.

Description

Technical Field

[0001] This invention relates to the field of road traffic detection, and in particular to a vehicle trajectory extraction method based on the complementarity of lidar and video technologies. Background Technology

[0002] The collection of road traffic data has always been fundamental to research on road traffic operations and safety. Previously, data collection methods primarily relied on simulations or natural driving data. However, due to limitations in the collection scope and the lack of data accuracy, it was impossible to obtain high-precision, multi-dimensional, and continuous all-time, all-element data. With the advancements in radar, image analysis, and sensor technologies, it has become possible to obtain more accurate, real-time, and comprehensive road traffic data. Currently, radar and video data each have their own focus in data collection. By fusing radar and video data, providing object encoding for radar and precise distance and location data for video, the accuracy and continuity of road traffic data collection can be effectively improved. This allows for the construction of a real-time, continuous, all-time, all-element road traffic research database, providing a solid data foundation and technical support for research on road traffic operations and safety.

[0003] Currently, my country still faces numerous challenges in collecting road traffic data using radar or other methods. These challenges include: data preprocessing often relies on cloud-based calculators, hindering real-time assessment and decision-making regarding road traffic conditions, thus delaying response times; relying solely on radar for road traffic condition identification can result in unclear identification objects and an inability to obtain crucial heterogeneous parameters such as license plates; relying solely on video data collection is affected by weather conditions, preventing accurate object capture and identification; and the fusion of radar and video data lacks a fixed and advanced algorithmic foundation, making it difficult to establish a complete mapping ratio between virtual and real objects, leading to poor identification results. In recent years, with my country's continuously strengthening economic power, particularly the rapid development of information technology, numerous industry transformations have occurred. Machine learning and artificial intelligence technologies have made rapid breakthroughs, data collection methods represented by LiDAR and high-definition cameras are constantly being updated, and radar-video fusion technology is continuously accumulating experience. Edge computing technology is also being optimized, and it is now possible to basically achieve mapping and data fusion between LiDAR and video. Therefore, it is necessary to extract vehicle trajectories based on LiDAR and video data to serve road traffic condition identification and handling, thereby improving accident prevention and response capabilities. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a vehicle trajectory extraction method based on LiDAR and video data, which solves the problem of poor recognition results caused by the technical limitations of LiDAR and video, improves the effect of road traffic data collection, and proposes a method to realize the construction of a real-time, continuous, all-time, and all-element road traffic trajectory database.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A vehicle trajectory extraction method based on the complementarity of lidar and video technologies includes:

[0007] Construct an integrated radar device; the integrated radar device includes high-definition video equipment and lidar acquisition equipment;

[0008] Based on the video acquisition data collected by the high-definition video equipment, key targets are identified, and a basic image information database is constructed through image optimization and data filtering.

[0009] Based on the aforementioned lidar acquisition device, a data acquisition frequency is specified, key radar data information is extracted, and the radar data is converted into readable road and vehicle information to construct a database of objects and behaviors for target identification.

[0010] By integrating the object and behavior database and the basic image information database, a driving behavior detection dataset is constructed to achieve road element calibration, vehicle element calibration, and behavior parameter calibration.

[0011] The system identifies and tracks calibrated objects, maps virtual objects to real objects using proportional calculations, measures the coordinate changes and speed indicators of target objects, collects basic vehicle data, and, through data filtering and fusion, depicts driving behavior trajectories and outputs a trajectory database.

[0012] Preferably, the integrated radar device adopts an edge computing device. The integrated radar device is used to acquire radar point cloud data and video data in real time, and calculate the current environment, weather and road information, realize the collection of information such as vehicle real-time location, speed and acceleration, monitor vehicle driving route and driving status, and determine vehicle stopping point, off-site driving point, speeding, rapid acceleration and rapid deceleration.

[0013] Preferably, based on the video acquisition data collected by the high-definition video device, key targets are identified, and a basic image information database is constructed through image optimization and data filtering, including:

[0014] Keyframes are set, and the read rate is defined as 24Hz. The VideoCapture function in the OpenCV algorithm is used to extract video keyframes.

[0015] The keyframes read are saved as images in the background database, and the RGB images are converted into grayscale images based on the cvtColor function;

[0016] The Kalman filter function is used to filter image noise;

[0017] The Canny edge detection function in OpenCV is used to filter and obtain the edges in the image;

[0018] The fillConvexPoly function is used to generate a trapezoidal mask region, and then a bitwise_and operation is performed with the original image to obtain the edge detection map within the region of interest, so as to mark lane line information and vehicle environment element information;

[0019] Based on lane line information, the HoughLines function in OpenCV is used to find straight lines in the image by pixel points;

[0020] Based on the straight lines in the image, the boundaries are identified and each element in the image is marked. The DBSCEAN clustering algorithm is used to cluster vehicle and road environment information to build a basic image information database.

[0021] Preferably, based on the lidar acquisition device, a data acquisition frequency is specified, key radar data information is extracted, and the radar data is converted into readable road and vehicle information to construct a database of target identification objects and behaviors, including:

[0022] Obtain the key radar data information;

[0023] Key frames are marked based on the key radar data information, and the lidar timestamp is matched based on the key frame at the current time point to generate point cloud data in PCD format that is inherent to lidar detection at each time point.

[0024] A data observation plane is generated by fitting the point cloud data using the fit function; the observation plane includes the feedback distance and position information of objects in each frame.

[0025] The PointNet 3D point cloud object detection model is used to extract the key contour points of the point cloud data objects in the current frame. The point clouds are then clustered using a Gaussian mixture clustering model, and the objects are divided according to the clustering results.

[0026] Based on the clustering results, dangerous driving behaviors are identified and defined. Finally, point cloud clustering is used to obtain the characteristics of the target objects, including road environment, lanes, vehicles and field of vision, so as to build the object and behavior database of the identified targets.

[0027] Preferably, the object and behavior database and the basic image information database are integrated to construct a driving behavior detection dataset, thereby achieving road element calibration, vehicle element calibration, and behavior parameter calibration, including:

[0028] Based on the features obtained from point cloud data, the specific locations of each element are determined by combining video data. By assigning position and distance parameters to video objects, the target ratio relationship between point cloud data and video data is obtained, and the coordinates of key points of target objects in the video are extracted.

[0029] The coordinates of the target object are fitted with a polynomial relationship to the actual distance between the key point coordinates and the point cloud coordinates, so as to realize the mapping coordinates of the target object in the video to the actual distance.

[0030] Based on the mapping relationship of the mapped coordinates, the road surface length, distance, vehicles, and road width of each key frame are calculated to obtain the actual parameters of various objects in the actual scene;

[0031] Based on point cloud features, combined with radar and video object data, vehicle location identification and calibration are performed, ultimately forming a road traffic element database with proportional relationships and real-time calculation.

[0032] Preferably, the calibrated object is identified and tracked, and a proportional mapping between the virtual object and the actual object is achieved through proportional calculation. The coordinate changes and speed-related indicators of the target object are calculated to collect basic vehicle data. Through data filtering and fusion, driving behavior trajectories are depicted, and a trajectory database is output, including:

[0033] Each input frame is analyzed based on the YOLOv5 object detection algorithm, and objects belonging to specific categories are identified, providing classification and coordinates.

[0034] The deep learning deepsort method is used to extract appearance features, motion or interaction features, and a trajectory predictor is used to predict the next position of the target.

[0035] Calculate the similarity between two targets based on their appearance and motion characteristics;

[0036] Based on the calculated similarity, the detected objects and trajectories belonging to the same target are associated, and the detected objects are assigned the same ID as the trajectories.

[0037] The estimated state and variance of the tracked object are used to predict its trajectory using Kalman filtering.

[0038] The distance threshold, dist_thresh, is used to track the range. When the threshold is exceeded, the track is deleted and a new track is created.

[0039] The vehicle speed is calculated by taking the associated vehicle information, calculating the position information of the current vehicle in the previous frame, calculating the speed based on the deviation distance between the two, and thus obtaining basic parameters such as acceleration.

[0040] Based on the lane line edge, obtain the coordinate set outside the edge, compare the current vehicle coordinates with the coordinates outside the lane line, and filter out vehicles outside the road that are not in the current road.

[0041] An object tracking algorithm is used to identify and track objects detected by the device, generate trajectories, and output the vehicle's position, speed, and other basic information in the video, saving them in an Excel file.

[0042] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] This invention provides a vehicle trajectory extraction method based on the complementarity of LiDAR and video technologies, comprising: constructing an integrated radar device; the integrated radar device includes a high-definition video device and a LiDAR acquisition device; identifying key targets based on video acquisition data acquired by the high-definition video device, and constructing a basic image information database through image optimization and data filtering; based on the LiDAR acquisition device, specifying the data acquisition frequency, extracting key radar data information, converting the radar data into readable road and vehicle information, and constructing an object and behavior database for target identification; fusing the object and behavior database and the basic image information database, and constructing a driving behavior detection dataset to achieve road element calibration, vehicle element calibration, and behavior parameter calibration; identifying and tracking the calibrated objects, achieving a proportional mapping between virtual and real objects through proportional calculation, calculating target object coordinate changes and speed-related indicators, achieving basic vehicle data acquisition, and characterizing driving behavior trajectories through data filtering and fusion, and outputting a trajectory database. This invention can combine lidar and high-definition video data to build an edge computing device, enabling the extraction of road environment elements, vehicle operation elements and other elements from trajectory data. It can also realize the collection, matching and database construction of road traffic data, providing a data foundation for road traffic operation and risk identification, and providing a scientific basis for traffic management departments to evaluate and manage road traffic. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1A flowchart of the method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the video recognition element results provided in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a point cloud for lidar recognition provided in an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of vehicle target detection results provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of road vehicle trajectory data results provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0052] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, including a series of steps, processes, methods, etc., is not limited to the steps listed, but may optionally include steps not listed, or may optionally include other steps inherent to these processes, methods, products, or devices.

[0053] The purpose of this invention is to provide a vehicle trajectory extraction method based on the complementarity of LiDAR and video technology. This method can combine LiDAR and high-definition video data to build an edge computing device, thereby extracting road environment elements, vehicle operation elements and other elements from trajectory data. It can also realize the collection, matching and database construction of road traffic data, provide a data foundation for road traffic operation and risk identification, and provide a scientific basis for traffic management departments to evaluate and manage road traffic.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a vehicle trajectory extraction method based on the complementarity of LiDAR and video technologies, including:

[0056] Step 100: Construct an integrated radar device; the integrated radar device includes a high-definition video device and a lidar acquisition device;

[0057] Step 200: Identify key targets based on the video acquisition data collected by the high-definition video device, and construct a basic image information database through image optimization and data filtering;

[0058] Step 300: Based on the lidar acquisition device, specify the data acquisition frequency, extract key radar data information, convert the radar data into readable road and vehicle information, and construct a database of objects and behaviors for target identification;

[0059] Step 400: Merge the object and behavior database and the basic image information database, and construct a driving behavior detection dataset to achieve road element calibration, vehicle element calibration, and behavior parameter calibration;

[0060] Step 500: Identify and track the calibrated objects, realize the proportional mapping between virtual objects and actual objects through proportional calculation, calculate the coordinate changes and speed indicators of the target objects, realize the collection of basic vehicle data, and characterize the driving behavior trajectory through data filtering and fusion, and output the trajectory database.

[0061] This invention is a vehicle trajectory extraction method based on the complementary use of LiDAR and video technologies. It involves developing road and vehicle environment detection equipment and constructing a vehicle trajectory data acquisition platform based on the installation of LiDAR and high-definition video equipment. First, based on video acquisition data, key targets are identified. Through image optimization and data filtering, a basic image information database is constructed, thereby marking different road and vehicle elements using element detection equipment. Second, based on LiDAR acquisition equipment, a data acquisition frequency is specified, key radar data information is extracted, and the radar data is converted into readable road and vehicle information, constructing a database of target objects and behaviors. Third, the video and LiDAR marking databases are fused, the basic element and behavior databases are optimized, and a driving behavior detection dataset is constructed, achieving road element calibration, vehicle element calibration, and behavior parameter calibration. Then, the calibrated objects are identified and tracked. Through proportional calculation, a proportional mapping between virtual and real objects is achieved, and the coordinate changes and speed indicators of the target objects are calculated, realizing the acquisition of basic vehicle data. Through data filtering and fusion, driving behavior trajectories are depicted, and a trajectory database is output. Compared with existing technologies, this invention overcomes the current difficulties in trajectory data acquisition, unclear calibration, high recognition difficulty, and low recognition accuracy, and can improve the reliability of driving behavior recognition and prediction.

[0062] The first step involves developing road and vehicle environment detection equipment based on the installation of LiDAR and high-definition video equipment, and constructing a vehicle trajectory data acquisition platform. Through independently developed high-precision LiDAR and high-definition video detection equipment (with a time accuracy of 24Hz), this equipment utilizes edge computing to integrate LiDAR and video acquisition equipment. It can acquire radar point cloud data and video data in real time, and calculate current environmental, weather, and road information. This enables the collection of real-time vehicle location, speed, acceleration, and other information, monitoring vehicle routes and driving status, and determining vehicle stopping points, off-site driving points, speeding, rapid acceleration, and rapid deceleration. The equipment uses edge computing to collect and process LiDAR and video data in real time, completing the edge fusion and storage of road traffic data. Currently, it has been installed and tested at the Huaxiang Bridge East Interchange on the South Fourth Ring Road in Beijing. Through edge computing, it has achieved the fusion of LiDAR and video data, realizing the collection and processing of all-time, all-element road traffic trajectory data, and constructing a basic database.

[0063] The second step involves identifying key targets based on video acquisition data. Through image optimization and data filtering, a basic image information database is constructed, which is then used by feature detection equipment to mark different road and vehicle elements. The image recognition algorithm first sets keyframes and defines a reading rate of 24Hz. It employs the VideoCapture function from the OpenCV algorithm, with parameters set to standard MP4 data, to extract video keyframes. Next, the keyframes are saved as images in the background database. The cvtColor function is used to convert the RGB images into grayscale images. To optimize the images, a Kalman filter is used to filter image noise. Kalman filtering has advantages such as not requiring a large amount of storage space, clear computational steps, and suitability for computer processing, and it meets the noise processing needs of this study. Therefore, it was chosen as the filtering method. The main steps are as follows:

[0064] First, Kalman filtering consists of an estimation system and an observation system, which can be described by state equations and observation equations, respectively:

[0065] x k =Ax k-1 +Bu k-1 +q k-1

[0066] y k =Hx k +r k

[0067] in:

[0068] x k —The system state vector at time k;

[0069] u k —The driving input vector at time k;

[0070] q k —State noise that follows a Gaussian distribution has a covariance of Q;

[0071] y k —Observations of the system at time k;

[0072] r k —Observation noise that conforms to a Gaussian distribution has a covariance of R;

[0073] A and B are the state transition matrix and control input gain matrix, respectively, and are constants for single input and single output.

[0074] H – Observation matrix, which is a constant when there is a single input and a single output.

[0075] The above two equations are mainly calculated from the five state update equations, which are mainly divided into time update and state update. ① and ② are time updates, which update the prior state estimate and prior estimate covariance based on the results of the previous iteration. ③, ④ and ⑤ are state updates, which measure the weight of the estimation error in the total error by calculating the Kalman gain, and are used to correct the values ​​and obtain the optimal estimate.

[0076] ① State prior estimation:

[0077] ②Prior estimation of variance:

[0078] ③ Kalman gain matrix:

[0079] ④ State estimation correction:

[0080] ⑤ Variance estimation correction:

[0081] In the formula:

[0082] —The prior state estimate at time k;

[0083] —The posterior state estimate at time k, which is also the Kalman optimal estimate at the corresponding time. —The prior estimate of the covariance at time k;

[0084] —The posterior estimated covariance at time k is one of the filtering results.

[0085] u k If we set it to 0, the initial covariance P0 value can be preset as a non-zero constant, and its value will converge quickly during the iteration process.

[0086] Next, the Canny edge detection function in OpenCV is used to filter out the edges in the image. However, the processed image contains a lot of environmental information, which can easily lead to errors in identifying vehicles and road environments later. Therefore, redundant information needs to be removed to highlight the points of interest. The fillConvexPoly function is used with normal parameters to generate a trapezoidal mask region. Then, a bitwise AND operation is performed with the original image to obtain the edge detection map within the region of interest, thereby obtaining lane line information and vehicle environment element information. Based on the lane line information, the HoughLines function in OpenCV is used with normal parameters to find straight lines in the image through pixels. To solve the problem of discontinuity between dashed lines, the line segments obtained by the Hough transform need to be processed. The slope and intercept are calculated based on the Hough line, and then a complete straight line can be drawn directly using the averaged parameters. Finally, after determining the recognition boundaries, each element within the image was labeled. The DBSCEAN clustering algorithm was used to cluster vehicle and road environment information, constructing a basic video dataset. Through video verification and comparison, the recognition accuracy reached 96%, with an error of less than 0.03. This result demonstrates that this technical approach can achieve accurate recognition of road traffic elements, and the recognition effect is shown in the figure. Figure 2 .

[0087] The third step involves using a LiDAR acquisition device, specifying the data acquisition frequency, extracting key radar data, and converting the radar data into readable road and vehicle information to construct a database of target objects and behaviors. A corresponding 5-minute video is selected, with 5200 data-labeled keyframes out of 7200 frames. Based on these keyframes and matching the LiDAR timestamp, point cloud data in PCD format (inherent from the LiDAR detection) is generated at each point. A data observation plane is then generated using a fit function based on the point cloud information to obtain the object feedback distance and position information for each frame. The 3D point cloud target detection model PointNet is used, with the following parameters: lovasz-softmax loss function weight 1, perception-aware loss function weight 0.5, confidence threshold 0.7, epochs: 100, lr: 0.001, warmup_epochs: 1, momentum: 0.9, weight_decay: 0.00001. This extracts the key contour points of objects in the current frame's point cloud data. A Gaussian mixture clustering model is then used to cluster the point clouds, and objects are classified based on the clustering results. A Gaussian mixture model structure is constructed using independent parameters between the diagonal covariance matrix and the Gaussian components, and the number of mixture components (K) and the number of clusters are estimated. In the algorithm, the Gaussian clustering model with K components can be expressed as the following equation:

[0088]

[0089] β={w i ,μ i ,∑i)}

[0090]

[0091] In the formula, x is a set of m-dimensional vectors, and β = {w i ,μ i ,∑i)} are the parameters of the Gaussian mixture model, K is the number of classes in the Gaussian mixture model, w i (i = 1, ..., k) are those that satisfy the constraints. The covariance matrix, μ i It is the mean vector, ∑i is the covariance matrix of the Gaussian mixture model, and g(x|μ) is the mean vector. i ∑i) is the Gaussian density of the components, and the density of each component is consistent with the average vector μ. i The m-dimensional Gaussian density function of the covariance matrix ∑i describes the probability density of vectors belonging to a cluster.

[0092] The parameter β of the Gaussian mixture clustering model was obtained using the Expectation-Maximization (EM) algorithm. This algorithm starts with an initial model and iteratively creates new models with the goal of maximizing the expectation, repeating this process until the expected improvement no longer becomes significant, at which point training stops. This study uses 1x10^6 scalar arrays. -6 The termination tolerance (ε) for the objective function value is used to stop model training when the error is less than this value. Generally, Gaussian mixture clustering will test several different clustering values ​​simultaneously and select the best classification value based on the Akaike Information Criterion (AIC), thus confirming the class centers and margins. The most reasonable number of classes is selected based on AIC; when AIC = 1120, the optimal model is obtained, and four clusters are identified, defined as road environment, traffic flow, lane markings, and vehicle trajectory. See the example of LiDAR point cloud recognition results. Figure 3 .

[0093] The fourth step involves fusing video and LiDAR marker databases, optimizing the basic element and behavior databases, and constructing a driving behavior detection dataset to calibrate road elements, vehicle elements, and behavior parameters. Using the keyframes constructed in the second and third steps, combined with video and radar data, the specific locations of each element are determined. By assigning position and distance parameters to video objects, the ratio of the target obtained from point cloud data and video data to the actual target is obtained as 1:325. Based on the keypoint coordinates of the target object and the actual distance between the point cloud coordinates, the `np.poly1d(np.polyfit(x,y,n_order))` function is used, with parameters set to the set of point coordinates and the polynomial exponent, to fit the polynomial relationship between the actual object and the virtual object as follows:

[0094] F(X)=-0.00002150023889154337*x*x+0.04679885332059244*x+4.535117056856188

[0095] The fitting effect analysis showed that the relationship had an R² value of 0.85, indicating a good fit. Based on this mapping relationship, combined with radar and video object data, vehicle position identification and calibration were performed, ultimately forming a road traffic element database with proportional relationships and real-time calculation.

[0096] The fifth step is to identify and track the calibrated objects. Through proportional calculation, a proportional mapping between virtual and real objects is achieved. The coordinate changes and speed indicators of the target objects are measured to collect basic vehicle data. Through data filtering and fusion, driving behavior trajectories are depicted, and a trajectory database is output. Based on the feature database, the YOLO model is used for vehicle identification, detection, and tracking. First, each input frame is analyzed using the YOLOv5 object detection algorithm, identifying objects belonging to specific categories and providing classification and coordinates. Second, the deep learning deepsort method is used with parameters set as follows: maximum distance 0.2, minimum confidence 0.5, maximum overlap 0.5, initial value 3, maximum consecutive misses 70, and correction sample count 100, to extract appearance features, motion, or interaction features. Furthermore, a trajectory predictor is used to predict the next position of the target. Simultaneously, based on the appearance and motion features of the target object, the similarity between two targets is calculated using Euclidean distance as the standard, and targets with a similarity greater than 0.85 are designated as the same target. Based on the calculated similarity, detected objects and trajectories belonging to the same target are associated, and the detected objects are assigned the same ID as the trajectory. Further, Kalman filtering is used to estimate the state and variance of the track for prediction of the target trajectory. The `dist_thresh` function is used with parameters set to normal values, setting a threshold of 20 to determine the tracking range. When the threshold is exceeded, the track is deleted, and a new track is created. Based on the marked vehicles, vehicle speed is calculated. According to the associated vehicle information, the position information of the current vehicle in the previous frame is calculated, and the speed is calculated based on the deviation distance between the two, thus obtaining basic parameters such as acceleration. Based on the lane line edges, the coordinate set of the outer edge is obtained. By comparing the current vehicle coordinates with the coordinates outside the lane line, vehicles outside the road that are not currently on the road are filtered out. Meanwhile, there are large error fluctuations at the near and far ends of the video; therefore, during the filtering process, targets whose aspect ratio changes by more than 2 times need to be filtered. Finally, an object tracking algorithm is used to identify and track the objects detected by the device to generate trajectories. The trajectory tracking effect is shown in [link to documentation]. Figure 4The system outputs the vehicle's location, speed, and other basic information from the video, saving it in an Excel file. See the final database example below. Figure 5 .

[0097] The beneficial effects of this invention are as follows:

[0098] 1) This invention utilizes independently developed road traffic data detection equipment, combined with lidar and high-definition video data to construct an edge computing device, enabling the extraction of road environmental elements, vehicle operation elements, and other elements from trajectory data. This facilitates the collection, matching, and database construction of road traffic data, providing a data foundation for road traffic operation and risk identification, and offering a scientific basis for traffic management departments to evaluate and manage road traffic.

[0099] 2) This invention uses a Gaussian mixture clustering model to cluster LiDAR point cloud data and integrates video data to label different road traffic operation elements. By using video, it addresses the weakness of LiDAR object labeling. At the same time, based on the precise distance and position data calculated by the radar, it optimizes the data robustness in video recognition. The YOLOv5 target detection algorithm is used to realize vehicle trajectory recognition and tracking. The final database not only contains static data of various road traffic elements but also dynamic data such as vehicle operation. This data can effectively match the traffic state in the actual scene through a virtual mapping method, providing a basic method for further research on road traffic operation and safety identification and prediction, and providing data and technical support for the management of road risks.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0101] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1.A vehicle trajectory extraction method based on complementation of lidar and video technology, characterized in that, The application relates to a radar integrated device. The radar integrated device comprises a high-definition video device and a laser radar acquisition device. Key targets are identified according to video acquisition data acquired by the high-definition video device, and a basic image information database is constructed through image optimization and data screening. Based on the laser radar acquisition device, a data acquisition frequency is specified, key radar data information is extracted, radar data is converted into readable road and vehicle information, and an object and behavior database for identifying targets is constructed. The object and behavior database and the basic image information database are fused, and a driving behavior detection data set is constructed to realize road element calibration, vehicle element calibration and behavior parameter calibration. The calibrated objects are identified and tracked, the proportional relationship between virtual objects and actual objects is mapped through proportional calculation, target object coordinate changes and speed indicators are calculated, vehicle basic data acquisition is realized, and driving behavior trajectories are described through data screening and fusion to output a trajectory database. Key frames are set, and a reading rate of 24Hz is defined, VideoCapture functions in OpenCV algorithms are used to extract video key frames. The read key frames are saved in the background database in the form of images, and RGB images are converted into grayscale images based on the cvtcolor function. Image noise is filtered by using a Kalman filter function. The Canny edge detection function in OpenCV is used to filter the edges in the image. A trapezoidal mask region is generated by using the fillConvexPoly function, and then bitwise_and operation is performed on the original image to obtain an edge detection image in the region of interest, so as to mark lane line information and vehicle environment element information. Based on the lane line information, the HoughLines function in OpenCV is used to find straight lines in the image through pixel points. The boundaries are identified according to the straight lines in the image, and each element in the image is marked, and the vehicle and road environment information is clustered through the DBSCEAN clustering algorithm to construct the basic image information database. Based on the laser radar acquisition device, a data acquisition frequency is specified, key radar data information is extracted, radar data is converted into readable road and vehicle information, and an object and behavior database for identifying targets is constructed. The key radar data information is acquired. Key frames are marked according to the key radar data information, PCD format point cloud data generated by the laser radar detection is generated at each time point based on the current time point key frame matching the laser radar timestamp. An observation plane is generated by fitting the point cloud data through a fit function; the observation plane comprises object feedback distance and position information of each frame. A 3D point cloud object detection model PointNet is used to extract key contour points of the current frame point cloud data object, each point cloud is clustered through a Gaussian mixture clustering model, and objects are divided according to the clustering results. ​ ​ According to the clustering result, the dangerous driving behavior is identified and defined, and finally the target object including the road environment, the lane, the vehicle and the field of view range characteristics is obtained through point cloud clustering, so as to realize the construction of the object and behavior database of the identified target. 2.The vehicle trajectory extraction method based on the complement of lidar and video technology, according to claim 1, wherein, The radar integrated device adopts an edge computing device, is used for acquiring radar point cloud data and video data in real time, and calculates current environment, weather and road information, realizes information collection such as real-time position, speed and acceleration of vehicle, monitors driving route and driving state of vehicle, and judges vehicle staying point, off-road driving point, overspeed driving, rapid acceleration and rapid deceleration. 3.The vehicle trajectory extraction method based on the complement of lidar and video technology, according to claim 1, wherein, The object and behavior database and the basic image information database are fused, and a driving behavior detection data set is constructed, realizing road element calibration, vehicle element calibration and behavior parameter calibration, including: According to the characteristics obtained from the point cloud, the specific point of each element is determined in combination with the video data, the position and distance parameters are given to the video object, the target proportion relationship obtained from the point cloud data and the video data is obtained, and the key point coordinates of the target object in the video are extracted; According to the key point coordinates of the target object and the real distance of the point cloud coordinates, the mapping coordinates of the coordinate generation real distance of the video target object are realized through the fitting of the polynomial relationship; According to the mapping relationship of the mapping coordinates, the road surface length distance, vehicle and road surface width of each key frame are calculated, and the actual parameters of each object in the actual scene are obtained. According to the point cloud characteristics, the vehicle position recognition and calibration are carried out in combination with the radar and video object data, and finally the road traffic element database with proportion relationship and real-time calculation is formed. 4.The vehicle trajectory extraction method based on the complement of lidar and video technology, according to claim 1, wherein, The calibrated objects are identified and tracked, the equal proportion relationship mapping of virtual objects and actual objects is realized through proportion measurement, the target object coordinate change and speed index are measured, the vehicle basic data acquisition is realized, and the driving behavior trajectory is described through data screening and fusion, and the trajectory database is output, including: Each input frame is analyzed based on YOLOv5 target detection algorithm, and objects belonging to specific categories are identified, and classification and coordinates are given; Deep learning deepsort method is used to extract apparent characteristics, motion or interaction characteristics, and a trajectory predictor is used to predict the next position of the target; According to the apparent characteristics and motion characteristics of the target object, the similarity between two targets is calculated; The similarity calculated is used as the basis to associate the detection objects and trajectories belonging to the same target, and the detection objects are assigned the same ID as the trajectories; The estimated state and estimated variance of Kalman filter type tracking are used to predict the target trajectory; The distance threshold dist_thresh is used to determine the tracking range, and when the threshold is exceeded, the track will be deleted and a new track will be created. The vehicle speed is calculated, the current vehicle position information is calculated according to the associated vehicle information, the speed is calculated according to the deviation distance between the two, and the basic parameters such as acceleration are obtained; According to the lane line edge, the coordinate set outside the edge is obtained, and the vehicles outside the current road are filtered by comparing the current vehicle coordinates with the coordinates outside the lane line. The object tracking algorithm is used to identify and track the object detected by the equipment to generate a track, and the position information, speed information and other basic information of the vehicle in the video are output and saved in an Excel file.

Images