A microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion

Abstract

The present invention belongs to the field of intelligent transportation, and specifically relates to a microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion. The specific perception of the microscopic vehicle trajectory perception device includes the following steps: 1) extracting the three-dimensional spatial information of the vehicle target; 2) using machine vision methods to identify the vehicle target in the image; 3) using coordinate calibration fusion strategy to obtain the semantic information and spatial coordinate information of the vehicle target; 4) using the DeepSORT algorithm to track the identified vehicle target, and applying the trained convolutional neural network to identify the vehicle target; 5) sending the information of traffic participants on the entire intersection or road to the traffic management center or the surrounding intelligent networked vehicles through the network port. The technical solution of the present invention brings the following results: 1) improving detection accuracy and environmental adaptability; 2) reducing the number of calculations and improving the system operation speed; 3) improving the target tracking accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation, and in particular relates to a microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion. Background Art

[0002] As an important research field in intelligent transportation, multi-sensor machine vision fusion aims to obtain more information about target vehicles in the area of ​​interest and the status of each target, and then track the vehicles in the area of ​​interest to obtain vehicle trajectories. This process includes target detection, tracking and data association, and data association mainly uses appearance features, motion features, shape features, etc. The current multi-sensor machine vision fusion is mainly based on deep learning and machine learning algorithms.

[0003] Microscopic vehicle trajectory perception is often achieved through real-time monitoring of vehicle conditions and driving roads through multiple sensors. Commonly used on-board sensors include cameras, laser radars, millimeter-wave radars, and infrared sensors. On-board cameras collect color information from the road or inside the car, and use machine vision technology to identify vehicles, pedestrians, road signs, or driver intentions; laser radars obtain three-dimensional point cloud information of the road, and can accurately judge the speed and distance of surrounding vehicles; these sensors also have their own hardware conditions in some aspects, and unilateral sensor data collection has certain limitations, which cannot fully meet the needs of autonomous vehicles at the environmental perception level. For example, cameras are greatly affected by light and shadows, and lack accurate location information; laser radars obtain fewer object features and are greatly affected by rainy days. Therefore, multiple sensors are needed to collect and fuse data, so that their respective advantages can be brought into play, their respective limitations can be compensated, and complete information can be provided for autonomous vehicles to ensure vehicle driving safety.

[0004] The research on multi-sensor machine vision fusion has very practical and realistic significance, and has important applications in the fields of intelligent monitoring, motion and behavior analysis, autonomous driving, virtual reality, and entertainment interaction. For example, in an autonomous driving system, the target tracking algorithm needs to track the movement of moving cars, pedestrians, and other animals, and predict their future positions, speeds, and other information; in the field of virtual reality, it is necessary to achieve the purpose of human-computer interaction based on the movements and trajectories of the characters captured by the camera; in the field of intelligent security, in the deployed monitoring system environment, security patrol robots are used to locate, track, and identify the identity of pedestrians. Therefore, multi-sensor machine vision fusion has a wide range of uses in various fields, making its research very important in practical significance.

[0005] Existing perception devices can only detect targets, but cannot track and classify them. With the development of artificial intelligence and convolutional neural networks, machine vision has been better developed. This product uses machine vision to extract features, thereby realizing target detection, tracking and classification. This device can not only realize target detection, but also track the detected targets to extract trajectories, such as classifying pedestrians, motor vehicles and non-motor vehicles in video sequences, so that the accuracy, practicality and speed of detection and tracking are improved. Summary of the invention

[0006] In view of the technical problems existing in the background technology, the present invention proposes a microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion.

[0007] The technical solution adopted to achieve the purpose of the present invention is: a microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion, and its specific perception steps are:

[0008] 1) The voxel downsampling algorithm is used to reduce the point cloud density while retaining the point cloud structure, and the point cloud is segmented according to the sensing range requirements of the laser radar. The grid filtering method or polar coordinate filtering method is used to identify and remove the ground point clouds that are not of concern. According to the physical properties of the laser radar, a DBSCAN algorithm with a radius that changes adaptively with the point cloud distance is used to cluster the obstacle point cloud. After clustering, the three-dimensional spatial information of the vehicle target is extracted;

[0009] 2) Use machine vision methods to identify vehicle targets in images, use the Yolov5 algorithm to train the pre-prepared training set, generate the weight file required to identify vehicle targets, and then call the Yolov5 algorithm and weight file to identify vehicle targets in camera images;

[0010] 3) A coordinate calibration fusion strategy is adopted to fuse the target information recognized by the lidar and the camera. The coordinate systems of the two are jointly calibrated based on the relative installation position and reference objects, so that two different sensors can recognize the same target in the same coordinate system. Then, the timestamp information is used to ensure the time synchronization of the two sensor information. After the information fusion is completed, the semantic information and spatial coordinate information of the vehicle target are obtained.

[0011] 4) Use the DeepSORT algorithm to track the identified vehicle targets, add appearance measurement information based on SORT, and apply the trained convolutional neural network to distinguish different models and identify vehicle targets;

[0012] 5) The acquired information is directly passed through the camera. The camera can obtain the integrated information of all traffic participants and send the information of traffic participants on the entire intersection or road to the traffic management center or the surrounding intelligent connected vehicles through the network port.

[0013] Preferably, in step 1), the constraints of the plane normal vector are introduced by utilizing the angle characteristics of the horizontal plane to improve the accuracy of ground plane fitting.

[0014] Preferably, in step 3), the information fusion index of intersection over union (IOU) is used to improve the fusion accuracy of the three-dimensional recognition result and the two-dimensional recognition result of the space object.

[0015] The technical solution of the present invention brings the following results:

[0016] (1) Improve detection accuracy and environmental adaptability. This device uses image information in vehicle detection and takes advantage of the deep convolutional neural network algorithm YOLOv5 in visual target recognition to improve the accuracy of the detection system. Radar point cloud information can also be used to extract vehicle features. It can integrate point cloud and visual information to detect vehicles, further improving detection accuracy and environmental adaptability.

[0017] (2) Reduce the number of calculations and improve the system operation speed. This device predicts the motion state of each tracked vehicle in each frame of the image. However, the calculation process of vehicle state prediction, especially the solution process of the Jacobian matrix in the extended Kalman filter, is relatively time-consuming. In addition, the period between each frame is relatively short, and the vehicle motion state does not change much. Therefore, the motion state can be predicted selectively in segments, which reduces the number of computer calculations and improves the system operation speed.

[0018] (3) Improve target tracking accuracy. Through the environmental information on the vehicle road provided by the fusion of the camera and the lidar, the vehicle target detected in each frame is tracked to obtain a unique ID of the vehicle and its corresponding motion trajectory. When the vehicle is not detected due to occlusion, lighting, etc., the position of the vehicle in this frame can be predicted based on its tracking trajectory, thereby improving the accuracy of vehicle tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a network algorithm structure diagram of the YoloV5 algorithm used in the present invention.

[0020] Figure 2 This is the DeepSORT workflow diagram used in the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with specific embodiments.

[0022] The present invention provides a microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion, and its key perception technologies include: target detection and target tracking, specifically:

[0023] Key Technology 1: Object Detection

[0024] The present invention adopts the YOLOv5 algorithm to detect target vehicles. The YOLOv5 algorithm is a classic target detection algorithm, which can achieve high-precision and high-timeliness target detection. Its main core idea is to divide the picture to be detected into N×N grids. When the center position of the target to be detected is located at a certain grid, then this grid will generate M target frames, and each target frame contains the following information: the coordinate position information of the target, the classification information of the detected target, and the confidence for the detected target. Finally, the non-maximum suppression method is used to select the target frame with the highest confidence from the generated M target frames as the target frame of the final detected object. The network algorithm structure diagram is shown in the figure below. It can be seen that the use of YOLOv5 to detect targets can be divided into four stages, namely the input end stage, the backbone network stage (Backbone), the feature fusion layer stage (Neck) and the detection layer stage (Prediction).

[0025] The Yolov5 algorithm can achieve higher recognition accuracy and faster recognition speed, and can accurately identify vehicle targets in complex image information, ensuring that the semantic information and pixel coordinate information of the vehicle target are accurately provided during the subsequent sensor information fusion.

[0026] Key Technology 2: Target Tracking

[0027] This product uses the DeepSORT algorithm for vehicle tracking. DeepSORT is an improved algorithm based on the idea of ​​SORT. The SORT algorithm uses a simple Kalman filter to process the correlation of frame-by-frame data and uses the Hungarian algorithm for correlation measurement. This simple algorithm achieves good performance at high frame rates. However, since SORT ignores the surface features of the detected object, it will only be accurate when the uncertainty of the object state estimation is low. In DeepSORT, a more reliable metric is used to replace the correlation metric, and a CNN network is used to train on a large-scale pedestrian data set and extract features, which has increased the network's robustness to loss and obstacles. The Hungarian algorithm and Kalman filter process are as follows:

[0028] 1. Hungarian Algorithm

[0029] The Hungarian algorithm (also called the KM algorithm) is a method used to solve the allocation problem. It is based on the theorem: if a row or column of the cost matrix is ​​added or subtracted with a certain number at the same time, the optimal allocation of the new cost matrix is ​​still the optimal allocation of the original cost matrix.

[0030] Algorithm steps (assuming the matrix is ​​an NxN square matrix):

[0031] (1) For each row of the matrix, subtract the smallest element;

[0032] (2) For each column of the matrix, subtract the smallest element;

[0033] (3) Cover all the elements in the matrix with the fewest horizontal or vertical lines;

[0034] (4) If the number of lines is equal to N, the optimal allocation is found and the algorithm ends, otherwise it goes to step 5;

[0035] (5) Find the smallest element that is not covered by any line, subtract this element from each row that is not covered by a line, add this element to each column that is covered by a line, and return to the previous step.

[0036] 2. Kalman Filter

[0037] It is widely used in drones, autonomous driving, satellite navigation and other fields. In simple terms, its function is to update the predicted value based on the sensor's measurement value to achieve a more accurate estimate. Suppose we want to track the position change of the car, as shown in the figure below, the blue distribution is the Kalman filter prediction value, the brown distribution is the sensor's measurement value, and the gray distribution is the optimal estimate of the predicted value after the measurement value is updated.

[0038] In target tracking, the following two states of the track need to be estimated: Mean: represents the location information of the target, which is composed of the center coordinates of the bbox (cx, cy), aspect ratio r, height h, and the respective speed change values. It is represented by an 8-dimensional vector x = [cx, cy, r, h, vx, vy, vr, vh], and each speed value is initialized to 0. Covariance: represents the uncertainty of the target location information, represented by an 8x8 diagonal matrix. The larger the number in the matrix, the greater the uncertainty. It can be initialized with any value. Kalman filtering is divided into two stages: (1) predicting the position of the track at the next moment, and (2) updating the predicted position based on detection. The prediction is to predict the state of the track at time t based on its state at time t-1.

[0039] The following is a further description with reference to specific examples.

[0040] The present invention provides a microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion, and its specific perception steps are as follows:

[0041] 1) First, the voxel downsampling algorithm is used to reduce the point cloud density while retaining the point cloud structure, and the point cloud is segmented according to the sensing range requirements of the laser radar. Then, the ground point clouds that are not of concern are identified and removed using the grid filtering method or the polar coordinate filtering method. In order to solve the problem that the vertical surface of tall buildings will interfere with the fitting plane, the plane normal vector constraint is introduced using the angle characteristics of the horizontal plane to improve the accuracy of ground plane fitting. According to the physical properties of the laser radar, a DBSCAN algorithm with a radius that changes adaptively with the point cloud distance is used to cluster the obstacle point cloud, which effectively overcomes the sensitivity of the clustering parameters and improves the clustering accuracy of the obstacle point cloud at different positions. After clustering, the three-dimensional spatial information of the vehicle target is extracted.

[0042] 2) Then, the machine vision method is used to identify the vehicle targets in the image. The Yolov5 algorithm is used to train the pre-prepared training set to generate the weight file required for identifying the vehicle target, and then the Yolov5 algorithm and the weight file are called to identify the vehicle target in the camera image. The algorithm has a small network structure and low computing power requirements. It can be deployed on the road monitoring camera end with a lower hardware configuration, and can ensure the real-time and accuracy of vehicle target recognition at the same time.

[0043] 3) Then the present invention adopts a coordinate calibration fusion strategy to fuse the target information recognized by the laser radar and the camera. In view of the phenomenon that the vertical scanning range of the laser radar is smaller than the field of view of the camera, the information fusion index of intersection over union (IOU) is proposed to improve the fusion accuracy of the three-dimensional recognition results and the two-dimensional recognition results of the spatial object. According to the relative installation position, the coordinate systems of the two are jointly calibrated through reference objects to realize the recognition of the same target by two different sensors in the same coordinate system, and then the timestamp information is used to ensure the time synchronization of the information of the two sensors. After the information fusion is completed, the semantic information and spatial coordinates of the vehicle target are obtained.

[0044] 4) The present invention then uses the DeepSORT algorithm to track the identified vehicle targets. SORT is a simpler framework that performs Kalman filtering and frame-by-frame data association in image space using the Hungarian algorithm and an association metric that can measure bounding box overlap. This simple method can achieve good performance at high frame rates. Although good performance was achieved overall in terms of tracking precision and accuracy, SORT returned a large number of identity switches, mainly because only motion association metrics were used. Therefore, SORT has defects in tracking occlusions, as occlusions often occur in front-view camera scenes. Therefore, this product uses the DeepSORT algorithm, an upgraded version of the SORT algorithm, which adds appearance metric information to SORT and applies a trained convolutional neural network to distinguish different vehicle models, so that the DeepSORT algorithm can better handle occlusion problems and better identify vehicle targets.

[0045] 5) Finally, the present invention directly passes the acquired information inside the camera, which can obtain the fused information of all traffic participants and send the information of traffic participants on the entire intersection or road to the traffic management center or the surrounding intelligent networked vehicles through the network port.

[0046] The present invention applies the YOLOv5 algorithm based on the deep learning framework to the multi-target detection and tracking algorithm. YOLOv5 is a multi-classification, multi-target detection algorithm with high accuracy and relatively fast running speed. The present invention applies the YOLOv5 algorithm to the detection and tracking system, significantly improving the detection accuracy at the detection level without occupying too much computing resources, ensuring the efficient operation of the subsequent tracking system.

[0047] The present invention uses the Deepsort algorithm for target tracking, which is an improvement on the Sort algorithm. Sort is a practical method that focuses on simple and effective algorithms for multi-object tracking. However, the Sort algorithm cannot handle occlusion problems well because the data association metric used is only based on the association of motion information. It is accurate only when the state estimation is relatively stable. By replacing the association metric with a more comprehensive metric that combines motion and appearance information, a trained Cosine deep network is applied to distinguish pedestrians on a large-scale person re-identification dataset. Due to this extension, target tracking under occlusion can be handled, thereby effectively reducing the number of identity switches.

Claims

1. A microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion, characterized by: The specific sensing steps of the microscopic vehicle trajectory sensing device are as follows: 1) The voxel downsampling algorithm is used to reduce the point cloud density while retaining the point cloud structure, and the point cloud is segmented according to the sensing range requirements of the laser radar. The grid filtering method or polar coordinate filtering method is used to identify and remove the ground point clouds that are not of concern. According to the physical properties of the laser radar, a DBSCAN algorithm with a radius that changes adaptively with the point cloud distance is used to cluster the obstacle point cloud. After clustering, the three-dimensional spatial information of the vehicle target is extracted; 2) Use machine vision methods to identify vehicle targets in images, use the Yolov5 algorithm to train the pre-prepared training set, generate the weight file required to identify vehicle targets, and then call the Yolov5 algorithm and weight file to identify vehicle targets in camera images; 3) A coordinate calibration fusion strategy is adopted to fuse the target information recognized by the lidar and the camera. The coordinate systems of the two are jointly calibrated based on the relative installation position and reference objects, so that two different sensors can recognize the same target in the same coordinate system. Then, the timestamp information is used to ensure the time synchronization of the two sensor information. After the information fusion is completed, the semantic information and spatial coordinate information of the vehicle target are obtained. 4) Use the DeepSORT algorithm to track the identified vehicle targets, add appearance measurement information based on SORT, and apply the trained convolutional neural network to distinguish different models and identify vehicle targets; 5) The acquired information is directly passed through the camera. The camera can obtain the integrated information of all traffic participants and send the information of traffic participants on the entire intersection or road to the traffic management center or the surrounding intelligent connected vehicles through the network port.

2. The microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion according to claim 1 is characterized in that: In step 1), the angle characteristics of the horizontal plane are used to introduce constraints on the plane normal vector to improve the accuracy of ground plane fitting.

3. The microscopic vehicle trajectory perception device based on multi-sensor machine vision fusion according to claim 1 is characterized in that: In step 3), the information fusion index of intersection over union (IOU) is used to improve the fusion accuracy of the three-dimensional recognition result and the two-dimensional recognition result of the spatial object.

Images