Vehicle-road cooperative intelligent sensing method

Abstract

The invention provides a vehicle-road cooperative intelligent sensing method, and relates to the technical field of artificial intelligence, automatic driving and machine vision. The method specifically comprises the following steps: for any road scene, respectively collecting and storing roadside point cloud data, roadside image data, vehicle-mounted point cloud data and vehicle-mounted image data; data processing is carried out on the roadside point cloud data and the vehicle-mounted point cloud data, target detection is carried out on the processed vehicle-mounted point cloud data, the vehicle-mounted image data, the processed roadside point cloud data and the processed roadside image data by adopting a YOLO intelligent sensing fusion algorithm, and obstacle information is identified; and according to the identified obstacle information, a vortex artificial potential field APFV algorithm is adopted to carry out path planning on the obstacle avoidance vehicle, the driving direction and the driving speed of the obstacle avoidance vehicle are generated, and the obstacle avoidance vehicle is enabled to carry out automatic driving according to the generated driving direction and the driving speed. According to the invention, the sensing range of the vehicle is expanded to overcome the defects of a single-road vehicle.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, autonomous driving, and machine vision, and particularly to a vehicle-road collaborative intelligent perception method. Background Art

[0002] Autonomous driving technology is a revolutionary technology with many potential benefits and application prospects. By utilizing advanced sensors, controllers, and decision-making algorithms, autonomous driving can provide higher safety and efficiency, and solve problems such as traffic congestion and accidents.

[0003] Currently, the research on autonomous driving technology mainly focuses on the autonomous driving of individual vehicles. These vehicles use various sensors to perceive the surrounding environment and make decisions and controls through algorithms. However, there are limitations in the autonomous driving of individual vehicles, such as blind spots in the field of vision and sensor failures. To overcome these problems, vehicle-road collaborative autonomous driving has become a new research direction. Vehicle-road collaborative autonomous driving uses sensors at both the vehicle end and the road end to jointly sense the road traffic environment and realizes data exchange and sharing to enhance the perception and prediction capabilities of autonomous vehicles. Vehicle-road collaborative autonomous driving can expand the perception range of vehicles, reduce blind spots in the field of vision, and provide a redundant perception system to cope with sensor failures. By placing sensors on the road, vehicle-road collaboration can obtain different perception perspectives and improve the perception ability. At the same time, by sharing data with road equipment, vehicles can obtain more comprehensive information, improve the accuracy of decision-making and control, and enhance the safety of autonomous driving.

[0004] Existing research on autonomous driving mainly focuses on the intelligence of individual vehicles. Autonomous vehicles use their sensors, such as lidar and cameras, to perceive the surrounding environment and make control decisions through their algorithms. In recent years, the perception and computing capabilities of individual vehicle autonomous driving have developed very advanced, almost reaching the bottleneck of development.

[0005] At the same time, the limitations of individual vehicle autonomous driving itself make it impossible to fundamentally solve the problems of autonomous driving. First, the perception range of sensors is limited, so that individual vehicles can only detect objects within a certain range, and the information obtained is also difficult to effectively support the comprehensive perception of the road conditions ahead. Second, the autonomous driving system of individual vehicles can only observe the environment from a single angle, and the occlusion of other objects may cause perception blind spots, thus increasing the risk of traffic accidents. In addition, the autonomous driving of individual vehicles usually can only obtain target position information from one side, which may also lead to inaccurate position judgment. Summary of the Invention

[0006] Aiming at the deficiencies of the above-mentioned existing technologies, based on the YOLO intelligent perception fusion algorithm and the Vortex Artificial Potential Field (APFV) algorithm, the present invention proposes a vehicle-road collaborative intelligent perception method, aiming to provide multi-field and multi-perspective perception, expand the perception range of vehicles, and overcome the defects of single-road vehicles.

[0007] A vehicle-road collaborative intelligent perception method proposed by the present invention includes:

[0008] For any road scene, the road scene includes: lanes, obstacle avoidance vehicles, and obstacles; where the obstacles include: vehicles and pedestrians; roadside sensors are set on the lanes, and on-vehicle sensors are set on the obstacle avoidance vehicles;

[0009] Use roadside sensors to collect roadside point cloud data and roadside image data of the lane, use on-vehicle sensors to collect on-vehicle point cloud data and on-vehicle image data of the obstacle avoidance vehicle, and save the roadside point cloud data and on-vehicle point cloud data in PCD format;

[0010] Perform data preprocessing, data screening, and data repair on the roadside point cloud data and on-vehicle point cloud data in sequence to obtain the processed roadside point cloud data and on-vehicle point cloud data;

[0011] Use the YOLO intelligent perception fusion algorithm to perform object detection on the processed on-vehicle point cloud data, on-vehicle image data, processed roadside point cloud data, and roadside image data to identify obstacle information;

[0012] According to the identified obstacle information, use the vortex artificial potential field APFV algorithm to perform path planning for the obstacle avoidance vehicle, generate the driving direction and driving speed of the obstacle avoidance vehicle, so that the obstacle avoidance vehicle performs autonomous driving according to the generated driving direction and driving speed;

[0013] The on-vehicle sensors include: on-vehicle lidar, on-vehicle camera, and Global Navigation Satellite System (GNSS) module;

[0014] The on-vehicle lidar is used to obtain radar point cloud data of the lane and obstacles in real time during the driving process of the obstacle avoidance vehicle by emitting laser beams and receiving reflected signals;

[0015] The on-vehicle camera is used to obtain image data of the lane and obstacles in real time during the driving process of the obstacle avoidance vehicle;

[0016] The Global Navigation Satellite System (GNSS) module is used to obtain the position of the obstacle avoidance vehicle and calculate the rotation angle of the obstacle avoidance vehicle;

[0017] The roadside sensors include: roadside lidar and roadside camera;

[0018] The roadside lidar is used to obtain the radar point cloud data of obstacle avoidance vehicles and obstacles on the lane in real time;

[0019] The roadside camera is used to obtain the image data of obstacle avoidance vehicles and obstacles on the lane in real time;

[0020] Saving the roadside point cloud data and in-vehicle point cloud data in PCD format is expressed as: each point cloud data in the roadside point cloud data and in-vehicle point cloud data includes the Cartesian coordinates of the point cloud and the intensity value of the point cloud, expressed as: (x, y, z, i), where x is the abscissa; y is the ordinate; z is the vertical coordinate; i is the intensity value;

[0021] The process of using the YOLO intelligent perception fusion algorithm to perform object detection on the processed in-vehicle point cloud data, in-vehicle image data, processed roadside point cloud data, and roadside image data includes:

[0022] Fusing the processed in-vehicle point cloud data, in-vehicle image data, processed roadside point cloud data, and roadside image data by using pixel-level fusion or feature-level fusion methods to obtain vehicle-road fusion data;

[0023] Using a fully connected layer to extract features from the vehicle-road fusion data, and then using the YOLO algorithm to perform object detection on the extracted feature map to identify the obstacle information in the vehicle-road fusion data;

[0024] The pixel-level fusion method is: projecting the processed in-vehicle point cloud data onto the in-vehicle image plane, and aligning the in-vehicle image data with the projected in-vehicle point cloud data; determining a global coordinate system, and unifying the aligned in-vehicle image data and in-vehicle point cloud data under the global coordinate system; respectively performing pixel-level feature extraction on the in-vehicle image data and in-vehicle point cloud data to obtain in-vehicle image features and in-vehicle point cloud features; then using the superpixel method to extract region-level features from the in-vehicle image data; using the membership regularization fuzzy clustering method to cluster the in-vehicle image features, in-vehicle point cloud features, and region-level features of the in-vehicle image data, and then performing pixel-level fusion on the in-vehicle image features and in-vehicle point cloud features according to the clustering results to obtain in-vehicle fusion data;

[0025] Project the processed roadside point cloud data onto the roadside image plane, align the roadside image data with the projected roadside point cloud data, and unify the aligned roadside image data and roadside point cloud data under the global coordinate system; perform pixel-level feature extraction on the roadside image data and roadside point cloud data respectively to obtain roadside image features and roadside point cloud features; then use the superpixel method to extract region-level features from the roadside image data; use the membership regularization fuzzy clustering method to cluster the roadside image features, roadside point cloud features and region-level features of the roadside image data, and then perform pixel-level fusion on the roadside image features and roadside point cloud features according to the clustering results to obtain roadside fusion data;

[0026] Finally, map all data points in the vehicle-mounted fusion data and roadside fusion data to the global coordinate system to obtain the positions of each data point in the global coordinate system, and obtain vehicle-road fusion data;

[0027] The method of feature-level fusion is as follows: convert the vehicle-mounted point cloud data and vehicle-mounted image data to the same coordinate, perform feature extraction on the vehicle-mounted point cloud data, and identify the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted point cloud data, perform semantic feature extraction on the vehicle-mounted image data to obtain semantic features; then combine the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted fusion data;

[0028] Convert the roadside point cloud data and roadside image data to the same coordinate, perform feature extraction on the roadside point cloud data, and identify the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data, perform semantic feature extraction on the roadside image data to obtain semantic features; then combine the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the roadside fusion data;

[0029] Calculate the overlap degree between different three-dimensional boundaries in the vehicle-mounted fusion data and roadside fusion data, and regard the pedestrians or vehicles corresponding to the two three-dimensional boundaries with the highest overlap degree as the same object and merge them; after completing the merging of all three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted fusion data and roadside fusion data, realize the feature-level data fusion of the vehicle-mounted fusion data and roadside fusion data to obtain vehicle-road fusion data;

[0030] The process of using the YOLO algorithm to perform object detection on the extracted feature map includes the following steps:

[0031] Step A1: Use the YOLO algorithm to perform object detection on the first frame image of the feature map to obtain the bounding boxes and category information of all obstacles in the first frame image;

[0032] Step A2: Use the Kalman filter to predict the position of each obstacle in the current frame respectively, and generate the predicted bounding boxes of all obstacles in the current frame;

[0033] Step A3: Calculate the intersection over union (IoU) between each bounding box in the current frame and each predicted bounding box, and use this IoU to calculate the cost matrix for the current frame;

[0034] Step A4: Based on the cost matrix of the current frame, use the Hungarian algorithm to perform linear matching between the bounding boxes of obstacles and the predicted bounding boxes, and generate the matching results for the current frame;

[0035] The matching states of the linear matching are divided into: uncertain state and confirmed state; where the uncertain state refers to the situation where the bounding box and the predicted bounding box do not achieve perfect matching; the confirmed state refers to the situation where the bounding box and the predicted bounding box achieve perfect matching; the perfect matching means that when the intersection over union of the bounding box and the predicted bounding box meets a preset threshold, the bounding box and the predicted bounding box become a perfect match;

[0036] Step A5: Use the YOLO algorithm to perform object detection on each frame image of the vehicle-road point cloud data except the first frame image, obtain the bounding boxes and class information of each obstacle in each frame image, and perform steps A2 - A4 on each frame image of the vehicle-road point cloud data except the first frame image respectively, count the matching results of each frame in the vehicle-road point cloud data, and delete all bounding boxes that have trajectory mismatches, to obtain all bounding boxes with a matching state of confirmed state in the vehicle-road point cloud data;

[0037] The trajectory mismatch is defined as: for any bounding box, if this bounding box fails to match a predicted bounding box for N consecutive frames, it is considered that this bounding box has a trajectory mismatch;

[0038] Step A6: For any frame image in the vehicle-road point cloud data, cascade and pair all bounding boxes with a matching state of confirmed state in this frame with the predicted bounding boxes, to obtain the cascade pairing results; where the cascade pairing results are divided into: the predicted bounding box and the current frame bounding box do not match at all, the predicted bounding box and the current frame bounding box are paired successfully, and the predicted bounding box and the current frame bounding box have an error;

[0039] Step A7: For all predicted bounding boxes that have an error with the current frame bounding box in the current frame, compare the current frame bounding box with all unmatched predicted bounding boxes in the current frame and calculate the cost matrix, and execute step A4 until all predicted bounding boxes of all obstacles in the current frame and the current frame bounding box are paired successfully;

[0040] Step A8; Repeat step A7 until all frame images in the feature map have completed cascade pairing, and identify all obstacle information in the vehicle-road fusion data;

[0041] The generation process of the matching result of the current frame described in step A4 is as follows: For the bounding box with an uncertain matching state, if the bounding box does not match the predicted bounding box, it indicates that there is a detection mismatch. Then, the Kalman filter is reused to generate a new predicted bounding box, and the cost matrix of the current frame is recalculated based on the bounding box in the current frame and the new predicted bounding box. Based on the new cost matrix, the Hungarian algorithm is used to linearly match the bounding box of the obstacle and the new predicted bounding box until the matching state of the bounding box becomes confirmed or the preset number of matching times is reached; Count all the bounding boxes with a confirmed matching state and all the unmatched bounding boxes in the current frame;

[0042] The process of using the vortex artificial potential field APFV algorithm to perform path planning for the obstacle avoidance vehicle includes the following steps:

[0043] Step B1: Classify all the identified obstacles into primary obstacle avoidance targets and secondary obstacle avoidance targets;

[0044] The primary obstacle avoidance targets include: normal driving obstacles and emergency obstacle avoidance obstacles; Among them, the normal driving obstacle is: an obstacle driving in the same lane as the obstacle avoidance vehicle in the same direction; The emergency obstacle avoidance obstacle is: an obstacle approaching the obstacle avoidance vehicle at a relative speed exceeding the maximum set threshold;

[0045] The secondary obstacle avoidance targets are the obstacles other than the primary obstacle avoidance targets;

[0046] Step B2: Set the secondary obstacle avoidance targets as the obstacle avoidance environment, obtain the current position and destination of the obstacle avoidance vehicle, and based on the primary obstacle avoidance targets, use the artificial potential field method to plan the initial obstacle avoidance path of the obstacle avoidance vehicle;

[0047] Step B3: The obstacle avoidance vehicle travels along the initial obstacle avoidance path. During the travel of the obstacle avoidance vehicle, if the primary obstacle avoidance target moves forward relative to the obstacle avoidance vehicle, the obstacle avoidance vehicle stops; Taking the current stop position as the starting point and the destination as the end point, adjust the travel direction of the obstacle avoidance vehicle according to the rotation angle of the obstacle avoidance vehicle, and use the artificial potential field method to re-plan the obstacle avoidance path for the obstacle avoidance vehicle until an obstacle-free obstacle avoidance path is generated; At this time, the obstacle avoidance vehicle travels along this obstacle-free obstacle avoidance path;

[0048] If the primary obstacle avoidance target and the obstacle avoidance vehicle are moving towards each other, calculate the relative speed between the obstacle avoidance vehicle and the primary obstacle avoidance target. If the relative speed is negative, adjust the travel speed of the obstacle avoidance vehicle so that the travel speed of the obstacle avoidance vehicle matches the speed of the primary obstacle avoidance target to ensure that the distance between the obstacle avoidance vehicle and the primary obstacle avoidance target is always greater than the set safety distance threshold;

[0049] If there is no main obstacle avoidance target on the obstacle avoidance path of the obstacle avoidance vehicle, the obstacle avoidance path is simplified into a polyline, and it is ensured that each segment of the polyline is parallel to the lane line, and the turning angle of the polyline meets the preset vehicle direction adjustment requirements, so as to obtain the final obstacle avoidance path of the obstacle avoidance vehicle, and then generate the driving direction and driving speed of the obstacle avoidance vehicle;

[0050] Step B2 further includes:

[0051] Step B2.1: Initialize parameters, including: gravitational coefficient, repulsive coefficient, vortex parameter, coordinates of the destination, and coordinates of the main obstacle avoidance target;

[0052] Step B2.2: Calculate the attractive potential field pointing to the destination and the repulsive potential field avoiding the main obstacle avoidance target respectively;

[0053] Step B2.3: Update the direction of the repulsive potential field using the vortex potential field;

[0054] Step B2.4: Calculate the resultant force using the attractive potential field and the updated repulsive potential field and update the position of the obstacle avoidance vehicle;

[0055] Step B2.5: Based on the updated position of the obstacle avoidance vehicle, repeat steps B2.2 - 2.4 until the obstacle avoidance vehicle reaches the destination, and plan the initial obstacle avoidance path of the obstacle avoidance vehicle according to all the generated positions of the obstacle avoidance vehicle.

[0056] The beneficial effects produced by adopting the above technical solutions are as follows:

[0057] The method of the present invention proposes an optimized artificial potential field method applicable to vehicle driving, namely the APFV obstacle avoidance algorithm, which is used to provide obstacle avoidance path planning for vehicles. The method of the present invention makes up for the problem that the path planned by the traditional artificial potential field method does not conform to traffic rules by adding traffic rules to the traditional artificial potential field method, so as to provide obstacle avoidance path planning for vehicles.

[0058] Aiming at the problem of the planning range, the method of the present invention designs a new path planning algorithm of the vehicle-road collaborative artificial potential field method. By obtaining the current road information through the existing intelligent transportation system, the feasible lane area is used to replace the traditional entire plane, so as to avoid problems such as reverse driving in path planning.

[0059] Aiming at the rationality problem of path planning, on the basis of the above content, the path planned by the artificial potential field method is improved. The path planned by the artificial potential field method is divided into points, and the turning points are found and connected into a polyline, and it is ensured that the part of the planned path without changing the direction is relatively parallel to the lane line.

[0060] Regarding the problem of the planning step size, based on the above content, the step size of the artificial potential field method is improved and changed to the relative speed between the vehicle itself and the target obstacle. And a new artificial potential field method calculation is carried out every 0.1 s, and only the next position point is output as the instruction.

[0061] In order to extract obstacle information from the radar point cloud and camera information transmitted by the sensor, the method of the present invention uses the fusion of radar perception and the YOLO algorithm to obtain obstacle information. The traditional YOLO algorithm mainly performs object detection based on image data. Although it can provide the two-dimensional bounding box information of the object, it has limitations in depth information and three-dimensional space positioning. However, due to the addition of the radar point cloud, a new vehicle-road collaborative YOLO perception algorithm proposed by the method of the present invention makes up for the problem of poor position information extraction ability of the YOLO algorithm.

[0062] In order to expand the perception range, the method of the present invention uses vehicle-road cooperation in autonomous driving, so that the vehicle can obtain the information sensed by the lidar at the far end of the road, and thus the perception range of the vehicle can break through the limitation of the sensor and reach farther places. The expansion of the perception range can, to a certain extent, solve the problem of traffic jams, that is, the vehicle can sense the traffic flow on the road ahead in advance, so as to form more effective traffic jam strategies such as lane changes.

[0063] In order to supplement the blind area, for autonomous driving with vehicle-road cooperation, the road-side equipment and the vehicle in the method of the present invention can sense objects from different angles, supplement and broaden the vehicle perception information, reduce the visual blind area, and eliminate potential hazards.

[0064] In order to improve the driving accuracy, for autonomous driving with vehicle-road cooperation, the method of the present invention can view objects from multiple angles, so as to obtain more information and more accurate estimated position and attitude.

[0065] In summary, the method of the present invention designs an optimized artificial potential field method suitable for vehicle driving, namely the APFV algorithm, a new vehicle-road collaborative YOLO perception algorithm, and a new vehicle-road collaborative artificial potential field method path planning algorithm, and can also be used to implement a vehicle collaborative intelligent perception system. The method of the present invention can not only realize the intelligent perception of vehicles and pedestrians, but also prove the feasibility and safety of the method by expanding the perception range, supplementing the perception blind area, and improving the perception accuracy. This also proves that vehicle-road cooperation is superior to individual vehicle autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flowchart of a vehicle-road collaborative intelligent perception method in this embodiment;

[0067] Figure 2Schematic diagram of a vehicle-road collaborative intelligent perception method in this embodiment;

[0068] Figure 3 Schematic diagram of ROS-CARLA joint simulation in this embodiment;

[0069] Figure 4 Result diagram of CARLA simulation in this embodiment; among them, (a) is the obstacle avoidance path diagram of the simulation; (b) is the schematic diagram of the simulation of the obstacle avoidance vehicle driving on a curve; (c) is the schematic diagram of the simulation of the obstacle avoidance vehicle driving on a straight road. Specific implementation mode

[0070] To facilitate the understanding of this application, the following combines the drawings and embodiments to further describe in detail the specific implementation mode of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure content of this application more thoroughly and comprehensively understood.

[0071] A vehicle-road collaborative intelligent perception method in this embodiment, as Figure 1 shown, this method includes the following processes:

[0072] For any road scene, this road scene includes: lanes, obstacle avoidance vehicles and obstacles; among them, the obstacles include: vehicles and pedestrians; roadside sensors are set on the lanes, and on-vehicle sensors are set on the obstacle avoidance vehicles.

[0073] The on-vehicle sensors include: on-vehicle lidar, on-vehicle camera and Global Navigation Satellite System (GNSS) module.

[0074] The on-vehicle lidar is used to obtain the radar point cloud data of the lanes and obstacles during the driving of the obstacle avoidance vehicle in real time by emitting laser beams and receiving reflected signals.

[0075] The on-vehicle camera is used to obtain the image data of the lanes and obstacles during the driving of the obstacle avoidance vehicle in real time.

[0076] The Global Navigation Satellite System GNSS module is used to obtain the position of the obstacle avoidance vehicle and calculate the rotation angle of the obstacle avoidance vehicle.

[0077] The roadside sensors include: roadside lidar and roadside camera.

[0078] The roadside lidar is used to obtain the radar point cloud data of the obstacle avoidance vehicle and obstacles on the lane in real time.

[0079] The roadside camera is used to obtain the image data of obstacle avoidance vehicles and obstacles on the lane in real time.

[0080] In this embodiment, as Figure 2 shown, in order to sense the environment, sensors are set to obtain environmental information. However, due to hardware limitations, a real physical camera, radar, or actual vehicle was not used during the research process of this embodiment. Instead, a vehicle-road collaborative intelligent perception simulation environment was built using the Ros-Carla joint simulation platform. That is, the PDC point cloud library was used to simulate the training of radar point cloud data, and the Carla road video was recorded by ourselves to train the model parameters. Then, combined with Carla and ROS for joint simulation to simulate the vehicle-road collaborative intelligent perception results and evaluate the algorithm. The algorithms in this embodiment were implemented using C language, MATLAB, and Python.

[0081] The Ros-Carla joint simulation is a technology that combines the Robot Operating System (ROS) with an open-source autonomous driving simulation platform Carla, enabling the testing and development of autonomous driving algorithms in a virtual environment. The specific process of building a vehicle-road collaborative intelligent perception simulation environment using the Ros-Carla joint simulation platform in this embodiment is as follows: Initialize the test environment. Load the map and 3D vehicles on the Carla platform and add obstacles. In this embodiment, an obstacle avoidance vehicle driving on a curved road was designed, and a roadside lidar was installed on one side of the curve to sense the traffic flow at both ends of the road. Start the status function package, ROS nodes, and the perception obstacle avoidance function package in the Ros-Carla joint simulation platform, and connect ROS and Carla through the ROS-Carla-bridge. On the Carla platform, an on-vehicle lidar and an on-vehicle camera are set on the top of the obstacle avoidance vehicle to sense pedestrians and other vehicles. They are installed on the top of the vehicle, 2 meters above the ground, and the sensing range is set to 20 meters. A roadside lidar is set at the end of the road, installed above the intersection, 3 meters above the ground, with a wider sensing range, up to 40 meters at most. Additional roadside cameras are installed at high points on both sides of the road, such as the roof, and the additional roadside cameras face forward to extract lane line information. A Global Navigation Satellite System (GNSS) module is respectively provided at the front and rear ends of the obstacle avoidance vehicle to obtain the vehicle position and calculate the rotation angle of the obstacle avoidance vehicle.

[0082] Collect the roadside point cloud data and roadside image data of the lane using roadside sensors, collect the on-vehicle point cloud data and on-vehicle image data of the obstacle avoidance vehicle using on-vehicle sensors, and save the roadside point cloud data and on-vehicle point cloud data in the PCD format.

[0083] In this embodiment, both the vehicle-mounted lidar and the roadside lidar use the VLP64 lidar. To better utilize the VLP64 lidar deployed on vehicles and roads, this embodiment adopts a new method, that is, storing the point cloud data collected by the VLP64 lidar at a certain moment in PCD format, including the Cartesian coordinates of the point cloud and the corresponding intensity values, to more accurately capture the environmental information at the end of a scan. This method allows the PCD file to store up to 256,000 (x, y, z, i) point cloud information, where x is the abscissa; y is the ordinate; z is the vertical coordinate; i is the intensity value.

[0084] Data preprocessing, data screening, and data repair are performed on the roadside point cloud data and the vehicle-mounted point cloud data in sequence to obtain the processed roadside point cloud data and vehicle-mounted point cloud data.

[0085] In this embodiment, data preprocessing refers to operations such as cleaning, deduplicating, and converting the format of the point cloud data during the processes of data acquisition, storage, and reading to ensure data quality and consistency. Data screening refers to screening out abnormal data, such as invalid data and unreasonable data, by setting thresholds or rules for subsequent processing and analysis. Data repair refers to repairing data missing or abnormal situations through methods such as interpolation and smoothing to ensure data integrity and accuracy.

[0086] The YOLO intelligent perception fusion algorithm is used to perform object detection on the processed vehicle-mounted point cloud data, vehicle-mounted image data, processed roadside point cloud data, and roadside image data to identify obstacle information.

[0087] In this embodiment, the traditional Yolo algorithm is for object detection of image data. Although it can provide two-dimensional bounding box information of the object, it has limitations in depth information and three-dimensional space positioning. In this embodiment, by fusing the vehicle-mounted image data, vehicle-mounted point cloud data, roadside image data, and roadside point cloud data, the fused vehicle-road fusion data can contain information about obstacle vehicles and obstacle pedestrians on the road. The most important cooperation method in the vehicle-road cooperation scheme is the sharing and fusion of vehicle-road perception data. Therefore, after obtaining the data at the vehicle end and the road end, this embodiment designs two data fusion schemes to fuse the data, and then uses the fused data for planning.

[0088] The process of using the YOLO intelligent perception fusion algorithm to perform object detection on the processed vehicle-mounted point cloud data, vehicle-mounted image data, processed roadside point cloud data, and roadside image data includes:

[0089] The processed vehicle-borne point cloud data, vehicle-borne image data, processed roadside point cloud data, and roadside image data are fused using pixel-level fusion or feature-level fusion methods to obtain vehicle-road fusion data.

[0090] The pixel-level fusion method is as follows: project the processed vehicle-borne point cloud data onto the vehicle-borne image plane, and align the vehicle-borne image data with the projected vehicle-borne point cloud data; determine a global coordinate system, and unify the aligned vehicle-borne image data and vehicle-borne point cloud data under the global coordinate system; perform pixel-level feature extraction on the vehicle-borne image data and vehicle-borne point cloud data respectively to obtain vehicle-borne image features and vehicle-borne point cloud features; then use the superpixel method to extract region-level features from the vehicle-borne image data; use the membership regularization fuzzy clustering method to cluster the vehicle-borne image features, vehicle-borne point cloud features, and region-level features of the vehicle-borne image data, and then perform pixel-level fusion on the vehicle-borne image features and vehicle-borne point cloud features according to the clustering results to obtain vehicle-borne fusion data.

[0091] Similarly, project the processed roadside point cloud data onto the roadside image plane, and align the roadside image data with the projected roadside point cloud data, and unify the aligned roadside image data and roadside point cloud data under the global coordinate system; perform pixel-level feature extraction on the roadside image data and roadside point cloud data respectively to obtain roadside image features and roadside point cloud features; then use the superpixel method to extract region-level features from the roadside image data; use the membership regularization fuzzy clustering method to cluster the roadside image features, roadside point cloud features, and region-level features of the roadside image data, and then perform pixel-level fusion on the roadside image features and roadside point cloud features according to the clustering results to obtain roadside fusion data.

[0092] Finally, map all data points in the vehicle-borne fusion data and roadside fusion data to the global coordinate system to obtain the positions of each data point in the global coordinate system, and obtain the vehicle-road fusion data.

[0093] In this embodiment, pixel-level fusion refers to directly sharing and merging the raw data of vehicle-side and road-side sensors and mapping each data point to the same space. By combining the depth information of the point cloud with the semantic information of the image, pixel-level fusion can significantly improve the performance of object detection and segmentation. By introducing a membership-regularized fuzzy clustering method for pixel and region level information fusion in image segmentation, the segmentation result can be further optimized. For the point cloud data obtained by lidar, each data point has three-dimensional coordinate information. To perform pixel-level fusion on the point cloud data of road-side radar and vehicle-side radar, what this embodiment needs to do is to unify the coordinate systems of the logarithmic points. The data fused at the pixel level requires special processing methods. Different from the lidar point cloud of a single vehicle, the fused point cloud in this embodiment has two sources, and the information therein cannot be well utilized by conventional object detection methods. Therefore, in order to take advantage of pixel-level fusion, a new object detection model needs to be proposed, that is, to assist the original target perception only through the camera with pixel-level fusion. Pixel-level fusion has the advantage of high accuracy. However, its disadvantage is that the amount of data to be processed is large, and it is difficult to fuse data from different types of sensors.

[0094] The method of the feature-level fusion is as follows: convert the vehicle-mounted point cloud data and the vehicle-mounted image data to the same coordinate, extract features from the vehicle-mounted point cloud data, and identify the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted point cloud data, extract semantic features from the vehicle-mounted image data to obtain semantic features; then combine the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted fusion data.

[0095] Similarly, convert the roadside point cloud data and the roadside image data to the same coordinate, extract features from the roadside point cloud data, and identify the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data, extract semantic features from the roadside image data to obtain semantic features; then combine the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the roadside fusion data.

[0096] Calculate the overlap degree between different three-dimensional boundaries in the vehicle-mounted fusion data and the roadside fusion data, regard the pedestrians or vehicles corresponding to the two three-dimensional boundaries with the highest overlap degree as the same object and merge them; after completing the merging of all three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted fusion data and the roadside fusion data, realize the feature-level data fusion of the vehicle-mounted fusion data and the roadside fusion data to obtain vehicle-road fusion data.

[0097] In this embodiment, feature-level fusion is achieved by separately processing sensor data and sharing and combining the extracted features. For the three-dimensional boundaries of the identified pedestrians and vehicles, each boundary consists of 8 coordinate points. To perform feature-level fusion on the three-dimensional boundaries of pedestrians and vehicles identified at the road end and vehicle end, what this embodiment needs to do is to unify the coordinate systems of the boundaries. The boundaries of the same type of object with a high degree of overlap are regarded as the same object and unified into one object, which is convenient for subsequent extraction of obstacle information using the YOLO intelligent perception fusion algorithm. The fusion data at the feature level can be easily used in existing autonomous driving frameworks. After fusing the boundaries obtained by three-dimensional object detection of point cloud data, the quantity and quality of the perceived boundaries can be improved, and there will not be too much change in subsequent processing methods. Therefore, feature-level fusion should be easily used by autonomous driving systems.

[0098] Use a fully connected layer to extract features from the vehicle-road fusion data, and then use the YOLO algorithm to perform object detection on the extracted feature map to identify the obstacle information in the vehicle-road fusion data.

[0099] In this embodiment, the radar perception and the YOLO algorithm are fused, and the YOLO algorithm is used for perception fusion to obtain obstacle information, and the YOLO algorithm trains the obtained obstacle information into a 3D model.

[0100] The process of using the YOLO algorithm to perform object detection on the extracted feature map includes the following steps:

[0101] Step A1: Use the YOLO algorithm to perform object detection on the first frame image of the feature map to obtain the bounding boxes and class information of all obstacles in the first frame image.

[0102] Step A2: Use the Kalman filter to predict the position of each obstacle in the current frame respectively, and generate the predicted bounding boxes of all obstacles in the current frame.

[0103] In this embodiment, in the first frame detection, the Kalman filter can be used to predict the position of the obstacles, and the corresponding predicted bounding boxes can be generated according to these prediction results, which can improve the prediction accuracy.

[0104] Step A3: Calculate the intersection over union (IoU) of each bounding box and each predicted bounding box in the current frame, and use this IoU to calculate the cost matrix of the current frame.

[0105] In this embodiment, the bounding box of the obstacle is compared with the predicted bounding box, that is, the intersection over union (IoU) of the bounding box and the predicted bounding box is calculated. The IoU is an index to measure the overlapping degree of two bounding boxes, and the calculation method is the ratio of the intersection area to the union area of the two bounding boxes. The expression of each element in the defined cost matrix is defined as 1 - IoU to determine the target of the current frame.

[0106] Step A4: Based on the cost matrix of the current frame, the Hungarian algorithm is used to linearly match the bounding box of the obstacle and the predicted bounding box to generate the matching result of the current frame.

[0107] The matching states of the linear matching are divided into: an uncertain state and a confirmed state; where the uncertain state refers to the situation where the bounding box and the predicted bounding box do not achieve a perfect match; the confirmed state refers to the situation where the bounding box and the predicted bounding box achieve a perfect match; the perfect match means that when the IoU of the bounding box and the predicted bounding box meets a preset threshold, the bounding box and the predicted bounding box become a perfect match.

[0108] The generation process of the matching result of the current frame is as follows: for the bounding box with an uncertain matching state, if the bounding box and the predicted bounding box do not match, it indicates that there is a detection mismatch. A new predicted bounding box is regenerated by using the Kalman filter, and the cost matrix of the current frame is recalculated based on the bounding box in the current frame and the new predicted bounding box. Based on the new cost matrix, the Hungarian algorithm is used to linearly match the bounding box of the obstacle and the new predicted bounding box until the matching state of the bounding box becomes a confirmed state or the preset number of matching times is reached; count all the bounding boxes with a confirmed matching state in the current frame and all the non-matching bounding boxes in the current frame.

[0109] Step A5: The YOLO algorithm is used to perform object detection on each frame of the vehicle-road point cloud data except the first frame image, to obtain the bounding box and category information of each obstacle in each frame image, and steps A2 - A4 are respectively executed for each frame of the vehicle-road point cloud data except the first frame image. The matching results of each frame in the vehicle-road point cloud data are counted, and all the bounding boxes with trajectory mismatches are deleted to obtain all the bounding boxes with a confirmed matching state in the vehicle-road point cloud data.

[0110] The trajectory mismatch is as follows: for any bounding box, if the bounding box fails to match the predicted bounding box for N consecutive frames, it is considered that the bounding box has a trajectory mismatch.

[0111] In this embodiment, according to the cost matrix, the Hungarian algorithm is used to linearly match the bounding boxes predicted by the motion trajectory with the bounding boxes of the current frame to delete the bounding boxes. In this process, the situation where the two do not achieve a perfect match is called the uncertain state, and the state of achieving a perfect match is called the confirmed state. After detection, three different situations will finally occur: The first situation is called Unmatched Tracks, that is, the predicted bounding box does not match the current frame. In the uncertain state, if the matching fails for 30 consecutive frames, the bounding box is deleted; The second situation is called Unmatched-Detections, that is, the predicted bounding box cannot match the bounding box of the current frame. In the uncertain state, the Kalman filter is used again for prediction to form a new predicted bounding box, and then the linear matching is used to delete the bounding box again until the confirmed state is reached, and then the processing can be stopped. In the third situation, if the detected box perfectly matches the predicted box, it means that the previous frame and the subsequent frame have been effectively tracked, that is, the confirmed state is reached.

[0112] In this embodiment, the Kalman filter can be used to estimate the state of the object corresponding to the predicted bounding box, and can be used to analyze their appearance features, motion information and other features. To achieve this goal, it is necessary to first save the appearance features of the first 100 frames, and then use these features to associate with the appearance features, motion information and other features of the obstacle corresponding to the predicted bounding box. Finally, the Kalman filter is used to estimate their state, so as to realize the recognition of the obstacle. Since the similarity between the bounding box of the current frame in the confirmed state and the predicted bounding box is relatively high, corresponding measures are taken.

[0113] Step A6: For any frame image in the vehicle-road point cloud data, cascade and pair the bounding boxes with the matching state of the confirmed state in this frame with the predicted bounding boxes to obtain the cascade pairing result; The cascade pairing result is divided into: the predicted bounding box and the current frame bounding box do not match at all, the predicted bounding box and the current frame bounding box are paired successfully, and the predicted bounding box and the current frame bounding box have an error.

[0114] In this embodiment, through the cascaded pairing of the predicted bounding box and the current frame bounding box in the confirmed state, three different conclusions will be obtained. First, when the predicted bounding box and the current frame bounding box are successfully paired, the Kalman filter is used to detect the Tracks variable, and the same conclusion as in step A3 will also be obtained. Second, when there is an error between the predicted bounding box and the current frame bounding box, it is caused by the fast movement, occlusion, or detection error of the target; at this time, the current frame bounding box is compared with the unpaired predicted bounding boxes one by one to estimate the cost matrix. Third, there is a complete mismatch, due to target loss, new target appearance, or detection failure. If the predicted bounding box does not match any bounding box in multiple consecutive frames, it may be considered that the target has been lost and the tracking of the target is stopped. For the new bounding box in the current frame that does not match the predicted bounding box, a new Kalman filter is initialized to start tracking the new target.

[0115] Step A7: For all the predicted bounding boxes that have an error with the current frame bounding box in the current frame, the current frame bounding box is respectively compared with all the unpaired predicted bounding boxes in the current frame to calculate the cost matrix, and step A4 is executed until all the predicted bounding boxes and the current frame bounding boxes of all the obstacles in the current frame are successfully paired.

[0116] Step A8; Repeat step A7 until the cascaded pairing of all the frame images in the feature map is completed, and all the obstacle information in the vehicle-road fusion data is recognized.

[0117] According to the recognized obstacle information, the vortex artificial potential field APFV algorithm is used to perform path planning for the obstacle avoidance vehicle, and the driving direction and driving speed of the obstacle avoidance vehicle are generated, so that the obstacle avoidance vehicle performs autonomous driving according to the generated driving direction and driving speed.

[0118] The process of using the vortex artificial potential field APFV algorithm to perform path planning for the obstacle avoidance vehicle includes the following steps:

[0119] Step B1: Divide all the recognized obstacles into main obstacle avoidance targets and secondary obstacle avoidance targets.

[0120] The main obstacle avoidance targets include: normal driving obstacles and emergency obstacle avoidance obstacles; where the normal driving obstacles are: obstacles driving in the same lane as the obstacle avoidance vehicle in the same direction; the emergency obstacle avoidance obstacles are: obstacles approaching the obstacle avoidance vehicle with a relative speed exceeding the maximum set threshold.

[0121] The secondary obstacle avoidance targets are the obstacles other than the main obstacle avoidance targets.

[0122] In this embodiment, the main obstacle avoidance target is selected and the secondary target is set as the obstacle avoidance environment instead of the main obstacle avoidance target. During vehicle driving, the main obstacle avoidance targets can be divided into two types: normal driving obstacle avoidance obstacles, i.e., obstacles driving in the same lane as the vehicle, and emergency obstacle avoidance, i.e., the target obstacle vehicle approaching at a high relative speed, which can come from any direction. Since the obstacle avoidance behavior may cause the host vehicle to change its route and affect vehicles that originally had no interaction, such as vehicles driving parallel in the same direction. Therefore, in this embodiment, the main obstacle avoidance vehicle is taken as the target in the APFV algorithm, and at the same time, the secondary obstacle vehicle and its expected driving route are regarded as the feasible planning range.

[0123] Step B2: Set the secondary obstacle avoidance target as the obstacle avoidance environment, obtain the current position and destination of the obstacle avoidance vehicle, and based on the main obstacle avoidance target, use the artificial potential field method to plan the initial obstacle avoidance path of the obstacle avoidance vehicle.

[0124] Step B2.1: Initialize the parameters, including: gravitational coefficient, repulsive coefficient, vortex parameter, coordinates of the destination, and coordinates of the main obstacle avoidance target.

[0125] Step B2.2: Calculate the attractive potential field for pointing to the destination and the repulsive potential field for avoiding the main obstacle avoidance target respectively.

[0126] In this embodiment, the gravitational force in the attractive potential field points to the target point, and its magnitude is proportional to the distance between the obstacle avoidance vehicle and the destination; the repulsive force in the repulsive potential field points in the direction away from the obstacle, and its magnitude is inversely proportional to the distance between the obstacle avoidance vehicle and the main obstacle avoidance target. It should be noted that when the distance between the obstacle avoidance vehicle and the main obstacle avoidance target is less than the set threshold, the repulsive force is calculated; otherwise, the repulsive force is zero.

[0127] Step B2.3: Update the direction of the repulsive potential field using the vortex potential field.

[0128] In this embodiment, the vortex potential field is realized by rotating the direction of the repulsive force, and the rotation angle is dynamically adjusted according to the relative position between the obstacle avoidance vehicle and the main obstacle avoidance target. When the robot is too close to the obstacle, the vortex potential field guides the robot to bypass the obstacle. When the obstacle avoidance vehicle falls into a local minimum, the vortex potential field will guide the obstacle avoidance vehicle to bypass the main obstacle avoidance target.

[0129] Step B2.4: Calculate the resultant force using the gravitational force and the updated repulsive force and update the position of the obstacle avoidance vehicle.

[0130] Step B2.5: Based on the updated position of the obstacle avoidance vehicle, repeat steps B2.2 - 2.4 until the obstacle avoidance vehicle reaches the destination, and plan the initial obstacle avoidance path of the obstacle avoidance vehicle according to all the generated positions of the obstacle avoidance vehicle.

[0131] Step B3: The obstacle avoidance vehicle travels along the initial obstacle avoidance path. During the driving process of the obstacle avoidance vehicle, if the main obstacle avoidance target moves forward relative to the obstacle avoidance vehicle, the obstacle avoidance vehicle stops; taking the current stop position as the starting point and the destination as the end point, adjust the driving direction of the obstacle avoidance vehicle according to the rotation angle of the obstacle avoidance vehicle, and use the artificial potential field method to re-plan the obstacle avoidance path for the obstacle avoidance vehicle until an obstacle-free obstacle avoidance path is generated; at this time, the obstacle avoidance vehicle travels along this obstacle-free obstacle avoidance path.

[0132] If the main obstacle avoidance target moves towards the obstacle avoidance vehicle relatively, calculate the relative speed between the obstacle avoidance vehicle and the main obstacle avoidance target. If the relative speed is negative, adjust the driving speed of the obstacle avoidance vehicle so that the driving speed of the obstacle avoidance vehicle matches the speed of the main obstacle avoidance target, to ensure that the distance between the obstacle avoidance vehicle and the main obstacle avoidance target is always greater than the set safety distance threshold.

[0133] If there is no main obstacle avoidance target on the obstacle avoidance path of the obstacle avoidance vehicle, simplify the obstacle avoidance path into a polyline, and ensure that each section of the polyline is parallel to the lane line, and the turning corners of the polyline meet the preset vehicle direction adjustment requirements, to obtain the final obstacle avoidance path of the obstacle avoidance vehicle, and then generate the driving direction and driving speed of the obstacle avoidance vehicle.

[0134] In this embodiment, it is judged whether the current obstacle avoidance environment meets the obstacle avoidance algorithm. First, use the traditional artificial potential field method to plan a curve connecting the current position and the target point. If this curve intersects any obstacle vehicle, execute the following instructions: If the host vehicle and the main obstacle avoidance vehicle move forward relatively, issue a stop command. Here, relative forward means that the host vehicle and the main obstacle avoidance vehicle are driving in the same direction, and the host vehicle is behind and the obstacle avoidance vehicle is in front. Taking the current stop position as the starting point and the target point as the predetermined destination, recalculate the attractive potential field and the repulsive potential field. According to the updated potential field, calculate a new safe path and determine the moving direction. If an obstacle-free path can be found, it can be restarted. In the above process, the artificial potential field method is always in the startup state. If the host vehicle and the main obstacle avoidance vehicle move towards each other relatively, issue an equal-speed following command, that is, drive at the same speed as the target vehicle, to prevent the host vehicle from stopping in the middle of the road due to obstacle avoidance. If the curve has no intersection with the obstacle vehicle, simplify the curve into a polyline, try to keep it parallel to the lane line, and the turning corners of the polyline meet the preset vehicle direction adjustment requirements, that is, according to the actual situation, reserve enough space for the direction adjustment part of the obstacle avoidance vehicle before performing polyline fitting.

[0135] In this embodiment, ROS-Carla joint simulation is adopted, as Figure 3 and Figure 4As shown, the intelligent perception algorithm and path planning algorithm are run in the ROS environment, and the vehicle motion information is output to Carla for 3D simulation demonstration. The map and 3D vehicle are loaded on the Carla platform, and obstacle avoidance vehicles and obstacles are added to complete the initialization of the algorithm test environment. The ROS node and the perception obstacle avoidance function package are started, and the ROS-Carla-bridge is started to connect ROS and Carla. Cameras and perception radars are manually added to the road end and vehicle end on the Carla platform, and the YOLO intelligent perception fusion algorithm is called for sensor recognition. Under the current simulation environment parameter settings, it is determined whether the virtual camera recognition is effective. If the recognition is ineffective or the recognition effect is not good, the virtual camera recognition model parameters are adjusted; if the recognition effect is good, the next step is entered. In the ROS environment, the function packages of the APFV algorithm and the intelligent perception algorithm are started. The intelligent perception detects obstacle information, that is, the vehicle and pedestrian obstacle information are obtained from the YOLO intelligent perception fusion algorithm. During the simulation of the APFV algorithm, although it is expected to use a more complex virtual map for the 3D model, for parameter adjustment, the recognition test starts from a simple map. Therefore, in this embodiment, the recognition test starts from a map with a single obstacle. If successful, one more obstacle is added, and then the YOLO intelligent perception fusion algorithm is called again for sensor recognition, and the above process is repeated until there are obstacle vehicles in all three directions of the vehicle on the map, that is, to ensure that the vehicle has a greater chance of getting out of the obstacle avoidance path. Finally, the vehicle is started for obstacle avoidance simulation.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present invention.

Claims

1. A vehicle-road collaboration intelligent perception method, characterized in that: The method includes the following steps: For any road scene, the road scene includes: a lane, an obstacle avoidance vehicle and an obstacle; wherein the obstacle includes: a vehicle and a pedestrian; a roadside sensor is arranged on the lane, and an onboard sensor is arranged on the obstacle avoidance vehicle; The roadside point cloud data and roadside image data of the lane are collected by using the roadside sensor, and the vehicle-mounted point cloud data and vehicle-mounted image data of the obstacle avoidance vehicle are collected by using the vehicle-mounted sensor, and the roadside point cloud data and vehicle-mounted point cloud data are saved in PCD format; The roadside point cloud data and the vehicle-mounted point cloud data are respectively subjected to data preprocessing, data screening and data repair in sequence to obtain processed roadside point cloud data and vehicle-mounted point cloud data; The YOLO intelligent perception fusion algorithm is used to detect targets on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data to identify obstacle information; According to the identified obstacle information, the vortex artificial potential field APFV algorithm is used to plan the path for the obstacle avoidance vehicle and generate the driving direction and speed of the obstacle avoidance vehicle, so that the obstacle avoidance vehicle can automatically drive according to the generated driving direction and speed.

2. According to claim 1, the vehicle-road collaboration intelligent perception method is characterized by: The vehicle-mounted sensors include: a vehicle-mounted laser radar, a vehicle-mounted camera and a global navigation satellite system GNSS module; The vehicle-mounted laser radar is used to acquire radar point cloud data of lanes and obstacles in real time during the driving process of the obstacle avoidance vehicle by emitting laser beams and receiving reflected signals; The vehicle-mounted camera is used to obtain real-time image data of lanes and obstacles during the driving process of the obstacle avoidance vehicle; The global navigation satellite system GNSS module is used to obtain the position of the obstacle avoidance vehicle and calculate the rotation angle of the obstacle avoidance vehicle; The roadside sensor includes: a roadside laser radar and a roadside camera; The roadside laser radar is used to obtain radar point cloud data of obstacle avoidance vehicles and obstacles on the lane in real time; The roadside camera is used to obtain image data of obstacle avoidance vehicles and obstacles on the lane in real time.

3. According to claim 2, a vehicle-road cooperative intelligent perception method is characterized in that: The saving of the roadside point cloud data and the vehicle-mounted point cloud data in the PCD format is expressed as follows: each point cloud data in the roadside point cloud data and the vehicle-mounted point cloud data includes the Cartesian coordinates of the point cloud and the intensity value of the point cloud, expressed as: (x, y, z, i), wherein x is the horizontal coordinate; y is the vertical coordinate; z is the vertical coordinate; and i is the intensity value.

4. According to claim 3, the vehicle-road cooperative intelligent perception method is characterized in that: The process of using the YOLO intelligent perception fusion algorithm to perform target detection on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data includes: The processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data are fused by using pixel-level fusion or feature-level fusion methods to obtain vehicle-road fusion data; The fully connected layer is used to extract features from the vehicle-road fusion data, and then the YOLO algorithm is used to perform target detection on the extracted feature map to identify obstacle information in the vehicle-road fusion data.

5. According to claim 4, the vehicle-road cooperative intelligent perception method is characterized in that: The pixel-level fusion method is: projecting the processed vehicle point cloud data onto the vehicle image plane, and aligning the vehicle image data with the projected vehicle point cloud data; Determine a global coordinate system, and unify the aligned vehicle image data and vehicle point cloud data into the global coordinate system; perform pixel-level feature extraction on the vehicle image data and vehicle point cloud data respectively to obtain vehicle image features and vehicle point cloud features; then use the superpixel method to extract regional features from the vehicle image data; use the membership regularization fuzzy clustering method to cluster the vehicle image features, vehicle point cloud features and regional features of the vehicle image data, and then perform pixel-level fusion of the vehicle image features and vehicle point cloud features based on the clustering results to obtain vehicle fusion data; Projecting the processed roadside point cloud data onto a roadside image plane, aligning the roadside image data with the projected roadside point cloud data, and unifying the aligned roadside image data and roadside point cloud data into a global coordinate system; Pixel-level feature extraction is performed on the roadside image data and the roadside point cloud data respectively to obtain roadside image features and roadside point cloud features; Then, the superpixel method is used to extract regional features from the roadside image data; the membership regularization fuzzy clustering method is used to cluster the roadside image features, roadside point cloud features and regional features of the roadside image data, and then the roadside image features and roadside point cloud features are fused at the pixel level according to the clustering results to obtain the roadside fusion data; Finally, all data points in the vehicle-mounted fusion data and the roadside fusion data are mapped to the global coordinate system to obtain the position of each data point in the global coordinate system and obtain the vehicle-road fusion data.

6. The vehicle-road collaboration intelligent perception method according to claim 5, characterized in that: The feature-level fusion method is as follows: converting the vehicle-mounted point cloud data and the vehicle-mounted image data to the same coordinate, performing feature extraction on the vehicle-mounted point cloud data, and identifying the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted point cloud data, performing semantic feature extraction on the vehicle-mounted image data to obtain semantic features; then combining the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the vehicle-mounted fusion data; The roadside point cloud data and the roadside image data are converted to the same coordinates, feature extraction is performed on the roadside point cloud data, and the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data are identified, and semantic feature extraction is performed on the roadside image data to obtain semantic features; Then, the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data are combined with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the roadside fusion data; Calculate the overlap between different 3D boundaries in the vehicle-mounted fusion data and the roadside fusion data, and regard the pedestrians or vehicles corresponding to the two 3D boundaries with the highest overlap as the same object and merge them; After completing the merging of the three-dimensional boundaries of all pedestrians and vehicles in the vehicle-mounted fusion data and the roadside fusion data, the feature-level data fusion of the vehicle-mounted fusion data and the roadside fusion data is realized to obtain the vehicle-road fusion data.

7. The vehicle-road collaboration intelligent perception method according to claim 6, characterized in that: The process of using the YOLO algorithm to perform target detection on the extracted feature map includes the following steps: Step A1: Use the YOLO algorithm to perform target detection on the first frame image of the feature map to obtain the bounding box and category information of all obstacles in the first frame image; Step A2: Use Kalman filtering to predict the position of each obstacle in the current frame and generate a predicted bounding box of all obstacles in the current frame; Step A3: Calculate the intersection over union (IoU) of each bounding box in the current frame and each predicted bounding box, and use the IoU to calculate the cost matrix of the current frame; Step A4: Based on the cost matrix of the current frame, the Hungarian algorithm is used to perform linear matching between the obstacle bounding box and the predicted bounding box to generate the matching result of the current frame; The matching states of the linear matching are divided into: an uncertain state and a confirmed state; wherein the uncertain state refers to the situation where the bounding box and the predicted bounding box do not achieve perfect matching; the confirmed state refers to the situation where the bounding box and the predicted bounding box perfectly match; the perfect matching refers to the situation where the bounding box and the predicted bounding box become a perfect match when the intersection-over-union ratio of the bounding box and the predicted bounding box meets a preset threshold; Step A5: Use the YOLO algorithm to perform target detection on each frame of the vehicle-road point cloud data except the first frame, obtain the bounding box and category information of each obstacle in each frame, and perform steps A2-A4 on each frame of the vehicle-road point cloud data except the first frame, count the matching results of each frame in the vehicle-road point cloud data, and delete all bounding boxes with trajectory mismatch, and obtain all bounding boxes in the vehicle-road point cloud data whose matching status is confirmed; The track mismatch is: for any bounding box, if the bounding box does not match the predicted bounding box for N consecutive frames, it is considered that the bounding box has a track mismatch; Step A6: for any frame image in the vehicle-road point cloud data, cascade pairing is performed with the predicted bounding box in all the bounding boxes in the frame whose matching status is confirmed, to obtain a cascade pairing result; wherein the cascade pairing result is divided into: the predicted bounding box and the current frame bounding box do not match at all, the predicted bounding box and the current frame bounding box are successfully paired, and an error occurs between the predicted bounding box and the current frame bounding box; Step A7: For all the predicted bounding boxes in the current frame that have errors with the current frame bounding box, the current frame bounding box is compared with all the unmatched predicted bounding boxes in the current frame and the cost matrix is ​​calculated, and step A4 is executed until the predicted bounding boxes of all obstacles in the current frame are successfully matched with the current frame bounding boxes; Step A8: Repeat step A7 until all frame images in the feature map are cascade paired and all obstacle information in the vehicle-road fusion data is identified.

8. The vehicle-road collaboration intelligent perception method according to claim 7, characterized in that: The generation process of the matching result of the current frame in step A4 is as follows: for a bounding box whose matching state is an uncertain state, if the bounding box does not match the predicted bounding box, it indicates that there is a detection mismatch, and the Kalman filter is reused to generate a new predicted bounding box, and the cost matrix of the current frame is recalculated based on the bounding box in the current frame and the new predicted bounding box. Based on the new cost matrix, the Hungarian algorithm is used to linearly match the bounding box of the obstacle and the new predicted bounding box until the matching state of the bounding box is a confirmed state or a preset number of matches is reached; all bounding boxes in the current frame whose matching state is a confirmed state and all unmatched bounding boxes in the current frame are counted.

9. The vehicle-road collaboration intelligent perception method according to claim 8, characterized in that: The process of using the vortex artificial potential field APFV algorithm to plan the path for the obstacle avoidance vehicle includes the following steps: Step B1: Divide all identified obstacles into primary obstacle avoidance targets and secondary obstacle avoidance targets; The main obstacle avoidance targets include: normal running obstacles and emergency obstacle avoidance obstacles; wherein the normal running obstacles are obstacles that are running in the same lane and in the same direction as the obstacle avoiding vehicle; and the emergency obstacle avoidance obstacles are obstacles that are approaching the obstacle avoiding vehicle at a relative speed exceeding a maximum set threshold; The secondary obstacle avoidance target is an obstacle other than the primary obstacle avoidance target; Step B2: Set the secondary obstacle avoidance target as the obstacle avoidance environment, obtain the current position and destination of the obstacle avoidance vehicle, and plan the initial obstacle avoidance path of the obstacle avoidance vehicle using the artificial potential field method based on the primary obstacle avoidance target; Step B3: the obstacle avoiding vehicle travels along the initial obstacle avoiding path. During the travel of the obstacle avoiding vehicle, if the main obstacle avoiding target moves forward relative to the obstacle avoiding vehicle, the obstacle avoiding vehicle stops. With the current parking position as the starting point and the destination as the end point, the travel direction of the obstacle avoiding vehicle is adjusted according to the rotation angle of the obstacle avoiding vehicle, and the obstacle avoiding path is replanned for the obstacle avoiding vehicle using the artificial potential field method until an obstacle-free obstacle avoiding path is generated. At this time, the obstacle avoiding vehicle travels along the obstacle-free obstacle avoiding path. If the main obstacle avoidance target and the obstacle avoidance vehicle are moving towards each other, the relative speed between the obstacle avoidance vehicle and the main obstacle avoidance target is calculated. If the relative speed is a negative value, the speed of the obstacle avoidance vehicle is adjusted to match the speed of the main obstacle avoidance target, so as to ensure that the distance between the obstacle avoidance vehicle and the main obstacle avoidance target is always greater than the set safety distance threshold; If there is no main obstacle avoidance target on the obstacle avoidance path of the obstacle avoidance vehicle, the obstacle avoidance path is simplified into a broken line, and each broken line is guaranteed to be parallel to the lane line, and the corners of the broken line meet the preset vehicle direction adjustment requirements, so as to obtain the final obstacle avoidance path of the obstacle avoidance vehicle, and then generate the driving direction and speed of the obstacle avoidance vehicle.

10. The vehicle-road collaboration intelligent perception method according to claim 9, characterized in that: The step B2 further comprises: Step B2.1: Initialize parameters, including: gravitational coefficient, repulsive coefficient, vortex parameters, coordinates of the destination and coordinates of the main obstacle avoidance target; Step B2.2: Calculate the attractive potential field for pointing to the destination and the repulsive potential field for avoiding the main obstacle avoidance target respectively; Step B2.3: Use the vortex potential field to update the direction of the repulsive potential field; Step B2.4: Calculate the resultant force using the attractive potential field and the updated repulsive potential field and update the position of the obstacle avoidance vehicle; Step B2.5: Based on the updated position of the obstacle avoiding vehicle, repeat steps B2.2-2.4 until the obstacle avoiding vehicle reaches the destination, and plan the initial obstacle avoiding path of the obstacle avoiding vehicle according to all the generated positions of the obstacle avoiding vehicle.

Images