A vehicle-road collaborative intelligent perception method

Abstract

The present invention provides a vehicle-road collaborative intelligent perception method, which relates to the fields of artificial intelligence, automatic driving and machine vision technology. The method specifically includes: for any road scene, respectively collecting and saving roadside point cloud data, roadside image data, vehicle-mounted point cloud data and vehicle-mounted image data; respectively processing the roadside point cloud data and vehicle-mounted point cloud data, and using the YOLO intelligent perception fusion algorithm to perform target detection on the processed vehicle-mounted point cloud data, vehicle-mounted image data, processed roadside point cloud data and roadside image data, and identify obstacle information; based on the identified obstacle information, using the vortex artificial potential field APFV algorithm to plan the path for the obstacle avoidance vehicle, generate the driving direction and driving speed of the obstacle avoidance vehicle, so that the obstacle avoidance vehicle performs automatic driving according to the generated driving direction and driving speed. The present invention overcomes the defects of single-road vehicles by expanding the perception range of the vehicle.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, autonomous driving and machine vision technologies, and in particular to a vehicle-road collaborative intelligent perception method. Background Art

[0002] Autonomous driving is a revolutionary technology with many potential benefits and applications. By utilizing advanced sensors, controllers, and decision-making algorithms, autonomous driving can provide greater safety and efficiency, and address issues such as traffic congestion and accidents.

[0003] Currently, research on autonomous driving technology primarily focuses on autonomous driving by individual vehicles. These vehicles utilize various sensors to perceive their surroundings and use algorithms to make decisions and control the vehicle. However, autonomous driving by individual vehicles has limitations, such as blind spots and sensor failures. To overcome these issues, vehicle-road collaborative autonomous driving has emerged as a new research direction. This technology utilizes both vehicle-side and road-side sensors to jointly perceive the road traffic environment and exchange and share data to enhance the perception and prediction capabilities of autonomous vehicles. This technology can expand the vehicle's perception range, reduce blind spots, and provide redundant perception systems to mitigate sensor failures. By placing sensors on the road, vehicle-road collaboration can gain diverse perspectives and enhance perception capabilities. Furthermore, by sharing data with roadside equipment, vehicles can obtain more comprehensive information, improve the accuracy of their decisions and control, and enhance the safety of autonomous driving.

[0004] Existing research on autonomous driving primarily focuses on the intelligence of individual vehicles. Autonomous vehicles use sensors such as lidar and cameras to perceive their surroundings and make control decisions through algorithms. However, in recent years, the perception and computing capabilities of individual vehicles have advanced to such an advanced level that they have almost reached a bottleneck.

[0005] At the same time, the inherent limitations of single-vehicle autonomous driving prevent it from fundamentally resolving the issues surrounding autonomous driving. First, the sensor's limited range means a single vehicle can only detect objects within a certain range, and the information obtained cannot effectively support a comprehensive understanding of the road conditions ahead. Second, a single-vehicle autonomous driving system can only observe the environment from a single angle. Occlusion by other objects can create blind spots, increasing the risk of traffic accidents. Furthermore, single-vehicle autonomous driving typically only obtains target location information from one side, which can also lead to inaccurate position judgments. Summary of the Invention

[0006] In response to the above-mentioned shortcomings of the existing technology, the present invention proposes a vehicle-road collaborative intelligent perception method based on the YOLO intelligent perception fusion algorithm and the vortex artificial potential field algorithm (APFV), aiming to provide multi-field and multi-perspective perception, expand the vehicle's perception range, and overcome the shortcomings of single-road vehicles.

[0007] The present invention proposes a vehicle-road collaborative intelligent perception method, which includes:

[0008] For any road scene, the road scene includes: a lane, an obstacle-avoiding vehicle, and an obstacle; wherein the obstacle includes: a vehicle and a pedestrian; a roadside sensor is set on the lane, and an onboard sensor is set on the obstacle-avoiding vehicle;

[0009] Use roadside sensors to collect roadside point cloud data and roadside image data of the lane, use vehicle-mounted sensors to collect vehicle-mounted point cloud data and vehicle-mounted image data of the obstacle avoidance vehicle, and save the roadside point cloud data and vehicle-mounted point cloud data in PCD format;

[0010] The roadside point cloud data and the vehicle-mounted point cloud data are respectively subjected to data preprocessing, data screening and data repair in sequence to obtain the processed roadside point cloud data and the vehicle-mounted point cloud data;

[0011] The YOLO intelligent perception fusion algorithm is used to detect targets on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data, and roadside image data to identify obstacle information;

[0012] Based on the identified obstacle information, the vortex artificial potential field (APFV) algorithm is used to plan the path for the obstacle avoidance vehicle, generate the driving direction and speed of the obstacle avoidance vehicle, and enable the obstacle avoidance vehicle to automatically drive according to the generated driving direction and speed.

[0013] The vehicle-mounted sensors include: a vehicle-mounted laser radar, a vehicle-mounted camera, and a global navigation satellite system GNSS module;

[0014] The vehicle-mounted laser radar is used to acquire radar point cloud data of lanes and obstacles in real time during the movement of the obstacle avoidance vehicle by emitting laser beams and receiving reflected signals;

[0015] The vehicle-mounted camera is used to obtain real-time image data of lanes and obstacles during the movement of the obstacle-avoiding vehicle;

[0016] The GNSS module is used to obtain the position of the obstacle-avoiding vehicle and calculate the rotation angle of the obstacle-avoiding vehicle;

[0017] The roadside sensors include: roadside lidar and roadside cameras;

[0018] The roadside laser radar is used to obtain radar point cloud data of obstacle avoidance vehicles and obstacles on the lane in real time;

[0019] The roadside camera is used to obtain real-time image data of obstacle-avoiding vehicles and obstacles on the lane;

[0020] Saving the roadside point cloud data and the vehicle-mounted point cloud data in PCD format is represented as follows: each point cloud data in the roadside point cloud data and the vehicle-mounted point cloud data includes the Cartesian coordinates of the point cloud and the intensity value of the point cloud, represented as: (x, y, z, i), where x is the horizontal coordinate; y is the vertical coordinate; z is the vertical coordinate; and i is the intensity value;

[0021] The process of using the YOLO intelligent perception fusion algorithm to perform target detection on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data includes:

[0022] The processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data are fused using pixel-level fusion or feature-level fusion methods to obtain vehicle-road fusion data.

[0023] The fully connected layer is used to extract features from the vehicle-road fusion data, and the YOLO algorithm is used to detect targets on the extracted feature maps to identify obstacle information in the vehicle-road fusion data.

[0024] The pixel-level fusion method comprises the following steps: projecting the processed onboard point cloud data onto the onboard image plane, and aligning the onboard image data with the projected onboard point cloud data; determining a global coordinate system, and unifying the aligned onboard image data and onboard point cloud data into the global coordinate system; performing pixel-level feature extraction on the onboard image data and onboard point cloud data, respectively, to obtain onboard image features and onboard point cloud features; then extracting regional features from the onboard image data using a superpixel method; clustering the onboard image features, onboard point cloud features, and regional features of the onboard image data using a membership regularization fuzzy clustering method, and then performing pixel-level fusion on the onboard image features and onboard point cloud features based on the clustering results, to obtain onboard fused data;

[0025] The processed roadside point cloud data is projected onto the roadside image plane, and the roadside image data is aligned with the projected roadside point cloud data, and the aligned roadside image data and roadside point cloud data are unified into a global coordinate system; pixel-level feature extraction is performed on the roadside image data and roadside point cloud data respectively to obtain roadside image features and roadside point cloud features; the superpixel method is then used to extract regional features from the roadside image data; the membership regularized fuzzy clustering method is used to cluster the roadside image features, roadside point cloud features and regional features of the roadside image data; the roadside image features and roadside point cloud features are then fused at the pixel level based on the clustering results to obtain roadside fused data;

[0026] Finally, all data points in the vehicle-mounted fusion data and roadside fusion data are mapped to the global coordinate system to obtain the position of each data point in the global coordinate system and obtain the vehicle-road fusion data;

[0027] The feature-level fusion method comprises the following steps: converting the onboard point cloud data and the onboard image data to the same coordinate system, performing feature extraction on the onboard point cloud data, identifying the three-dimensional boundaries of pedestrians and vehicles in the onboard point cloud data, performing semantic feature extraction on the onboard image data to obtain semantic features; and then combining the three-dimensional boundaries of pedestrians and vehicles in the onboard point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the onboard fused data.

[0028] The roadside point cloud data and roadside image data are converted to the same coordinate system, and features are extracted from the roadside point cloud data. The 3D boundaries of pedestrians and vehicles in the roadside point cloud data are identified. Semantic features are extracted from the roadside image data to obtain semantic features. The 3D boundaries of pedestrians and vehicles in the roadside point cloud data are then combined with the semantic features to obtain the 3D boundaries of pedestrians and vehicles in the roadside fusion data.

[0029] The overlap between different 3D boundaries in the vehicle-mounted fusion data and the roadside fusion data is calculated, and the pedestrians or vehicles corresponding to the two 3D boundaries with the highest overlap are considered as the same object and merged. After completing the merging of the 3D boundaries of all pedestrians and vehicles in the vehicle-mounted fusion data and the roadside fusion data, the feature-level data fusion of the vehicle-mounted fusion data and the roadside fusion data is achieved to obtain the vehicle-road fusion data.

[0030] The process of using the YOLO algorithm to perform target detection on the extracted feature map includes the following steps:

[0031] Step A1: Use the YOLO algorithm to perform target detection on the first frame of the feature map to obtain the bounding boxes and category information of all obstacles in the first frame;

[0032] Step A2: Use Kalman filtering to predict the position of each obstacle in the current frame and generate the predicted bounding boxes of all obstacles in the current frame;

[0033] Step A3: Calculate the intersection over union (IoU) of each bounding box in the current frame and each predicted bounding box, and use the IoU to calculate the cost matrix of the current frame;

[0034] Step A4: Based on the cost matrix of the current frame, the Hungarian algorithm is used to perform linear matching between the obstacle bounding box and the predicted bounding box to generate the matching result of the current frame;

[0035] The matching states of the linear matching are divided into: uncertain state and confirmed state; wherein the uncertain state refers to the situation where the bounding box and the predicted bounding box do not achieve perfect matching; the confirmed state refers to the situation where the bounding box and the predicted bounding box achieve perfect matching; the perfect matching refers to the situation where the bounding box and the predicted bounding box achieve perfect matching when the intersection-over-union ratio of the bounding box and the predicted bounding box meets a preset threshold;

[0036] Step A5: Use the YOLO algorithm to perform target detection on each frame of the vehicle-road point cloud data except the first frame, obtain the bounding box and category information of each obstacle in each frame, and perform steps A2-A4 on each frame of the vehicle-road point cloud data except the first frame. Count the matching results of each frame in the vehicle-road point cloud data, delete all bounding boxes with trajectory mismatches, and obtain all bounding boxes in the vehicle-road point cloud data with a confirmed matching status.

[0037] The trajectory mismatch is: for any bounding box, if the bounding box does not match the predicted bounding box for N consecutive frames, it is considered that the bounding box has a trajectory mismatch;

[0038] Step A6: For any frame image in the vehicle-road point cloud data, cascade pairing is performed on all bounding boxes in the frame whose matching status is confirmed with the predicted bounding box to obtain a cascade pairing result; wherein the cascade pairing result is divided into: the predicted bounding box and the current frame bounding box do not match at all, the predicted bounding box and the current frame bounding box are successfully matched, and there is an error between the predicted bounding box and the current frame bounding box;

[0039] Step A7: For all predicted bounding boxes in the current frame that have errors with the current frame bounding box, compare the current frame bounding box with all unmatched predicted bounding boxes in the current frame and calculate the cost matrix, and execute step A4 until all predicted bounding boxes of obstacles in the current frame are successfully matched with the current frame bounding box;

[0040] Step A8: Repeat step A7 until all frame images in the feature map are cascade-paired and all obstacle information in the vehicle-road fusion data is identified;

[0041] The process of generating the matching result of the current frame in step A4 is as follows: for a bounding box whose matching state is uncertain, if the bounding box does not match the predicted bounding box, it indicates that there is a detection mismatch, and the Kalman filter is re-used to generate a new predicted bounding box, and the cost matrix of the current frame is recalculated based on the bounding box in the current frame and the new predicted bounding box. Based on the new cost matrix, the Hungarian algorithm is used to linearly match the bounding box of the obstacle and the new predicted bounding box until the matching state of the bounding box is confirmed or a preset number of matches is reached; all bounding boxes in the current frame whose matching state is confirmed and all unmatched bounding boxes in the current frame are counted;

[0042] The process of using the vortex artificial potential field (APFV) algorithm to plan a path for an obstacle avoidance vehicle includes the following steps:

[0043] Step B1: All identified obstacles are divided into primary obstacle avoidance targets and secondary obstacle avoidance targets;

[0044] The main obstacle avoidance targets include: normal driving obstacles and emergency obstacle avoidance obstacles; wherein the normal driving obstacles are obstacles that are traveling in the same lane and in the same direction as the obstacle avoiding vehicle; the emergency obstacle avoidance obstacles are obstacles that are approaching the obstacle avoiding vehicle at a relative speed exceeding a maximum set threshold;

[0045] The secondary obstacle avoidance target is an obstacle other than the primary obstacle avoidance target;

[0046] Step B2: Set the secondary obstacle avoidance target as the obstacle avoidance environment, obtain the current position and destination of the obstacle avoidance vehicle, and use the artificial potential field method to plan the initial obstacle avoidance path of the obstacle avoidance vehicle based on the primary obstacle avoidance target;

[0047] Step B3: The obstacle-avoiding vehicle travels along the initial obstacle-avoiding path. During the travel of the obstacle-avoiding vehicle, if the primary obstacle avoidance target moves relative to the obstacle-avoiding vehicle, the obstacle-avoiding vehicle stops. With the current parking position as the starting point and the destination as the end point, the obstacle-avoiding vehicle's travel direction is adjusted according to the rotation angle of the obstacle-avoiding vehicle, and the obstacle-avoiding path is replanned for the obstacle-avoiding vehicle using the artificial potential field method until an obstacle-free obstacle-avoiding path is generated. At this point, the obstacle-avoiding vehicle continues to travel along the obstacle-free obstacle-avoiding path.

[0048] If the primary obstacle avoidance target and the obstacle avoiding vehicle are traveling towards each other, the relative speed between the obstacle avoiding vehicle and the primary obstacle avoidance target is calculated. If the relative speed is negative, the speed of the obstacle avoiding vehicle is adjusted to match the speed of the primary obstacle avoidance target, ensuring that the distance between the obstacle avoiding vehicle and the primary obstacle avoidance target is always greater than the set safety distance threshold;

[0049] If there is no main obstacle avoidance target on the obstacle avoidance path of the obstacle avoidance vehicle, the obstacle avoidance path is simplified into a broken line, and each broken line segment is guaranteed to be parallel to the lane line, and the corners of the broken line meet the preset vehicle direction adjustment requirements. The final obstacle avoidance path of the obstacle avoidance vehicle is obtained, and then the driving direction and speed of the obstacle avoidance vehicle are generated;

[0050] The step B2 further comprises:

[0051] Step B2.1: Initialize parameters, including: attraction coefficient, repulsion coefficient, vortex parameters, coordinates of the destination and the coordinates of the main obstacle avoidance target;

[0052] Step B2.2: Calculate the attractive potential field for pointing to the destination and the repulsive potential field for avoiding the main obstacle avoidance target;

[0053] Step B2.3: Use the vortex potential field to update the direction of the repulsive potential field;

[0054] Step B2.4: Calculate the resultant force using the attractive potential field and the updated repulsive potential field and update the position of the obstacle avoidance vehicle;

[0055] Step B2.5: Based on the updated position of the obstacle-avoiding vehicle, repeat steps B2.2-2.4 until the obstacle-avoiding vehicle reaches the destination, and plan the initial obstacle-avoiding path of the obstacle-avoiding vehicle according to all the generated positions of the obstacle-avoiding vehicle.

[0056] The beneficial effects of adopting the above technical solution are:

[0057] The method of the present invention proposes an optimized artificial potential field method suitable for vehicle driving, namely the APFV obstacle avoidance algorithm, which is used to provide obstacle avoidance path planning for vehicles. The method of the present invention adds traffic rules to the traditional artificial potential field method to compensate for the problem that the path planned by the traditional artificial potential field method does not comply with traffic rules, thereby providing obstacle avoidance path planning for vehicles.

[0058] To address the planning scope issue, the method of the present invention designs a new vehicle-road collaborative artificial potential field method path planning algorithm. It obtains current road information through the existing intelligent transportation system and replaces the traditional entire plane with the drivable lane area, thereby avoiding problems such as reverse driving in path planning.

[0059] In order to solve the rationality of path planning, the path planned by the artificial potential field method is improved based on the above content. The path planned by the artificial potential field method is divided into points, and the turning points are found and connected into broken lines. It is also ensured that the planned part that does not change direction is relatively parallel to the lane line.

[0060] To address the planning step size issue, the step size of the artificial potential field method is improved based on the above content. It is changed to the relative speed between the vehicle itself and the target obstacle. A new artificial potential field method calculation is performed every 0.1s and only the next position point is output as an instruction.

[0061] To extract obstacle information from the radar point cloud and camera information transmitted by the sensor, the method proposed in this paper fuses radar perception with the YOLO algorithm to obtain obstacle information. The traditional YOLO algorithm primarily detects objects based on image data. While it can provide two-dimensional bounding box information for objects, it has limitations in depth information and three-dimensional spatial positioning. However, by incorporating radar point clouds, the method proposed in this paper proposes a new vehicle-road collaborative YOLO perception algorithm, which overcomes the YOLO algorithm's poor ability to extract position information.

[0062] To expand perception range, the present invention utilizes vehicle-road collaboration in autonomous driving, enabling the vehicle to obtain information from lidar sensors at the far end of the road. This extends the vehicle's perception range beyond the limitations of sensors and extends further. This expanded perception range can, to a certain extent, alleviate traffic congestion. Specifically, the vehicle can proactively sense traffic flow on the road ahead, enabling it to formulate more effective strategies for navigating traffic jams, such as lane changes.

[0063] In order to supplement blind spots, for autonomous driving with vehicle-road cooperation, the road-side equipment and vehicles in the method of the present invention can perceive objects from different angles, supplement and broaden the vehicle perception information, reduce blind spots in the field of view, and eliminate hidden dangers.

[0064] In order to improve driving accuracy, for autonomous driving with vehicle-road cooperation, the method of the present invention can view objects from multiple angles, thereby obtaining more information and more accurate estimation of position and posture.

[0065] In summary, the method of the present invention designs an optimized artificial potential field method (APFV) algorithm suitable for vehicle driving, a new YOLO perception algorithm for vehicle-infrastructure collaboration, and a new artificial potential field method path planning algorithm for vehicle-infrastructure collaboration. It can also be used to implement a vehicle-infrastructure collaborative intelligent perception system. The method of the present invention not only enables intelligent perception of vehicles and pedestrians, but also demonstrates its feasibility and safety by expanding the perception range, filling in blind spots, and improving perception accuracy. This also proves that vehicle-infrastructure collaboration is superior to autonomous driving of individual vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of a vehicle-road collaborative intelligent perception method in this embodiment;

[0067] Figure 2This is a schematic diagram of a vehicle-road collaborative intelligent perception method in this embodiment;

[0068] Figure 3 Schematic diagram of ROS-CARLA joint simulation in this implementation;

[0069] Figure 4 CARLA simulation result diagram in this embodiment; among them, (a) is the simulated obstacle avoidance path diagram; (b) is the simulation diagram of the obstacle avoidance vehicle driving on a curve; (c) is the simulation diagram of the obstacle avoidance vehicle driving on a straight road. DETAILED DESCRIPTION

[0070] For ease of understanding of the present application, the specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thoroughly and comprehensively understood.

[0071] A vehicle-road cooperative intelligent perception method in this embodiment, such as Figure 1 As shown, the method includes the following processes:

[0072] For any road scene, the road scene includes: a lane, an obstacle-avoiding vehicle and an obstacle; wherein the obstacles include: vehicles and pedestrians; a roadside sensor is set on the lane, and an onboard sensor is set on the obstacle-avoiding vehicle.

[0073] The vehicle-mounted sensors include: a vehicle-mounted laser radar, a vehicle-mounted camera and a Global Navigation Satellite System (GNSS) module.

[0074] The vehicle-mounted laser radar is used to acquire radar point cloud data of lanes and obstacles in real time during the movement of the obstacle-avoiding vehicle by emitting laser beams and receiving reflected signals.

[0075] The vehicle-mounted camera is used to obtain real-time image data of lanes and obstacles during the movement of the obstacle-avoiding vehicle.

[0076] The Global Navigation Satellite System (GNSS) module is used to obtain the position of the obstacle-avoiding vehicle and calculate the rotation angle of the obstacle-avoiding vehicle.

[0077] The roadside sensors include: roadside lidar and roadside cameras.

[0078] The roadside lidar is used to obtain radar point cloud data of obstacle avoidance vehicles and obstacles on the lane in real time.

[0079] The roadside camera is used to obtain image data of obstacle-avoiding vehicles and obstacles on the lane in real time.

[0080] In this embodiment, if Figure 2 As shown, in order to sense the environment, sensors are set up to obtain environmental information. However, due to hardware limitations, this implementation did not use real physical cameras, radars, or actual vehicles during the research process. Instead, a vehicle-road collaborative intelligent perception simulation environment was built using the Ros-Carla joint simulation platform. Specifically, the PDC point cloud library was used to simulate radar point cloud data training, and Carla road videos were recorded to train model parameters. Carla and ROS were combined for joint simulation to simulate vehicle-road collaborative intelligent perception results and evaluate the algorithm. The algorithm in this implementation was implemented using C, MATLAB, and Python.

[0081] ROS-Carla co-simulation is a technology that combines the Robot Operating System (ROS) with Carla, an open-source autonomous driving simulation platform. It enables testing and developing autonomous driving algorithms in a virtual environment. In this implementation, the ROS-Carla co-simulation platform is used to build a vehicle-road collaborative intelligent perception simulation environment. The specific process for building this environment using the ROS-Carla co-simulation platform is as follows: Initializing the test environment, loading a map and 3D vehicles on the Carla platform, and adding obstacles. This implementation design involves an obstacle-avoidance vehicle driving on a curved road, with a roadside LiDAR installed on one side of the curve to sense traffic flow at both ends. The state package, ROS node, and perception and obstacle avoidance package are activated in the ROS-Carla co-simulation platform, and the ROS-Carla bridge is used to connect ROS and Carla. On the Carla platform, a LiDAR and camera are installed on the top of the obstacle-avoidance vehicle to detect pedestrians and other vehicles. These cameras are mounted 2 meters above the ground, with a sensing range of 20 meters. A roadside LiDAR is installed at the end of the road, above the intersection, 3 meters above the ground, with a wider sensing range of up to 40 meters. Additional roadside cameras are installed at high points on both sides of the road, such as rooftops, facing forward to capture lane information. A Global Navigation Satellite System (GNSS) module is located at each end of the obstacle avoidance vehicle to obtain the vehicle's position and calculate its rotation angle.

[0082] Roadside point cloud data and roadside image data of the lane are collected using roadside sensors, and on-board point cloud data and on-board image data of the obstacle avoidance vehicle are collected using on-board sensors, and the roadside point cloud data and on-board point cloud data are saved in PCD format.

[0083] In this embodiment, both the vehicle-mounted laser radar and the roadside laser radar use VLP64 laser radar. In order to better utilize the VLP64 laser radar deployed on vehicles and roads, this embodiment adopts a new method, that is, storing the point cloud data collected by the VLP64 laser radar at a certain moment in PCD format, including: the Cartesian coordinates of the point cloud and the corresponding intensity values, so as to more accurately capture the environmental information at the end of a scan. This method allows the PCD file to store up to 256,000 (x, y, z, i) point cloud information, where x is the horizontal coordinate; y is the vertical coordinate; z is the vertical coordinate; and i is the intensity value.

[0084] The roadside point cloud data and vehicle-mounted point cloud data are respectively subjected to data preprocessing, data screening and data repair in sequence to obtain the processed roadside point cloud data and vehicle-mounted point cloud data.

[0085] In this implementation, data preprocessing refers to operations such as cleaning, deduplication, and format conversion of point cloud data during data acquisition, storage, and reading to ensure data quality and consistency. Data screening refers to filtering out abnormal data, such as invalid or unreasonable data, by setting thresholds or rules to facilitate subsequent processing and analysis. Data repair refers to repairing missing or abnormal data through methods such as interpolation and smoothing to ensure data integrity and accuracy.

[0086] The YOLO intelligent perception fusion algorithm is used to perform target detection on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data to identify obstacle information.

[0087] In this embodiment, the traditional Yolo algorithm performs target detection on image data. Although it can provide two-dimensional bounding box information of the target, it has limitations in depth information and three-dimensional spatial positioning. This embodiment fuses on-board image data, on-board point cloud data, roadside image data, and roadside point cloud data, so that the fused vehicle-road fusion data can include information about obstacle avoidance vehicles and pedestrians on the road. The most important collaborative method in the vehicle-road collaboration solution is the sharing and fusion of vehicle-road perception data. Therefore, after obtaining data from the vehicle and road sides, this embodiment designs two data fusion schemes to fuse the data, and then reuses the fused data for planning.

[0088] The process of using the YOLO intelligent perception fusion algorithm to perform target detection on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data includes:

[0089] The processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data are fused using pixel-level fusion or feature-level fusion methods to obtain vehicle-road fusion data.

[0090] The pixel-level fusion method is as follows: projecting the processed on-board point cloud data onto the on-board image plane, and aligning the on-board image data with the projected on-board point cloud data; determining a global coordinate system, and unifying the aligned on-board image data and on-board point cloud data into the global coordinate system; performing pixel-level feature extraction on the on-board image data and on-board point cloud data respectively to obtain on-board image features and on-board point cloud features; then using a superpixel method to extract regional-level features from the on-board image data; using a membership regularized fuzzy clustering method to cluster the on-board image features, on-board point cloud features, and regional-level features of the on-board image data, and then performing pixel-level fusion on the on-board image features and on-board point cloud features based on the clustering results to obtain on-board fused data.

[0091] Similarly, the processed roadside point cloud data is projected onto the roadside image plane, and the roadside image data is aligned with the projected roadside point cloud data, and the aligned roadside image data and roadside point cloud data are unified into the global coordinate system; pixel-level feature extraction is performed on the roadside image data and roadside point cloud data respectively to obtain roadside image features and roadside point cloud features; the superpixel method is then used to extract regional-level features from the roadside image data; the membership regularized fuzzy clustering method is used to cluster the roadside image features, roadside point cloud features and regional-level features of the roadside image data, and then the roadside image features and roadside point cloud features are fused at the pixel level based on the clustering results to obtain roadside fused data.

[0092] Finally, all data points in the vehicle-mounted fusion data and roadside fusion data are mapped to the global coordinate system to obtain the position of each data point in the global coordinate system and obtain the vehicle-road fusion data.

[0093] In this embodiment, pixel-level fusion directly shares and merges the raw data from vehicle-side and road-side sensors, mapping each data point to the same space. By combining the depth information of the point cloud with the semantic information of the image, pixel-level fusion can significantly improve object detection and segmentation performance. By introducing pixel and region-level information fusion in membership-regularized fuzzy clustering for image segmentation, segmentation results can be further optimized. For LiDAR point cloud data, each data point has three-dimensional coordinate information. To perform pixel-level fusion on the point cloud data from the roadside and vehicle-side radars, this embodiment requires unifying the coordinate systems of the logarithmic points. Pixel-level fusion requires special processing. Unlike a single vehicle LiDAR point cloud, the fused point cloud in this embodiment has two sources, and conventional object detection methods cannot effectively utilize the information contained therein. Therefore, to leverage the advantages of pixel-level fusion, a new object detection model is needed. This model uses pixel-level fusion to supplement object perception that is typically achieved through camera-only detection. Pixel-level fusion offers the advantage of high accuracy, but its drawback is the large amount of data required, making it difficult to fuse data from different sensor types.

[0094] The feature-level fusion method is as follows: converting the on-board point cloud data and the on-board image data to the same coordinate system, performing feature extraction on the on-board point cloud data, and identifying the three-dimensional boundaries of pedestrians and vehicles in the on-board point cloud data; performing semantic feature extraction on the on-board image data to obtain semantic features; and then combining the three-dimensional boundaries of pedestrians and vehicles in the on-board point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the on-board fusion data.

[0095] Similarly, the roadside point cloud data and roadside image data are converted to the same coordinate system, feature extraction is performed on the roadside point cloud data, and the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data are identified. Semantic feature extraction is performed on the roadside image data to obtain semantic features. The three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data are then combined with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the roadside fusion data.

[0096] The degree of overlap between different three-dimensional boundaries in the vehicle-mounted fusion data and the roadside fusion data is calculated, and the pedestrians or vehicles corresponding to the two three-dimensional boundaries with the highest overlap are regarded as the same object and merged; after completing the merging of the three-dimensional boundaries of all pedestrians and vehicles in the vehicle-mounted fusion data and the roadside fusion data, the feature-level data fusion of the vehicle-mounted fusion data and the roadside fusion data is realized to obtain the vehicle-road fusion data.

[0097] In this embodiment, feature-level fusion is achieved by processing the sensor data separately and sharing and combining the extracted features. For the three-dimensional boundaries of the identified pedestrians and vehicles, each boundary consists of 8 coordinate points. In order to perform feature-level fusion of the three-dimensional boundaries of pedestrians and vehicles identified on the road side and the vehicle side, what this embodiment needs to do is to unify the coordinate system of the boundaries. The boundaries of objects of the same type with a high degree of overlap are regarded as the same object and unified into one object, which facilitates the subsequent use of the YOLO intelligent perception fusion algorithm to extract obstacle information. Feature-level fusion data can be easily used in existing autonomous driving frameworks. After fusing the boundaries obtained by three-dimensional target detection of point cloud data, the quantity and quality of perceived boundaries can be improved, and the subsequent processing methods will not change much. Therefore, feature-level fusion should be easy to use by autonomous driving systems.

[0098] The fully connected layer is used to extract features from the vehicle-road fusion data, and then the YOLO algorithm is used to perform target detection on the extracted feature map to identify obstacle information in the vehicle-road fusion data.

[0099] In this embodiment, radar perception and the YOLO algorithm are fused, and obstacle information is obtained by using the YOLO algorithm perception fusion, and the YOLO algorithm trains the obtained obstacle information into a 3D model.

[0100] The process of using the YOLO algorithm to perform target detection on the extracted feature map includes the following steps:

[0101] Step A1: Use the YOLO algorithm to perform target detection on the first frame image of the feature map to obtain the bounding boxes and category information of all obstacles in the first frame image.

[0102] Step A2: Use Kalman filtering to predict the position of each obstacle in the current frame and generate the predicted bounding boxes of all obstacles in the current frame.

[0103] In this embodiment, in the first frame detection, Kalman filtering can be used to predict the position of the obstacle, and a corresponding prediction bounding box can be generated based on these prediction results, which can improve the accuracy of the prediction.

[0104] Step A3: Calculate the intersection over union (IoU) of each bounding box in the current frame and each predicted bounding box, and use the IoU to calculate the cost matrix of the current frame.

[0105] In this implementation, the obstacle's bounding box is compared with the predicted bounding box. This involves calculating the Intersection over Union (IoU) ratio (IoU), a measure of the degree of overlap between two bounding boxes. This ratio is calculated as the ratio of the area of ​​their intersection to the area of ​​their union. Each element in the cost matrix is ​​defined as 1 – IoU, which is used to determine the object in the current frame.

[0106] Step A4: Based on the cost matrix of the current frame, the Hungarian algorithm is used to perform linear matching between the obstacle bounding box and the predicted bounding box to generate the matching result of the current frame.

[0107] The matching states of the linear matching are divided into: an uncertain state and a confirmed state; wherein the uncertain state refers to the situation where the bounding box and the predicted bounding box do not achieve a perfect match; the confirmed state refers to the situation where the bounding box and the predicted bounding box achieve a perfect match; and the perfect match refers to the situation where the bounding box and the predicted bounding box achieve a perfect match when their intersection-over-union ratio meets a preset threshold.

[0108] The generation process of the matching result of the current frame is as follows: for a bounding box whose matching state is uncertain, if the bounding box does not match the predicted bounding box, it indicates that there is a detection mismatch, and the Kalman filter is re-used to generate a new predicted bounding box, and the cost matrix of the current frame is recalculated according to the bounding box in the current frame and the new predicted bounding box. Based on the new cost matrix, the Hungarian algorithm is used to linearly match the bounding box of the obstacle and the new predicted bounding box until the matching state of the bounding box is confirmed or the preset number of matches is reached; and all bounding boxes in the current frame whose matching state is confirmed and all unmatched bounding boxes in the current frame are counted.

[0109] Step A5: Use the YOLO algorithm to perform target detection on each frame of the vehicle-road point cloud data except the first frame, obtain the bounding box and category information of each obstacle in each frame, and perform steps A2-A4 on each frame of the vehicle-road point cloud data except the first frame. Count the matching results for each frame in the vehicle-road point cloud data, delete all bounding boxes with trajectory mismatches, and obtain all bounding boxes in the vehicle-road point cloud data with a confirmed matching status.

[0110] The track mismatch is as follows: for any bounding box, if the bounding box does not match the predicted bounding box for N consecutive frames, it is considered that the bounding box has a track mismatch.

[0111] In this embodiment, based on the cost matrix, the Hungarian algorithm is used to linearly match the bounding box predicted by the motion trajectory with the bounding box of the current frame to prune the bounding box. In this process, situations where the two do not achieve a perfect match are referred to as an uncertain state, while situations where a perfect match is achieved are considered a confirmed state. After detection, three different situations may arise: the first, known as unmatched tracks, indicates that the predicted bounding box does not match the current frame. In the uncertain state, if matching fails for 30 consecutive frames, the bounding box is pruned. The second, known as unmatched detections, indicates that the predicted bounding box does not match the current frame's bounding box. In the uncertain state, the Kalman filter is re-used to predict a new predicted bounding box. Linear matching is then used to prune the bounding box again until a confirmed state is reached, at which point processing can cease. In the third situation, if the detected box perfectly matches the predicted box, it indicates that the previous and subsequent frames have been effectively tracked, thus reaching a confirmed state.

[0112] In this implementation, a Kalman filter is used to estimate the state of the objects corresponding to the predicted bounding box and analyze their appearance, motion, and other features. To achieve this, the appearance features of the previous 100 frames are first saved and then correlated with the appearance, motion, and other features of the obstacles corresponding to the predicted bounding box. Finally, a Kalman filter is used to estimate their states, thereby enabling obstacle recognition. Because the confirmed bounding box of the current frame has a high degree of similarity to the predicted bounding box, appropriate measures are taken.

[0113] Step A6: For any frame image in the vehicle-road point cloud data, cascade pairing is performed on all bounding boxes in the frame whose matching status is confirmed with the predicted bounding box to obtain a cascade pairing result. The cascade pairing result is divided into: the predicted bounding box and the current frame bounding box do not match at all, the predicted bounding box and the current frame bounding box are successfully matched, and there is an error between the predicted bounding box and the current frame bounding box.

[0114] In this embodiment, three different conclusions are obtained through the cascade pairing of the predicted bounding box and the current frame bounding box in the confirmed state. The first is that when the predicted bounding box and the current frame bounding box are successfully paired, the Kalman filter is used to detect the Tracks variable, and the same conclusion as step A3 is obtained. The second is that when an error occurs between the predicted bounding box and the current frame bounding box, it is caused by the rapid movement, occlusion or detection error of the target; at this time, the current frame bounding box is compared with the unmatched predicted bounding boxes one by one to estimate the cost matrix; the third is a complete mismatch due to target loss, new target appearance or detection failure. If the predicted bounding box does not match any bounding box in multiple consecutive frames, it may be considered that the target has been lost and tracking of the target is stopped. For the new bounding box that does not match the predicted bounding box in the current frame, a new Kalman filter is initialized and tracking of the new target begins.

[0115] Step A7: For all predicted bounding boxes in the current frame that have errors with the current frame bounding box, compare the current frame bounding box with all unmatched predicted bounding boxes in the current frame and calculate the cost matrix, and execute step A4 until the predicted bounding boxes of all obstacles in the current frame are successfully matched with the current frame bounding box.

[0116] Step A8: Repeat step A7 until all frame images in the feature map are cascade-paired and all obstacle information in the vehicle-road fusion data is identified.

[0117] According to the identified obstacle information, the vortex artificial potential field APFV algorithm is used to plan the path for the obstacle avoidance vehicle, generate the driving direction and speed of the obstacle avoidance vehicle, and enable the obstacle avoidance vehicle to automatically drive according to the generated driving direction and speed.

[0118] The process of using the vortex artificial potential field (APFV) algorithm to plan a path for an obstacle avoidance vehicle includes the following steps:

[0119] Step B1: All identified obstacles are divided into primary obstacle avoidance targets and secondary obstacle avoidance targets.

[0120] The main obstacle avoidance targets include: normal driving obstacles and emergency obstacle avoidance obstacles; wherein the normal driving obstacles are obstacles that are traveling in the same lane and in the same direction as the obstacle avoiding vehicle; the emergency obstacle avoidance obstacles are obstacles that are approaching the obstacle avoiding vehicle at a relative speed exceeding a maximum set threshold.

[0121] The secondary obstacle avoidance target is an obstacle other than the primary obstacle avoidance target.

[0122] In this embodiment, a primary obstacle avoidance target is selected, and the secondary target is set as the obstacle avoidance environment rather than the primary obstacle avoidance target. During vehicle travel, primary obstacle avoidance targets can be divided into two types: normal driving obstacle avoidance targets, i.e., obstacles traveling in the same lane as the vehicle, and emergency obstacle avoidance targets, i.e., target obstacles approaching at high relative speeds from any direction. Because obstacle avoidance may cause the ego vehicle to change its route, impacting vehicles with no prior interaction, such as parallel vehicles, this embodiment uses the primary obstacle avoidance vehicle as the target in the APFV algorithm, while considering secondary obstacle vehicles and their expected routes as the feasible planning range.

[0123] Step B2: Set the secondary obstacle avoidance target as the obstacle avoidance environment, obtain the current position and destination of the obstacle avoidance vehicle, and use the artificial potential field method to plan the initial obstacle avoidance path of the obstacle avoidance vehicle based on the primary obstacle avoidance target.

[0124] Step B2.1: Initialize parameters, including: attraction coefficient, repulsion coefficient, vortex parameters, coordinates of the destination and coordinates of the main obstacle avoidance target.

[0125] Step B2.2: Calculate the attractive potential field for pointing to the destination and the repulsive potential field for avoiding the main obstacle avoidance target respectively.

[0126] In this implementation, the attractive force in the attractive potential field is directed toward the target point, and its magnitude is proportional to the distance between the obstacle-avoiding vehicle and the destination. The repulsive force in the repulsive potential field is directed away from the obstacle, and its magnitude is inversely proportional to the distance between the obstacle-avoiding vehicle and the primary obstacle avoidance target. It's important to note that the repulsive force is calculated when the distance between the obstacle-avoiding vehicle and the primary obstacle avoidance target is less than a set threshold; otherwise, the repulsive force is zero.

[0127] Step B2.3: Use the vortex potential field to update the direction of the repulsive potential field.

[0128] In this embodiment, the vortex potential field is achieved by rotating the direction of the repulsive force, with the rotation angle dynamically adjusted based on the relative position of the obstacle-avoiding vehicle and the primary obstacle. When the robot approaches an obstacle too close, the vortex potential field guides the robot around it. If the obstacle-avoiding vehicle is trapped in a local minimum, the vortex potential field guides the vehicle around the primary obstacle.

[0129] Step B2.4: Calculate the resultant force using the gravitational force and the updated repulsive force and update the position of the obstacle avoidance vehicle.

[0130] Step B2.5: Based on the updated position of the obstacle-avoiding vehicle, repeat steps B2.2-2.4 until the obstacle-avoiding vehicle reaches the destination, and plan the initial obstacle-avoiding path of the obstacle-avoiding vehicle according to all the generated positions of the obstacle-avoiding vehicle.

[0131] Step B3: The obstacle-avoiding vehicle travels along the initial obstacle-avoiding path. During the travel of the obstacle-avoiding vehicle, if the main obstacle avoidance target moves relative to the obstacle-avoiding vehicle, the obstacle-avoiding vehicle stops. With the current parking position as the starting point and the destination as the end point, the travel direction of the obstacle-avoiding vehicle is adjusted according to the rotation angle of the obstacle-avoiding vehicle, and the obstacle avoidance path is replanned for the obstacle-avoiding vehicle using the artificial potential field method until an obstacle-free obstacle avoidance path is generated. At this time, the obstacle-avoiding vehicle travels along the obstacle-free obstacle avoidance path.

[0132] If the primary obstacle avoidance target and the obstacle avoiding vehicle are traveling towards each other, the relative speed between the obstacle avoiding vehicle and the primary obstacle avoidance target is calculated. If the relative speed is a negative value, the speed of the obstacle avoiding vehicle is adjusted to match the speed of the primary obstacle avoidance target to ensure that the distance between the obstacle avoiding vehicle and the primary obstacle avoidance target is always greater than the set safety distance threshold.

[0133] If there is no main obstacle avoidance target on the obstacle avoidance path of the obstacle avoidance vehicle, the obstacle avoidance path is simplified into a broken line, and each broken line segment is guaranteed to be parallel to the lane line, and the corners of the broken line meet the preset vehicle direction adjustment requirements. The final obstacle avoidance path of the obstacle avoidance vehicle is obtained, and then the driving direction and speed of the obstacle avoidance vehicle are generated.

[0134] In this embodiment, it is determined whether the current obstacle avoidance environment satisfies the obstacle avoidance algorithm. First, a curve is planned using the traditional artificial potential field method to connect the current position and the target point. If this curve intersects with any obstacle vehicle, the following instructions are executed: If the ego vehicle and the main obstacle avoidance vehicle are moving relative to each other, a stop instruction is issued. Relative moving forward means that the ego vehicle and the main obstacle avoidance vehicle are traveling in the same direction, and the ego vehicle is at the rear and the obstacle avoidance vehicle is in the front. With the current parking position as the starting point and the target point as the predetermined destination, the attractive potential field and the repulsive potential field are recalculated. Based on the updated potential field, a new safe path is calculated and the direction of movement is determined. If an obstacle-free path can be found, it can be restarted. During the above process, the artificial potential field method is always started. If the ego vehicle and the main obstacle avoidance vehicle are traveling in opposite directions, a constant speed following instruction is issued, that is, the ego vehicle travels at the same speed as the target vehicle to avoid the ego vehicle stopping in the middle of the road due to obstacle avoidance. If the curve has no intersection with the obstacle vehicle, the curve is simplified to a broken line, which is kept as parallel to the lane line as possible, and the corners of the broken line meet the preset vehicle direction adjustment requirements. That is, according to the actual situation, sufficient space is reserved for the direction adjustment part of the obstacle avoidance vehicle before the broken line fitting is performed.

[0135] In this implementation, ROS-Carla joint simulation is used, such as Figure 3 and Figure 4As shown, the intelligent perception algorithm and road planning algorithm run in the ROS environment, outputting vehicle motion information to Carla for a 3D simulation demonstration. The map and 3D vehicle are loaded on the Carla platform, and the obstacle avoidance vehicle and obstacles are added to initialize the algorithm test environment. The ROS node and the perception and obstacle avoidance function package are activated, and the ROS-Carla-bridge is started to connect ROS and Carla. On the Carla platform, cameras and perception radars are manually added to the road and vehicle sides, and the YOLO intelligent perception fusion algorithm is invoked for sensor recognition. Under the current simulation environment parameter settings, the virtual camera recognition is verified to be effective. If not, or the recognition effect is poor, the virtual camera recognition model parameters are adjusted. If the recognition effect is good, proceed to the next step. In the ROS environment, the APFV algorithm and intelligent perception algorithm function packages are activated. Intelligent perception detects obstacle information, and vehicle and pedestrian obstacle information is obtained from the YOLO intelligent perception fusion algorithm. During the APFV algorithm simulation, although a more complex virtual map is desirable for 3D modeling, parameter adjustment requires starting with a simple map for recognition testing. Therefore, this implementation begins with a single obstacle map for recognition testing. Upon successful recognition, a new obstacle is added, and the YOLO intelligent perception fusion algorithm is re-invoked for sensor recognition. This process is repeated until the map shows obstacles in all three directions, ensuring a high probability of the vehicle navigating the obstacle avoidance path. Finally, the vehicle is started for obstacle avoidance simulation.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A vehicle-road collaborative intelligent perception method, characterized in that: The method includes the following steps: For any road scene, the road scene includes: a lane, an obstacle-avoiding vehicle, and an obstacle; wherein the obstacle includes: a vehicle and a pedestrian; a roadside sensor is set on the lane, and an onboard sensor is set on the obstacle-avoiding vehicle; Use roadside sensors to collect roadside point cloud data and roadside image data of the lane, use vehicle-mounted sensors to collect vehicle-mounted point cloud data and vehicle-mounted image data of the obstacle avoidance vehicle, and save the roadside point cloud data and vehicle-mounted point cloud data in PCD format; The roadside point cloud data and the vehicle-mounted point cloud data are respectively subjected to data preprocessing, data screening and data repair in sequence to obtain the processed roadside point cloud data and the vehicle-mounted point cloud data; The YOLO intelligent perception fusion algorithm is used to detect targets on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data, and roadside image data to identify obstacle information; Based on the identified obstacle information, the vortex artificial potential field (APFV) algorithm is used to plan the path for the obstacle avoidance vehicle, generate the driving direction and speed of the obstacle avoidance vehicle, and enable the obstacle avoidance vehicle to automatically drive according to the generated driving direction and speed. The steps include: Step B1: All identified obstacles are divided into primary obstacle avoidance targets and secondary obstacle avoidance targets; The main obstacle avoidance targets include: normal driving obstacles and emergency obstacle avoidance obstacles; wherein the normal driving obstacles are obstacles that are traveling in the same lane and in the same direction as the obstacle avoiding vehicle; the emergency obstacle avoidance obstacles are obstacles that are approaching the obstacle avoiding vehicle at a relative speed exceeding a maximum set threshold; The secondary obstacle avoidance target is an obstacle other than the primary obstacle avoidance target; Step B2: Set the secondary obstacle avoidance target as the obstacle avoidance environment, obtain the current position and destination of the obstacle avoidance vehicle, and use the artificial potential field method to plan the initial obstacle avoidance path of the obstacle avoidance vehicle based on the primary obstacle avoidance target; Step B3: The obstacle-avoiding vehicle travels along the initial obstacle-avoiding path. During the travel of the obstacle-avoiding vehicle, if the primary obstacle avoidance target moves relative to the obstacle-avoiding vehicle, the obstacle-avoiding vehicle stops. With the current parking position as the starting point and the destination as the end point, the obstacle-avoiding vehicle's travel direction is adjusted according to the rotation angle of the obstacle-avoiding vehicle, and the obstacle-avoiding path is replanned for the obstacle-avoiding vehicle using the artificial potential field method until an obstacle-free obstacle-avoiding path is generated. At this point, the obstacle-avoiding vehicle continues to travel along the obstacle-free obstacle-avoiding path. If the primary obstacle avoidance target and the obstacle avoiding vehicle are traveling towards each other, the relative speed between the obstacle avoiding vehicle and the primary obstacle avoidance target is calculated. If the relative speed is negative, the speed of the obstacle avoiding vehicle is adjusted to match the speed of the primary obstacle avoidance target, ensuring that the distance between the obstacle avoiding vehicle and the primary obstacle avoidance target is always greater than the set safety distance threshold; If there is no main obstacle avoidance target on the obstacle avoidance path of the obstacle avoidance vehicle, the obstacle avoidance path is simplified into a broken line, and each broken line segment is guaranteed to be parallel to the lane line, and the corners of the broken line meet the preset vehicle direction adjustment requirements. The final obstacle avoidance path of the obstacle avoidance vehicle is obtained, and then the driving direction and speed of the obstacle avoidance vehicle are generated.

2. The vehicle-road collaborative intelligent perception method according to claim 1, characterized in that: The vehicle-mounted sensors include: a vehicle-mounted laser radar, a vehicle-mounted camera, and a global navigation satellite system GNSS module; The vehicle-mounted laser radar is used to acquire radar point cloud data of lanes and obstacles in real time during the movement of the obstacle avoidance vehicle by emitting laser beams and receiving reflected signals; The vehicle-mounted camera is used to obtain real-time image data of lanes and obstacles during the movement of the obstacle-avoiding vehicle; The GNSS module is used to obtain the position of the obstacle-avoiding vehicle and calculate the rotation angle of the obstacle-avoiding vehicle; The roadside sensors include: roadside lidar and roadside cameras; The roadside laser radar is used to obtain radar point cloud data of obstacle avoidance vehicles and obstacles on the lane in real time; The roadside camera is used to obtain image data of obstacle-avoiding vehicles and obstacles on the lane in real time.

3. The vehicle-road collaborative intelligent perception method according to claim 2, characterized in that: The roadside point cloud data and the vehicle-mounted point cloud data are saved in PCD format as follows: each point cloud data in the roadside point cloud data and the vehicle-mounted point cloud data includes the Cartesian coordinates of the point cloud and the intensity value of the point cloud, expressed as: (x, y, z, i), where x is the horizontal coordinate; y is the vertical coordinate; z is the vertical coordinate; and i is the intensity value.

4. The vehicle-road collaborative intelligent perception method according to claim 3, characterized in that: The process of using the YOLO intelligent perception fusion algorithm to perform target detection on the processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data includes: The processed vehicle point cloud data, vehicle image data, processed roadside point cloud data and roadside image data are fused using pixel-level fusion or feature-level fusion methods to obtain vehicle-road fusion data. The fully connected layer is used to extract features from the vehicle-road fusion data, and then the YOLO algorithm is used to perform target detection on the extracted feature map to identify obstacle information in the vehicle-road fusion data.

5. The vehicle-road collaborative intelligent perception method according to claim 4, characterized in that: The pixel-level fusion method is as follows: projecting the processed vehicle point cloud data onto the vehicle image plane, and aligning the vehicle image data with the projected vehicle point cloud data; A global coordinate system is determined, and the aligned vehicle image data and vehicle point cloud data are unified under the global coordinate system. Pixel-level feature extraction is performed on the vehicle image data and vehicle point cloud data respectively to obtain vehicle image features and vehicle point cloud features. The superpixel method is then used to extract regional features from the vehicle image data. The membership regularized fuzzy clustering method is used to cluster the vehicle image features, vehicle point cloud features, and regional features of the vehicle image data. Based on the clustering results, the vehicle image features and vehicle point cloud features are then pixel-fused to obtain vehicle fused data. Projecting the processed roadside point cloud data onto the roadside image plane, aligning the roadside image data with the projected roadside point cloud data, and unifying the aligned roadside image data and roadside point cloud data into a global coordinate system; Pixel-level feature extraction is performed on the roadside image data and the roadside point cloud data respectively to obtain roadside image features and roadside point cloud features; Then, the superpixel method is used to extract regional features from the roadside image data. The membership regularized fuzzy clustering method is used to cluster the roadside image features, roadside point cloud features, and regional features of the roadside image data. Based on the clustering results, the roadside image features and roadside point cloud features are fused at the pixel level to obtain the roadside fused data. Finally, all data points in the vehicle-mounted fusion data and roadside fusion data are mapped to the global coordinate system to obtain the position of each data point in the global coordinate system and obtain the vehicle-road fusion data.

6. The vehicle-road collaborative intelligent perception method according to claim 5, characterized in that: The feature-level fusion method comprises the following steps: converting the onboard point cloud data and the onboard image data to the same coordinate system, performing feature extraction on the onboard point cloud data, identifying the three-dimensional boundaries of pedestrians and vehicles in the onboard point cloud data, performing semantic feature extraction on the onboard image data to obtain semantic features; and then combining the three-dimensional boundaries of pedestrians and vehicles in the onboard point cloud data with the semantic features to obtain the three-dimensional boundaries of pedestrians and vehicles in the onboard fused data. Convert the roadside point cloud data and roadside image data to the same coordinate system, perform feature extraction on the roadside point cloud data, identify the three-dimensional boundaries of pedestrians and vehicles in the roadside point cloud data, and perform semantic feature extraction on the roadside image data to obtain semantic features. The 3D boundaries of pedestrians and vehicles in the roadside point cloud data are then combined with semantic features to obtain the 3D boundaries of pedestrians and vehicles in the roadside fusion data. Calculate the overlap between different 3D boundaries in the vehicle-mounted fusion data and the roadside fusion data, and consider the pedestrians or vehicles corresponding to the two 3D boundaries with the highest overlap as the same object and merge them; After completing the merging of the three-dimensional boundaries of all pedestrians and vehicles in the vehicle-mounted fusion data and the roadside fusion data, the feature-level data fusion of the vehicle-mounted fusion data and the roadside fusion data is realized to obtain the vehicle-road fusion data.

7. The vehicle-road collaborative intelligent perception method according to claim 6, characterized in that: The process of using the YOLO algorithm to perform target detection on the extracted feature map includes the following steps: Step A1: Use the YOLO algorithm to perform target detection on the first frame of the feature map to obtain the bounding boxes and category information of all obstacles in the first frame; Step A2: Use Kalman filtering to predict the position of each obstacle in the current frame and generate the predicted bounding boxes of all obstacles in the current frame; Step A3: Calculate the intersection over union (IoU) of each bounding box in the current frame and each predicted bounding box, and use the IoU to calculate the cost matrix of the current frame; Step A4: Based on the cost matrix of the current frame, the Hungarian algorithm is used to perform linear matching between the obstacle bounding box and the predicted bounding box to generate the matching result of the current frame; The matching states of the linear matching are divided into: uncertain state and confirmed state; wherein the uncertain state refers to the situation where the bounding box and the predicted bounding box do not achieve perfect matching; the confirmed state refers to the situation where the bounding box and the predicted bounding box achieve perfect matching; the perfect matching refers to the situation where the bounding box and the predicted bounding box achieve perfect matching when the intersection-over-union ratio of the bounding box and the predicted bounding box meets a preset threshold; Step A5: Use the YOLO algorithm to perform target detection on each frame of the vehicle-road point cloud data except the first frame, obtain the bounding box and category information of each obstacle in each frame, and perform steps A2-A4 on each frame of the vehicle-road point cloud data except the first frame. Count the matching results of each frame in the vehicle-road point cloud data, delete all bounding boxes with trajectory mismatches, and obtain all bounding boxes in the vehicle-road point cloud data with a confirmed matching status. The trajectory mismatch is: for any bounding box, if the bounding box does not match the predicted bounding box for N consecutive frames, it is considered that the bounding box has a trajectory mismatch; Step A6: For any frame image in the vehicle-road point cloud data, cascade pairing is performed on all bounding boxes in the frame whose matching status is confirmed with the predicted bounding box to obtain a cascade pairing result; wherein the cascade pairing result is divided into: the predicted bounding box and the current frame bounding box do not match at all, the predicted bounding box and the current frame bounding box are successfully matched, and there is an error between the predicted bounding box and the current frame bounding box; Step A7: For all predicted bounding boxes in the current frame that have errors with the current frame bounding box, compare the current frame bounding box with all unmatched predicted bounding boxes in the current frame and calculate the cost matrix, and execute step A4 until all predicted bounding boxes of obstacles in the current frame are successfully matched with the current frame bounding box; Step A8: Repeat step A7 until all frame images in the feature map are cascade-paired and all obstacle information in the vehicle-road fusion data is identified.

8. The vehicle-road collaborative intelligent perception method according to claim 7, characterized in that: The generation process of the matching result of the current frame in step A4 is as follows: for the bounding box whose matching status is uncertain, if the bounding box does not match the predicted bounding box, it means that there is a detection mismatch, and the Kalman filter is reused to generate a new predicted bounding box, and the cost matrix of the current frame is recalculated based on the bounding box in the current frame and the new predicted bounding box. Based on the new cost matrix, the Hungarian algorithm is used to linearly match the bounding box of the obstacle and the new predicted bounding box until the matching status of the bounding box is confirmed or the preset number of matches is reached; all bounding boxes in the current frame whose matching status is confirmed and all unmatched bounding boxes in the current frame are counted.

9. The vehicle-road collaborative intelligent perception method according to claim 8, characterized in that: The step B2 further comprises: Step B2.1: Initialize parameters, including: attraction coefficient, repulsion coefficient, vortex parameters, coordinates of the destination and the coordinates of the main obstacle avoidance target; Step B2.2: Calculate the attractive potential field for pointing to the destination and the repulsive potential field for avoiding the main obstacle avoidance target; Step B2.3: Use the vortex potential field to update the direction of the repulsive potential field; Step B2.4: Calculate the resultant force using the attractive potential field and the updated repulsive potential field and update the position of the obstacle avoidance vehicle; Step B2.5: Based on the updated position of the obstacle-avoiding vehicle, repeat steps B2.2-2.4 until the obstacle-avoiding vehicle reaches the destination, and plan the initial obstacle-avoiding path of the obstacle-avoiding vehicle according to all the generated positions of the obstacle-avoiding vehicle.

Images