A perception information fusion representation and target detection method for vehicle-road cooperation

Abstract

A kind of perception information fusion representation and target detection method for vehicle-road cooperation, comprising the following steps: deploying road laser radar, configuring corresponding road side computing device for road laser radar;Calibrate road laser radar external parameter;Road side computing device calculates the relative position of automatic driving vehicle relative to road laser radar according to automatic driving vehicle positioning data and road laser radar external parameter;Road side computing device deflects the road side laser radar point cloud detected by road laser radar to the coordinate system of automatic driving vehicle according to relative position, obtains deflected point cloud;Road side computing device carries out voxel processing to deflected point cloud, obtains voxel deflected point cloud.Automatic driving vehicle carries out voxel processing to the vehicle laser radar point cloud detected by vehicle laser radar, and obtains voxel vehicle laser radar point cloud;Road side computing device calculates the voxel level feature of voxel deflected point cloud, and obtains deflected point cloud voxel level feature.Automatic driving vehicle calculates the voxel level feature of voxel vehicle laser radar point cloud, and obtains vehicle laser radar point cloud voxel level feature;Each point cloud voxel level feature is compressed and transmitted to computing device, and transmission device can be automatic driving vehicle, road side computing device or cloud.The computing device carries out data splicing and data aggregation to vehicle laser radar point cloud voxel level feature and deflected point cloud voxel level feature, and obtains aggregated voxel level feature;Computing device inputs aggregated voxel level feature into three-dimensional target detection network model based on voxel level feature, and obtains target detection result;When computing device is road side computing device or cloud, finally, target detection result is sent to automatic driving vehicle.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving vehicle-road cooperative technology, and relates to a vehicle-road cooperative target detection method that utilizes perception information fusion representation. Background Technology

[0002] In the 21st century, with the continuous development of urban roads and the automotive industry, automobiles have become one of the essential means of transportation, bringing great convenience to people's daily lives and work. However, the overuse of automobiles has also brought problems such as environmental pollution, traffic congestion, and traffic accidents. To alleviate the problem of excessive automobile use, to remove people from the transportation system, and to improve vehicle driving capabilities while freeing the driver's hands, autonomous vehicles have gradually become an important direction for future automotive development. With the rise of deep learning technology and the widespread attention given to artificial intelligence, autonomous driving, as a key area of ​​focus within AI, has also become a hot topic.

[0003] Autonomous driving is a complete hardware and software interactive system. Core technologies for autonomous driving include hardware (automotive manufacturing technology, autonomous driving chips), autonomous driving software, high-precision maps, and sensor communication networks. From a software perspective, it can be broadly divided into three modules: environmental perception, behavioral decision-making, and motion control.

[0004] Perception is the first link in autonomous driving, serving as the link between the vehicle and its environment. The overall performance of an autonomous driving system depends primarily on the performance of its perception system. Autonomous vehicles achieve perception through sensors, with LiDAR (Light Detection and Ranging) utilizing lasers for detection and measurement. Its principle involves emitting pulsed laser light, which reflects back upon encountering objects. By calculating the time difference between the reflection and the return, the distance is determined, thus creating a three-dimensional model of the surrounding environment. LiDAR offers high detection accuracy and long range; due to the short wavelength of laser light, it can detect very small targets over long distances. The point cloud data obtained by LiDAR is information-rich and highly accurate, making it widely used for target detection and classification within the perception loop of autonomous driving. On the one hand, lidar has overturned the traditional two-dimensional projection imaging mode, and can collect the depth information of the target surface to obtain relatively complete spatial information of the target. After data processing, the three-dimensional surface of the target can be reconstructed to obtain a three-dimensional graphic that better reflects the geometric shape of the target. At the same time, it can also obtain rich feature information such as the surface reflection characteristics and movement speed of the target, providing sufficient information support for data processing such as target detection, identification and tracking, and reducing the difficulty of algorithms. On the other hand, the application of active laser technology makes it have the characteristics of high measurement resolution, strong anti-interference ability, strong anti-stealth ability, strong penetration ability and all-weather operation.

[0005] Based on the presence or absence of mechanical components, lidar can be divided into mechanical lidar and solid-state lidar. Although solid-state lidar is considered the future trend, mechanical lidar still dominates the current lidar market. Mechanical lidar has rotating components that control the laser emission angle, while solid-state lidar does not require mechanical rotating components and mainly relies on electronic components to control the laser emission angle.

[0006] In existing autonomous driving solutions, LiDAR is generally the primary sensor in the environmental perception module, undertaking most tasks such as real-time map building, localization, and target detection. For example, Google's Waymo incorporates five LiDARs in its sensor configuration: four side LiDARs distributed at the front, rear, left, and right of the vehicle, which are short-to-medium range multi-line radars used to supplement blind spots; and a high-line-count LiDAR on the top for wide-area perception, with its blind spots supplemented by the four side LiDARs.

[0007] LiDAR sensors record scanning data in the form of point clouds. Point cloud data refers to a set of vectors in a three-dimensional coordinate system. These vectors are usually represented in the form of X, Y, Z three-dimensional coordinates. In addition to containing three-dimensional coordinates, each point may also contain color information (RGB) or reflectance intensity information.

[0008] The X, Y, and Z columns represent the three-dimensional position of the point data in the sensor coordinate system or the world coordinate system, usually in meters. The Intensity column represents the laser reflection intensity at each point, and its value is dimensionless and is generally normalized to between 0 and 255.

[0009] Because the installation height of vehicle-mounted LiDAR is limited by vehicle size, typically only around two meters, the information it can detect is easily affected by obstructions around the vehicle. For example, a truck driving in front of a small car can almost completely block the forward view of the LiDAR, severely weakening its environmental perception capabilities. Furthermore, the performance of the LiDAR itself is limited by the overall cost of the vehicle; vehicles often do not equip themselves with expensive, high-line-count LiDAR. Therefore, the point cloud data obtained by vehicle-mounted LiDAR often has blind spots or sparse data, making it difficult to complete autonomous driving perception tasks relying solely on the vehicle's sensors. Compared to vehicle-mounted LiDAR, roadside LiDAR installed on infrastructure can be deployed on higher gantries or lampposts, providing a more transparent and less obstructed field of view. Additionally, roadside LiDAR has a higher cost tolerance, allowing the use of higher-line-count LiDAR and more powerful roadside computing units to achieve higher detection performance and faster detection speeds.

[0010] Currently, vehicle-road cooperative systems are in the midst of a research and testing boom. Intelligent vehicle-road cooperative solutions based on V2X technology can enhance the currently achievable assisted driving functions, improve vehicle driving safety and road operation efficiency, and provide data services and technical support for autonomous driving in the long term.

[0011] Existing LiDAR-based vehicle-to-infrastructure (V2I) solutions involve both vehicles and roadside infrastructure performing target detection based on LiDAR point cloud data. The infrastructure then transmits the detection results to the vehicle. Most research focuses on the reliability of transmitted data, the relative pose calculation between the vehicle and roadside infrastructure, or the handling of data transmission latency, assuming that the V2I process directly sends the target detection results. While this approach has lower data transmission volume, it still cannot fully utilize the detection data from both ends. For example, if neither the vehicle nor roadside infrastructure detects a complete target point cloud, false positives and missed detections are likely, leading to errors in the target detection results after collaboration. To address this, some researchers have proposed directly sending the original point cloud data to prevent information loss. For instance, the Cooper framework, proposed in 2019, was the first to propose a cooperative perception scheme at the original point cloud data level, significantly improving perception performance by fusing point cloud data from different sources.

[0012] However, at the same time, the size of a single frame of LiDAR point cloud data often exceeds ten megabytes or even tens of megabytes, and the existing vehicle-to-infrastructure (V2I) communication conditions are insufficient to support such a large amount of real-time point cloud data transmission. Therefore, autonomous driving technology urgently needs a better collaborative detection method that utilizes LiDAR data from both ends, which can meet the requirements for target detection accuracy while minimizing the amount of data transmission.

[0013] Existing target recognition and classification algorithms based on lidar point cloud data are all based on deep neural network technology.

[0014] Existing technology

[0015] Patent document US9562971B2

[0016] Patent document US20150187216A1

[0017] Patent document CN110989620A

[0018] Patent document CN110781927A

[0019] Patent document CN111222441A

[0020] Patent document CN108010360A Summary of the Invention

[0021] To address the aforementioned issues, this invention provides a perception information fusion representation and target detection method for vehicle-road cooperation, and a vehicle-road cooperation scheme based on LiDAR point cloud data that balances the size of transmitted data and the degree of information loss, in order to solve the problems of insufficient single-vehicle perception capability and insufficient vehicle-road cooperative communication bandwidth in current autonomous vehicles.

[0022] The specific technical problems to be solved include determining the roadside lidar deployment scheme, selecting the roadside lidar extrinsic parameter calibration method, calculating the deflection parameters based on the relative pose of autonomous vehicles and roadside lidar, and determining a suitable information representation format for vehicle-road cooperation.

[0023] The objective of this invention is to reduce the amount of information transmitted while ensuring vehicle-road cooperative perception capabilities.

[0024] The technical problem solved by this invention patent is divided into a preparation stage and an application stage. The steps of the preparation stage are as follows:

[0025] A. Deploy roadside lidar and equip it with corresponding roadside computing devices;

[0026] B. Calibrate the external parameters of the roadside lidar.

[0027] The steps in the application phase are as follows:

[0028] C. The roadside computing device calculates the relative pose of the autonomous vehicle with respect to the roadside lidar based on the autonomous vehicle's positioning data and the external parameters of the roadside lidar;

[0029] D. The roadside computing device deflects the roadside lidar point cloud detected by the roadside lidar into the autonomous vehicle coordinate system according to the relative pose, thus obtaining the deflected point cloud.

[0030] E. Roadside computing equipment performs voxelization on the deflection point cloud to obtain a voxelized deflection point cloud. Autonomous vehicles perform voxelization on the point cloud detected by the onboard LiDAR to obtain a voxelized onboard LiDAR point cloud.

[0031] F. Roadside computing equipment calculates the voxel-level features of the deflected point cloud to obtain the voxel-level features of the deflected point cloud. Autonomous vehicles calculate the voxel-level features of the onboard LiDAR point cloud to obtain the voxel-level features of the onboard LiDAR point cloud;

[0032] The subsequent steps are divided into three sub-schemes: I, II, and III. Sub-scheme I completes steps G1, H1, and I1 on the roadside computing device; sub-scheme II completes steps G2, H2, and I2 on the autonomous vehicle; and sub-scheme III completes steps G3, H3, and I3 in the cloud.

[0033] In sub-scheme I:

[0034] G1. The autonomous vehicle compresses the voxel-level features of the on-board LiDAR point cloud to obtain compressed voxel-level features of the on-board LiDAR point cloud, and transmits them to the roadside computing device. The roadside computing device receives the compressed voxel-level features of the on-board LiDAR point cloud and restores them to the voxel-level features of the on-board LiDAR point cloud.

[0035] H1. The roadside computing device performs data stitching and data aggregation on the voxel-level features of the vehicle-mounted lidar point cloud and the voxel-level features of the deflection point cloud to obtain aggregated voxel-level features;

[0036] I1. The roadside computing device aggregates voxel-level features and inputs them into a three-dimensional target detection network model based on voxel-level features to obtain target detection results, and then transmits the target detection results to the autonomous vehicle;

[0037] In Sub-scheme II:

[0038] G2. The roadside computing device compresses the voxel-level features of the deflection point cloud to obtain compressed voxel-level features of the deflection point cloud, and transmits them to the autonomous vehicle. The autonomous vehicle receives the compressed voxel-level features of the deflection point cloud and restores them to voxel-level features of the deflection point cloud.

[0039] H2. Autonomous vehicles perform data stitching and data aggregation on the voxel-level features of the onboard LiDAR point cloud and the voxel-level features of the deflected point cloud to obtain aggregated voxel-level features.

[0040] I2. Autonomous vehicles input aggregated voxel-level features into a 3D target detection network model based on voxel-level features to obtain target detection results;

[0041] In Sub-scheme III:

[0042] G3. The autonomous vehicle compresses the voxel-level features of the onboard LiDAR point cloud to obtain compressed onboard LiDAR point cloud voxel-level features, and transmits them to the cloud. The roadside computing device compresses the voxel-level features of the deflection point cloud to obtain compressed deflection point cloud voxel-level features, and transmits them to the cloud. The cloud receives the compressed deflection point cloud voxel-level features and the compressed onboard LiDAR point cloud voxel-level features, restores the compressed deflection point cloud voxel-level features to the original deflection point cloud voxel-level features, and restores the compressed onboard LiDAR point cloud voxel-level features to the original onboard LiDAR point cloud voxel-level features.

[0043] H3. The cloud performs data stitching and data aggregation on the voxel-level features of the vehicle-mounted LiDAR point cloud and the voxel-level features of the deflected point cloud to obtain aggregated voxel-level features.

[0044] I3. The cloud inputs the aggregated voxel-level features into the 3D target detection network model based on voxel-level features to obtain the target detection results, and then transmits the target detection results to the autonomous vehicle.

[0045] The specific technical solutions in the above steps of this invention patent are as follows:

[0046] A. Deploy lidar

[0047] The deployment of roadside lidar is determined based on the existing roadside pillar facilities and the type of lidar installed in the vehicle-road cooperative scenario. Existing roadside lidar installation methods include poles or crossbars, specifically on roadside gantries, streetlights, traffic light poles, and other electrically powered infrastructure pillars.

[0048] Based on whether or not they have internal rotating parts, lidar can be divided into mechanical rotating lidar, hybrid lidar, and solid-state lidar. Among them, mechanical rotating lidar and solid-state lidar are the two types of lidar commonly used on roadsides.

[0049] For scenarios such as intersections, deploying a single roadside LiDAR with a detection range equal to or greater than the scenario's area, or including key areas within the scenario, is sufficient. For long-distance, large-scale, and complex scenarios such as expressways, highways, and industrial parks, it is recommended to follow the following roadside LiDAR deployment guidelines to ensure that the roadside LiDAR coverage meets the scenario's full coverage requirements. This means that a single roadside LiDAR should supplement the detection blind spots of other roadside LiDARs within the scenario, thereby achieving better vehicle-road cooperative target detection results.

[0050] The roadside lidar deployment guidelines are divided into roadside mechanical rotating lidar deployment guidelines and roadside all-solid-state lidar deployment guidelines, depending on the type of roadside lidar used.

[0051] A1) Deployment schemes for roadside mechanical rotating lidar and roadside hybrid solid-state lidar

[0052] Mechanical rotating lidar achieves laser scanning through mechanical rotation. The laser emitting components are arranged in a vertical linear array of laser sources, and lenses can generate beams pointing at different angles within the vertical plane. Driven by a motor, continuous rotation transforms the beams within the vertical plane from "lines" to "surfaces," forming multiple laser "surfaces" through rotational scanning, thus achieving detection within the detection area. Hybrid solid-state lidar, on the other hand, uses semiconductor "micro-motion" devices (such as MEMS scanning mirrors) to replace macroscopic mechanical scanners, achieving laser scanning at the radar transmitter at a microscopic scale.

[0053] The deployment guidelines for roadside mechanical rotating lidar and roadside hybrid solid-state lidar require that these be installed horizontally to ensure full utilization of beam information from all directions. For example... Figure 2 The deployment of roadside mechanical rotating lidar and roadside hybrid solid-state lidar should at least meet the following requirements:

[0054]

[0055] in:

[0056] H a Indicates the installation height of roadside mechanical rotating lidar or roadside hybrid solid-state lidar;

[0057] This indicates the angle between the highest elevation beam of a roadside mechanical rotating lidar or a roadside hybrid solid-state lidar and the horizontal direction.

[0058] L a This indicates the distance between two adjacent roadside mechanical rotating lidar or roadside hybrid solid-state lidar installation poles.

[0059] A2) Roadside All-Solid-State LiDAR Deployment Scheme

[0060] All-solid-state lidar completely eliminates the mechanical scanning structure; its horizontal and vertical laser scanning is achieved electronically. A phased-array laser emitter consists of a rectangular array of several transmitting and receiving units. By changing the phase difference of the emitted light from different units in the array, the angle and direction of the emitted laser can be adjusted. The laser source passes through an optical beam splitter and enters an optical waveguide array. The phase of the light wave is changed by external control on the waveguides, and beam scanning is achieved using the phase difference between the waveguides.

[0061] like Figure 3 The guidelines for the deployment of roadside all-solid-state lidar require that the deployment of roadside all-solid-state lidar should at least meet the following requirements:

[0062]

[0063] in:

[0064] H b Indicates the installation height of the roadside all-solid-state lidar;

[0065] This indicates the field of view angle of the roadside all-solid-state lidar in the vertical direction;

[0066] This indicates the angle between the highest elevation beam of the roadside all-solid-state lidar and the horizontal direction;

[0067] L b This indicates the distance between two adjacent roadside all-solid-state lidar installation poles.

[0068] For scenarios involving the installation of all-solid-state LiDAR, two reverse LiDARs can be installed on the same pole to compensate for blind spots in roadside sensing, reducing the need for a certain number of roadside poles. In this case, the following conditions must be met: Figure 4 The requirements shown are as follows:

[0069]

[0070] in:

[0071] H c Indicates the installation height of the roadside all-solid-state lidar;

[0072] This indicates the angle between the highest elevation beam of the roadside all-solid-state lidar and the horizontal direction;

[0073] L c This indicates the distance between two adjacent roadside all-solid-state lidar installation poles.

[0074] For LiDAR vehicle-to-infrastructure (V2I) scenarios that meet the above conditions, roadside mechanical rotating LiDARs or all-solid-state LiDARs should be deployed as required above, and the scanning area of ​​each LiDAR should be increased when conditions permit. For LiDAR V2I scenarios that do not meet the above conditions, the deployment conditions of roadside LiDARs should be met by deploying new poles and increasing the number of roadside LiDARs, thus ensuring compliance with the roadside LiDAR deployment guidelines.

[0075] B. External parameter calibration

[0076] To calculate the relative pose of the roadside lidar and the vehicle-mounted lidar, it is necessary to calibrate the installation position and angle of the roadside lidar, i.e., extrinsic parameter calibration, which obtains the coordinate position parameters and angular attitude parameters of the lidar relative to a certain reference coordinate system. The extrinsic parameters of the lidar can be represented by the following vectors:

[0077] V0=[x0y0z0α0β0γ0](4)

[0078] in:

[0079] x0 represents the X coordinate of the roadside lidar in the reference coordinate system;

[0080] y0 represents the Y-coordinate of the roadside lidar in the reference coordinate system;

[0081] z0 represents the Z coordinate of the roadside lidar in the reference coordinate system;

[0082] α0 represents the rotation angle of the roadside lidar around the X-axis in the reference coordinate system;

[0083] β0 represents the rotation angle of the roadside lidar around the Y-axis in the reference coordinate system;

[0084] γ0 represents the rotation angle of the roadside lidar around the Z-axis in the reference coordinate system;

[0085] The aforementioned reference coordinate system can be a latitude and longitude coordinate system represented by GCJ02 and WGS84, or a geodetic coordinate system based on a specific geographical point, such as the Beijing 54 coordinate system and the Xi'an 80 coordinate system. Correspondingly, the relationship between the actual coordinates of a point in the reference coordinate system and the coordinates in the roadside lidar coordinate system obtained after detection by the aforementioned lidar is as follows:

[0086]

[0087]

[0088]

[0089]

[0090] in:

[0091] x lidar This is the X-coordinate of the point in the roadside lidar coordinate system;

[0092] y lidar This is the Y-coordinate of the point in the roadside lidar coordinate system;

[0093] z lidar This is the Z-coordinate of the point in the roadside lidar coordinate system;

[0094] x real This is the X-coordinate of the point in the reference coordinate system;

[0095] y real This is the Y-coordinate of the point in the reference coordinate system;

[0096] z real This is the Z-coordinate of the point in the reference coordinate system;

[0097] R x (α0), R y (β0), R z (γ0) is a sub-rotation matrix calculated based on the three angular extrinsic parameters α0, β0 and γ0;

[0098] The specific values ​​of the external parameters of the roadside lidar are calculated by measuring the coordinates of the control points in the roadside lidar coordinate system and the reference coordinate system. The steps are as follows:

[0099] ① Select at least four reflectivity feature points as control points within the detection range of the roadside lidar. Reflectivity feature points refer to points whose reflectivity differs significantly from surrounding objects, such as traffic signs and license plates. The purpose of selecting reflectivity feature points as control points is to facilitate the rapid identification of corresponding points in the point cloud data based on their location and reflection intensity, thereby quickly establishing the correspondence between multiple pairs of points in the point cloud and a coordinate in the reference coordinate system. Control points should be distributed as discretely as possible. Under the condition that the scene environment allows and the control points meet the following requirements, the more control points, the better the calibration effect. Control point selection requirements include: they should be discretely distributed, and no three control points should be collinear; within the detection range of the roadside lidar, the selected control points should be as far away from the lidar as possible, typically greater than 50% of the lidar's maximum detection distance. If it is difficult to select control points at 50% of the lidar's maximum detection distance due to scene limitations, control points can be selected at less than 50% of the lidar's maximum detection distance, but the number of control points should be increased.

[0100] ② Use handheld high-precision RTK or other high-precision measuring instruments to measure the precise coordinates of control points and find the corresponding point coordinates in the roadside lidar point cloud; when you have a high-precision map file of the roadside lidar deployment scene, you do not need to use handheld high-precision RTK or other high-precision measuring instruments to measure, and you can directly find the coordinates of the corresponding feature points in the high-precision map.

[0101] ③ The optimal value of the lidar extrinsic parameter vector V0 is calculated using a three-dimensional registration algorithm, and the result is used as the calibration result. Commonly used three-dimensional registration algorithms include the ICP algorithm and the NDT algorithm, among which the ICP algorithm is mainly used when applying lidar extrinsic parameter calibration. The basic principle of the ICP algorithm is to calculate the optimal matching extrinsic parameters in the matching target point set P (the set of coordinates of control points in the roadside lidar coordinate system) and the source point set Q (the set of coordinates of control points in the reference coordinate system) to minimize the error function. The error function is:

[0102]

[0103] R = R x (α0)R y (β0)R z (γ0) (10)

[0104] T = [x0y0z0] T (11)

[0105] in:

[0106] E(R,T) is the target error function;

[0107] R is the rotation transformation matrix;

[0108] T is the translation transformation matrix;

[0109] n is the number of nearest neighbor pairs in the point set;

[0110] p i Let be the coordinates of the i-th point in the target point set P;

[0111] q i For the source point set Q and point p i Points that form the nearest neighbor pair;

[0112] C. Calculate relative pose

[0113] The relative pose between the autonomous vehicle and the roadside LiDAR is determined based on the autonomous vehicle's positioning data and the extrinsic parameter calibration results of the roadside LiDAR during preliminary preparation. The relative pose is calculated using the following formula:

[0114] V′=[V′ xyz V′ αβγ (12)

[0115]

[0116] V′ αβγ =[α′ β′ γ′] T =[α′1β′1γ′1] T -[α′0β′0γ′0] T (14)

[0117] V1=[x1y1z1α1β1γ1] T (15)

[0118] in:

[0119] V′ is the position and angle vector of the autonomous vehicle relative to the roadside lidar.

[0120] V′ xyz The position vector of the autonomous vehicle relative to the roadside LiDAR.

[0121] V′ αβγ The angle vector of the autonomous vehicle relative to the roadside lidar

[0122] V1 is the position and angle vector of the autonomous vehicle in the reference coordinate system.

[0123] D. Deflection

[0124] The roadside lidar point cloud D is calculated using the following formula. r Deflected to the autonomous vehicle coordinate system:

[0125]

[0126]

[0127] R = R x (α′)R y (β′)R z (γ′)(18)

[0128] T=[x′ y′ z′] (19)

[0129] in:

[0130] H rc This is the transformation matrix for shifting the coordinate system of the roadside lidar to that of the autonomous vehicle.

[0131] x ego y ego z ego Let [x] be the coordinates of a point in the roadside lidar point cloud after it has been deflected into the autonomous vehicle coordinate system. The corresponding coordinates of the point in the roadside lidar coordinate system are [x]. lidar y lidar z lidar ] T ;

[0132] O is the perspective transformation vector. Since there is no perspective transformation in this scene, O is set to [0 0 0].

[0133] E. Voxelization

[0134] A voxel, short for volume pixel, is the smallest unit of digital data segmentation in three-dimensional space. Conceptually, it's similar to a pixel, the smallest unit in two-dimensional space. After segmenting point cloud data using voxels, data features can be calculated for each voxel's points. The features of the set of points within each voxel are called voxel-level features. A large class of existing 3D object detection algorithms processes LiDAR point cloud data based on voxel-level features. They extract voxel-level features after voxelizing the point cloud data and input them into a subsequent 3D object detection network model based on voxel-level features to obtain the object detection result.

[0135] The steps for voxelizing point cloud data are as follows:

[0136] E1) Based on the point cloud of the vehicle-mounted lidar D c The spatial dimension [DWH] of the location, and the size of the design voxel is [D V W V H V The vehicle-mounted LiDAR is divided into voxels based on the designed voxel size.

[0137] E2) For deflection point clouds Using vehicle-mounted LiDAR point cloud D c Divide the data using the same voxel partitioning method to ensure that the partitioning deflects the point cloud. Spatial grid and vehicle-mounted LiDAR point cloud D c Completely overlapping. For example, the point cloud of an onboard LiDAR (Light Detection and Ranging) vehicle. c The distribution space is [-31m, 33m] in the X-axis direction, and the voxel D V The value is 4m. If the point cloud is deflected at this time... If the distribution space is [-32m, 34m] in the X-axis direction, it should be expanded to [-35m, 37m] to obtain the expanded deflection point cloud. To ensure the D-point cloud of the vehicle-mounted LiDAR c And the expanded deflection point cloud The voxel mesh is consistent. The specific calculation formula is as follows:

[0138]

[0139]

[0140]

[0141] n1, n2 ∈ N

[0142] in:

[0143] S ego Point cloud for vehicle-mounted LiDAR c The spatial range;

[0144] S lidar ′ represents the expanded deflection point cloud The spatial range;

[0145] K lidar_start ′、K lidar_end ′ is the deflection point cloud expanded in the K-dimensional dimension The range of starting and ending values;

[0146] K lidar_start K lidar_end Deflection point cloud in K dimension The range of starting and ending values;

[0147] K ego_start K ego_end D, the point cloud of the vehicle-mounted LiDAR in the K dimension c The range of starting and ending values;

[0148] V K The size of the voxel in the K-dimensional dimension;

[0149] E3) Based on the point cloud of the vehicle-mounted lidar Dc And the expanded deflection point cloud The voxels containing the scattered data are grouped, with scattered data within the same voxel belonging to the same group. Due to the non-uniformity and sparsity of the points, the number of scattered data in each voxel may not be the same, and some voxels may have no scattered data.

[0150] E4) To reduce computational burden and eliminate discrimination problems caused by density inconsistencies, voxels with more than a certain threshold of scattered data are randomly sampled. A threshold of 35 is recommended. When there are fewer scattered data points in the point cloud data, the threshold can be appropriately reduced. This strategy can save computational resources and reduce the imbalance between voxels.

[0151] Through steps E1 to E4, the voxelized point cloud of the vehicle-mounted LiDAR is obtained. c Obtaining voxelized point clouds of vehicle-mounted LiDAR Voxelized expanded deflection point cloud Obtain voxelized deflection point cloud

[0152] F. Calculating voxel-level features

[0153] The methods for calculating voxel-level features in point clouds vary depending on the object detection model used by the autonomous vehicle. Taking the VoxelNet model for object detection in an autonomous vehicle as an example, the steps are as follows:

[0154] ① First, organize the voxelized point cloud. For the i-th point in voxel A, the original data collected is as follows:

[0155] a i =[x i y i z i r i ](twenty three)

[0156] in:

[0157] x i y i z i Let X, Y, and Z be the X, Y, and Z coordinates of the i-th point, respectively.

[0158] r i Let be the reflection intensity at the i-th point;

[0159] ②Then calculate the mean of the coordinates of all points within the voxel, denoted as [v x v y v z ].

[0160] ③Then, the information of the i-th point is supplemented by an offset relative to the center, that is:

[0161]

[0162] in:

[0163] This is the information for the i-th point after supplementation;

[0164] ④ The processed voxelized point cloud is input into a series of cascaded VFE layers. A schematic diagram of the VFE layers processing the voxelized point cloud data is shown below. Figure 5 The processing logic of the VFE layer is to first enable each After passing through a fully connected network layer, point-level features are obtained for each point. Then, the point-level features are subjected to max pooling to obtain voxel-level features. Finally, the voxel-level features are concatenated with the point-level features obtained in the previous step to obtain the point-level concatenated feature result.

[0165] ⑤ After processing through cascaded VFE layers, the final voxel-level features are obtained by integration and max pooling through fully connected layers. Each voxel-level feature is a 1×C dimensional vector.

[0166] Voxelized vehicle-mounted LiDAR point cloud and voxelized deflection point cloud The above methods can be used to obtain the voxel-level features of the point cloud of the vehicle-mounted lidar. and deflection point cloud voxel-level features

[0167] G. Point cloud voxel-level feature transfer

[0168] Because point clouds exist sparsely in space, many voxels lack scattered point data and therefore corresponding voxel-level features. Storing point cloud voxel-level features using a special structure can significantly compress the data size, reducing the transmission difficulty when sending them to processing devices; this is known as voxel-level feature compression. One such special structure is a hash table, a data structure that allows direct access based on a key value. It accesses records by mapping key values ​​to a location in the table, thus speeding up the search. In this hash table, the hash key is the spatial coordinate of the voxel, and the corresponding value is the voxel-level feature.

[0169] When using sub-scheme I, subsequent processing is performed on the roadside computing equipment.

[0170] G1) Voxel-level features of point clouds from onboard LiDAR for autonomous vehicles Compression processing is performed to obtain the voxel-level features of the compressed vehicle-mounted lidar point cloud. The data is then transmitted to a roadside computing device, which receives and compresses the voxel-level features of the point cloud from the vehicle-mounted LiDAR. Compress the voxel-level features of the point cloud of the vehicle-mounted LiDAR Restored to voxel-level features of point cloud from vehicle-mounted LiDAR

[0171] When using Sub-Scheme II, subsequent processing is performed on the autonomous vehicle.

[0172] G2) Roadside computing equipment for deflection point cloud voxel-level features Compression processing is performed to obtain compressed deflection point cloud voxel-level features. And transmit it to the autonomous vehicle; the autonomous vehicle receives the compressed deflection point cloud voxel-level features. Compressed deflection point cloud voxel-level features Reduced to voxel-level features of deflected point cloud

[0173] When using Sub-Scheme III, subsequent processing is performed in the cloud.

[0174] G3) Voxel-level features of point clouds from onboard LiDAR for autonomous vehicles Compression processing is performed to obtain the voxel-level features of the compressed vehicle-mounted lidar point cloud. And transmit it to the cloud. Roadside computing equipment analyzes the voxel-level features of the deflection point cloud. Compression processing is performed to obtain compressed deflection point cloud voxel-level features. And transmit it to the cloud. The cloud receives the compressed deflection point cloud voxel-level features. and compressed vehicle-mounted lidar point cloud voxel-level features Compressed deflection point cloud voxel-level features Reduced to voxel-level features of deflected point cloud Compress the voxel-level features of the point cloud of the vehicle-mounted LiDAR Restored to voxel-level features of point cloud from vehicle-mounted LiDAR

[0175] H. Data splicing and data aggregation

[0176] Perform data stitching operations, that is, to extract voxel-level features from the point cloud of the vehicle-mounted LiDAR. and deflection point cloud voxel-level features Alignment is performed based on the position of the voxels in the autonomous vehicle coordinate system.

[0177] Perform data aggregation operations, that is, perform voxel-level feature aggregation on the point cloud of the vehicle-mounted LiDAR. and deflection point cloud voxel-level features For any position where one voxel is empty, the voxel-level feature of the non-empty side is taken as the aggregated voxel-level feature. For voxels where neither side is empty, the final aggregated voxel-level feature is calculated using the following formula:

[0178]

[0179]

[0180] in:

[0181] It is a polymer-level characteristic;

[0182] f k Features of aggregates at the element level The value at position k;

[0183] f ego_k Voxel-level features of point clouds for vehicle-mounted LiDAR The value at position k;

[0184] f lidar_k For deflection point cloud voxel-level features The value at position k;

[0185] Even if the maximum pooling method is used to aggregate features of the same coordinate voxels.

[0186] When using sub-scheme I, subsequent processing is performed on the roadside computing equipment.

[0187] H1) The roadside computing equipment performs voxel-level feature analysis of the point cloud from the vehicle-mounted lidar using the method described above. and deflection point cloud voxel-level features Data splicing and aggregation are performed to obtain aggregated voxel-level features.

[0188] When using Sub-Scheme II, subsequent processing is performed on the autonomous vehicle.

[0189] H2) Autonomous vehicles perform voxel-level feature analysis of onboard LiDAR point clouds using the method described above. and deflection point cloud voxel-level features Data splicing and aggregation are performed to obtain aggregated voxel-level features.

[0190] When using Sub-Scheme III, subsequent processing is performed in the cloud.

[0191] H3) Cloud-based voxel-level features of vehicle-mounted LiDAR point cloud are processed using the method described above. and deflection point cloud voxel-level features Data splicing and aggregation are performed to obtain aggregated voxel-level features.

[0192] I. Target Detection

[0193] The aggregated voxel-level features are input into a subsequent 3D object detection network model to obtain the detected target. Taking VoxelNet as an example, after obtaining the aggregated voxel-level features, they are input into a 3D object detection network model based on voxel-level features to obtain the object detection result.

[0194] The target detection result can be represented as U, specifically:

[0195] U = [u1... u] n (27)

[0196]

[0197] in:

[0198] u i This refers to the information of the i-th target in the target detection results;

[0199] x i Let x be the x-axis coordinate of the i-th detected target in the autonomous vehicle coordinate system;

[0200] y i Let be the y-axis coordinate of the i-th detected target in the autonomous vehicle coordinate system;

[0201] z i Let be the z-axis coordinate of the i-th detected target in the autonomous vehicle coordinate system;

[0202] C i Let be the confidence level of the i-th detected target;

[0203] W i The width of the detection box corresponding to the i-th detection target;

[0204] D i Let be the length of the detection box corresponding to the i-th detection target;

[0205] H i Let be the height of the detection box corresponding to the i-th detection target;

[0206] Let be the orientation angle of the detection box corresponding to the i-th detection target;

[0207] v xi Let be the projection of the velocity of the i-th detected target onto the x-axis in the coordinate system of the autonomous vehicle.

[0208] v yi Let be the projection of the velocity of the i-th detected target onto the y-axis in the coordinate system of the autonomous vehicle.

[0209] v ziLet be the projection of the velocity of the i-th detected target onto the z-axis in the coordinate system of the autonomous vehicle.

[0210] For any 3D object detection network model based on voxel-level features, its object detection result should at least include the object's location, i.e., x. i y i z i For high-performance 3D target detection network models based on voxel-level features, the target detection results should include the C-values ​​of the detected target. i W i D i H i , v xi v yi v zi Some or all of the attributes. Among them, W i D i H i The three attributes can only exist simultaneously or none of them can exist in the target detection result. xi v yi v zi The three attributes can either exist simultaneously or not exist simultaneously in the target detection result.

[0211] When using sub-scheme I, target detection is performed on roadside computing equipment.

[0212] I1) Roadside computing equipment aggregates voxel-level features Input a 3D target detection network model based on voxel-level features to obtain the target detection result U, and then transmit the target detection result to the autonomous vehicle.

[0213] When using Sub-Scheme II, object detection is performed on the autonomous vehicle.

[0214] I2) Autonomous vehicles will aggregate voxel-level features Input a 3D target detection network model based on voxel-level features to obtain the target detection result U.

[0215] When using Sub-Scheme III, object detection is performed in the cloud.

[0216] I3) Cloud will aggregate voxel-level features Input a 3D target detection network model based on voxel-level features to obtain target detection results, and transmit the target detection results U to the autonomous vehicle.

[0217] The key technical points and advantages of this invention include:

[0218] Using roadside LiDAR as a supplement to the perception of autonomous vehicles improves the range and accuracy of their recognition of surrounding objects. Meanwhile, using voxelized features for data transmission between vehicles and the road ensures almost no loss of original data information while reducing bandwidth requirements during data transmission.

[0219] The above symbols and their meanings are summarized in the table below:

[0220]

[0221]

[0222]

[0223] The above terms and their meanings are summarized in the table below:

[0224]

[0225]

[0226] Brief description of the attached figures

[0227] Figure 1 This paper proposes a perception information fusion representation and target detection method for vehicle-road cooperation.

[0228] Figure 2 Schematic diagram of roadside mechanical rotating lidar deployment

[0229] Figure 3 Schematic diagram of roadside all-solid-state lidar deployment

[0230] Figure 4 Schematic diagram for deploying roadside all-solid-state lidar (two reverse lidars installed on the same pole).

[0231] Figure 5 Schematic diagram of point cloud data processing for VFE layer

[0232] Figure 6 A schematic diagram of voxelization feature extraction and aggregation.

[0233] Figure 7 A schematic diagram of target detection in the merged voxel point cloud.

[0234] Figure 8 Schematic diagram of point cloud coordinate transformation for roadside lidar

[0235] Figure 9 The diagram illustrates the comparison of target detection results (the left image shows the vehicle-road cooperative detection method proposed in this patent, and the right image shows the detection results of each high-confidence target directly). Detailed Implementation

[0236] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0237] This invention relates to a method for sensor information fusion representation and target detection oriented towards vehicle-road cooperation. It can be divided into three main steps:

[0238] The first step is the installation and initial calibration of the roadside lidar sensors.

[0239] The deployment of roadside lidar is determined based on the existing roadside pillar facilities and the type of lidar installed in the vehicle-road cooperative scenario. Existing roadside lidar is installed using poles or crossbars, specifically on roadside gantries, streetlights, traffic light poles, and other infrastructure pillars with power support.

[0240] For scenarios such as intersections, deploying a single roadside LiDAR with a detection range equal to or greater than the scenario's area, or including key areas within the scenario, is sufficient. For long-distance, large-scale, and complex scenarios such as expressways, highways, and industrial parks, it is recommended to follow the roadside LiDAR deployment guidelines outlined in this invention. This ensures that the roadside LiDAR coverage meets the scenario's full coverage requirements, meaning a single roadside LiDAR should supplement the blind spots of other roadside LiDARs within the scenario, achieving better vehicle-to-infrastructure (V2I) target detection. In V2I solutions, roadside LiDAR enhances the perception capabilities of autonomous vehicles, enabling them to obtain information such as the position, type, size, and direction of travel of targets around the vehicle relative to the vehicle. Therefore, the roadside LiDAR itself should also possess the strongest possible perception capabilities, including parameters such as the number of radar lines and sampling frequency, which should be no less than those of onboard LiDARs. Furthermore, to compensate for the vulnerability of onboard LiDARs to obstruction and to achieve data redundancy, the roadside LiDAR's perception range should ensure coverage of all frequently obstructed areas, and the detection line of the roadside LiDAR should be kept clear and unobstructed.

[0241] After installing the roadside lidar sensor, to calculate the relative pose between the roadside lidar and the vehicle-mounted lidar, the installation position and angle of the roadside lidar need to be calibrated, i.e., extrinsic parameter calibration, which obtains the coordinate position parameters and angular attitude parameters of the lidar relative to a certain reference coordinate system. First, at least four reflectivity feature points are selected as control points within the detection area of ​​the roadside lidar. Reflectivity feature points refer to points whose reflectivity differs significantly from surrounding objects, such as traffic signs and license plates. The purpose of selecting reflectivity feature points as control points is to facilitate the rapid identification of corresponding points in the point cloud data based on their position and reflection intensity, thereby quickly establishing the correspondence between multiple pairs of points in the point cloud and a coordinate in the reference coordinate system. The control points should be distributed as discretely as possible. Under the condition that the scene environment allows and the control points meet the following requirements, the more control points, the better the calibration effect. The requirements for control point selection include: they should be discretely distributed, and no three control points should be collinear; within the detection range of the roadside lidar, the selected control points should be as far away from the lidar as possible, typically greater than 50% of the lidar's maximum detection distance. If, due to scene limitations, it is difficult to select control points at 50% of the lidar's maximum detection distance, control points can be selected at less than 50%, but the number of control points should be increased. Subsequently, high-precision measuring instruments such as handheld high-precision RTK are used to measure the precise coordinates of the control points, and the corresponding coordinates are found in the roadside lidar point cloud. When a high-precision map file of the roadside lidar deployment scene is available, it is not necessary to use handheld high-precision measuring instruments; the coordinates of the corresponding feature points can be directly found in the high-precision map. Finally, a 3D registration algorithm is used to calculate the optimal value of the lidar extrinsic parameter vector, and the result is used as the calibration result. Commonly used 3D registration algorithms include the ICP algorithm and the NDT algorithm, among which the ICP algorithm is mainly used when applying lidar extrinsic parameter calibration. The basic principle of the ICP algorithm is to calculate the optimal matching extrinsic parameters from the target point set P (the set of coordinates of control points in the roadside lidar coordinate system) and the source point set Q (the set of coordinates of control points in the reference coordinate system) to minimize the error function.

[0242] The method used to calibrate the extrinsic parameters of the roadside lidar is not limited here, but it should be ensured that the calibration results include the sensor's three-dimensional world coordinates as well as pitch, yaw and roll angles for use in point cloud deflection in subsequent steps.

[0243] The second step is the processing and feature extraction of point cloud data from vehicle-side and roadside LiDAR.

[0244] In actual vehicle-road cooperative autonomous driving, the real-time world coordinates, pitch angle, yaw angle, and roll angle of the vehicle are first obtained based on the positioning module built into the autonomous driving system. Based on the vehicle's RTK positioning results and the extrinsic parameter calibration results of the roadside LiDAR, the relative pose of the autonomous vehicle with respect to the roadside LiDAR is calculated, and the roadside LiDAR point cloud data is deflected into the vehicle coordinate system.

[0245] Based on the spatial dimension of the vehicle-mounted LiDAR point cloud, the voxel size is designed, and the vehicle-mounted LiDAR is divided into voxels. For the deflection point cloud, the same voxel division method as the vehicle-mounted LiDAR point cloud is used to ensure that the spatial grid of the deflection point cloud completely overlaps with that of the vehicle-mounted LiDAR point cloud. Scattered data within the same voxel are grouped according to their voxel location in both the vehicle-mounted LiDAR point cloud and the expanded deflection point cloud. Due to the non-uniformity and sparsity of the points, the number of scattered data points in each voxel may not be the same; some voxels may have no scattered data points. To reduce computational burden and eliminate discrimination problems caused by inconsistent density, voxels with more than a certain threshold of scattered data points are randomly sampled. A threshold of 35 is recommended; this threshold can be appropriately reduced when there are fewer scattered data points in the point cloud data. This strategy saves computational resources and reduces the imbalance between voxels. See [link to documentation]. Figure 6 Two sets of point cloud data are divided into several discrete voxels using a fixed-size lattice. This is then expanded, and the feature vector of each voxel is calculated using the voxelization method described above. Taking the classic VoxelNet network model in 3D object detection algorithms as an example, several consecutive VFE layers are used to extract the feature vector of each voxel. This involves supplementing the system information by using the offset of each scattered data point within the voxel relative to the center, and then inputting the processed voxelized point cloud into cascaded consecutive VFE layers. A schematic diagram of the VFE layers processing the voxelized point cloud data is shown below. Figure 5 The VFE layer's processing logic is as follows: First, each augmented scatter plot data point is passed through a fully connected network to obtain point-level features. Then, the point-level features are subjected to max pooling to obtain voxel-level features. Finally, the voxel-level features are concatenated with the point-level features obtained in the previous step to obtain the concatenated point-level features. After cascading VFE layer processing, the final voxel-level features are obtained through fully connected layers and max pooling.

[0246] Because point clouds exist sparsely in space, many voxels lack scattered point data and therefore corresponding voxel-level features. Storing point cloud voxel-level features using a special structure can significantly compress data size, reducing the transmission difficulty when sending them to processing devices. One such special structure is a hash table, a data structure that allows direct access based on a key value. It accesses records by mapping key values ​​to a location in the table, thus speeding up the search. In this hash table, the hash key is the spatial coordinate of the voxel, and the corresponding value is the voxel-level feature.

[0247] The third step involves stitching and aggregating the voxel-level features of the vehicle-mounted lidar point cloud and the voxel-level features of the deflection point cloud to obtain aggregated voxel-level features, and then performing target detection.

[0248] Before data aggregation and stitching, the voxel-level features of the point cloud must first be compressed and transmitted to a computing device. This computing device can be a roadside computing device, an autonomous vehicle, or the cloud. When using Sub-Solution I, data aggregation, stitching, and subsequent processing are performed on the roadside computing device; when using Sub-Solution II, these processes are performed on the autonomous vehicle; and when using Sub-Solution III, they are performed in the cloud.

[0249] During data stitching and aggregation, since voxelization does not change the spatial relative position of the point cloud, the voxel-level features of the vehicle-mounted LiDAR point cloud can still be supplemented based on the voxel-level features of the deflected point cloud from the previous step. This allows for data stitching by aligning the voxel-level features of the vehicle-mounted LiDAR point cloud and the deflected point cloud based on the positions of their voxels in the autonomous vehicle's coordinate system. For data aggregation, if either the vehicle-mounted LiDAR point cloud voxel-level feature or the deflected point cloud voxel-level feature is empty, the voxel-level feature of the non-empty voxel-level feature is used as the aggregated voxel-level feature. For voxel-level feature vectors with the same spatial coordinates in both sets of data, max pooling is used to aggregate the feature vectors. For non-overlapping voxel-level feature vectors, the feature vector value of the non-empty voxel is retained.

[0250] The aggregate pixel-level features are input into the subsequent 3D object detection network model to obtain the detected target. See Figure 7 Taking the VoxelNet network model as an example, the concatenated data is input into the continuous convolutional layers of the VoxelNet network model to obtain spatial feature maps, and finally input into the RPN (Region Proposal Network) of the VoxelNet network model to obtain the final target detection results.

[0251] The present invention has the following key technical points and advantages:

[0252] Using roadside LiDAR as a supplement to the perception of autonomous vehicles improves the range and accuracy of the vehicle's recognition of surrounding objects. At the same time, using point cloud voxelized features as data for vehicle-to-infrastructure transmission ensures that almost no original data information is lost, while reducing the bandwidth requirements for data transmission.

[0253] An experimental scenario was set up at the intersection of the School of Transportation Engineering, Jiading Campus, Tongji University. In this scenario, a 6.4m high pole was placed every 20m along the road. The Innovusion Jaguar 300-line array LiDAR and the Ouster 128-line 360° LiDAR were used as roadside LiDARs. The Innovusion Jaguar 300-line array LiDAR has a vertical field of view of 40° and a maximum detection distance of 200m. The Ouster 128-line 360° LiDAR has a vertical field of view of 45° and a maximum detection distance of 140m. For autonomous vehicles, the Ouster 64-line 360° LiDAR was used as the vehicle-mounted LiDAR, installed horizontally at a height of 2m. The vehicle-mounted LiDAR was rigidly connected to the vehicle body, and the relative attitude and displacement between them remained constant. It was calibrated at the factory, and the position and angle of the vehicle-mounted LiDAR were corrected in real-time based on the vehicle's real-time displacement and deflection measured by the onboard RTK.

[0254] Example 1 is as follows:

[0255] (1) Deployment and calibration of roadside lidar sensors

[0256] Using only the Ouster 128-line 360° lidar, considering the size of the lidar itself, the installation height of the Ouster 128-line 360° lidar is 6.5m, and one is installed between every 5 poles. This complies with the deployment guidelines for roadside mechanical rotating lidar and roadside hybrid solid-state lidar.

[0257] Six reflectivity feature points were selected as control points within the lidar area. These six control points were located at the bases of the columns on both sides of the lidar mounting column at distances of 80m, 100m, and 120m, respectively. Due to the curvature of the road section, any three control points were ensured to be non-collinear. The precise coordinates of the control points were measured using a handheld RTK scanner, and the corresponding control point coordinates were matched in the lidar point cloud. The lidar was then calibrated using the ICP algorithm.

[0258] (2) Processing and feature extraction of point cloud data.

[0259] After the calibration work in (1), the position of the roadside lidar point cloud in the coordinate system of the autonomous vehicle can be obtained, such as Figure 8As shown, the roadside LiDAR point cloud is aligned to the autonomous vehicle coordinate system. The deflected point cloud is divided into voxels and expanded according to the autonomous vehicle coordinate system and a lattice of fixed size [0.4m 0.4m 0.5m], resulting in a voxelized deflected point cloud. After supplementing each scatter data point within the voxelized deflected point cloud with voxel mean information, it is input into a multi-layer VFE to calculate voxel-level features. Voxels without scatter data are not calculated. Each voxel is ultimately represented by a 128-dimensional feature vector. The roadside computing device stores the calculated voxel-level features in a hash table, using the spatial location of the voxel as the hash key, and the corresponding content as the voxel-level feature of the corresponding voxel, resulting in compressed deflected point cloud voxel-level features. The autonomous vehicle performs the same processing on the vehicle-mounted LiDAR point cloud until the vehicle-mounted LiDAR point cloud voxel-level features are obtained, meaning there is no need to build a hash table for the vehicle-mounted LiDAR point cloud data. At this point, compared to the original point cloud data, the data size is reduced to approximately 1 / 10.

[0260] (3) Data stitching, data aggregation and target detection of voxel-level features

[0261] The autonomous vehicle receives compressed, deflected point cloud voxel-level features sent by the roadside computing device and decompresses them to restore the deflected point cloud voxel-level features. Since the coordinate system of the received deflected point cloud voxel-level features has been deflected to the autonomous vehicle's coordinate system, these features can be directly concatenated with voxel-level features from the vehicle-mounted LiDAR point cloud in the same coordinate system. Max pooling is used to aggregate voxel-level features with the same coordinate system; for example, the aggregation result of voxel-level features [15,45,90, ...,17] and [8,17,110, ...,43] is [15,45,110, ...,43]. After concatenating and aggregating all voxel-level features, the data is input into the subsequent RPN to obtain the target detection result. The vehicle-road cooperative detection method proposed in this patent and the target detection results and confidence scores of directly fusing vehicle-mounted LiDAR point clouds and roadside LiDAR point clouds are plotted on a top-down view of the point cloud, as shown below. Figure 9 It is evident that using shared neural network features for vehicle-road cooperative target detection can significantly improve target detection accuracy and reduce data transmission bandwidth requirements.

[0262] Example 2 is as follows:

[0263] (1) Deployment and calibration of roadside lidar sensors

[0264] When using only the Innovusion Jaguar array 300-line lidar and installing only one lidar per pole, the lidar installation height is 6.5m, the depression angle is 7°, and one lidar is installed between every 8 poles. This conforms to the roadside all-solid-state lidar deployment scheme.

[0265] Six reflectivity feature points were selected as control points within the lidar area. These six control points were located at the bases of the columns on both sides of the lidar installation column, at distances of 100m, 120m, and 140m respectively. Due to the curvature of the road section, any three control points were ensured to be non-collinear. The precise coordinates of the control points were measured using a handheld RTK scanner, and the corresponding control point coordinates were matched in the lidar point cloud. The lidar was then calibrated using the ICP algorithm.

[0266] (2) Processing and feature extraction of point cloud data.

[0267] Similar to step (2) in Example 1, the voxel-level features of the deflection point cloud and the voxel-level features of the vehicle-mounted LiDAR point cloud are obtained. The autonomous vehicle stores the calculated voxel-level features of the vehicle-mounted LiDAR point cloud in a hash table, with the spatial location of the voxel as the hash key, and the corresponding content is the voxel-level feature of the corresponding voxel, thus obtaining the compressed voxel-level features of the vehicle-mounted LiDAR point cloud.

[0268] (3) Data stitching, data aggregation and target detection of voxel-level features

[0269] The roadside computing device receives the compressed voxel-level features of the vehicle-mounted LiDAR point cloud sent by the autonomous vehicle and decompresses them to restore the voxel-level features of the vehicle-mounted LiDAR point cloud. The subsequent data stitching, data aggregation and target detection steps are the same as in Example 1 (3). After obtaining the target detection result, the roadside computing device sends the target detection result to the autonomous vehicle.

[0270] Example 3 is as follows:

[0271] (1) Deployment and calibration of roadside lidar sensors

[0272] When using only the Innovusion Jaguar array 300-line lidar and installing two opposing lidars on each pole, the lidar installation height is 6.5m, the depression angle is 7°, and two are installed between every nine poles, which complies with the roadside all-solid-state lidar deployment guidelines.

[0273] Six reflectivity feature points were selected as control points within the lidar area. These six control points were located at the bases of the columns on both sides of the lidar installation column, at distances of 100m, 120m, and 140m respectively. Due to the curvature of the road section, any three control points were ensured to be non-collinear. The precise coordinates of the control points were measured using a handheld RTK scanner, and the corresponding control point coordinates were matched in the lidar point cloud. The lidar was then calibrated using the ICP algorithm.

[0274] (2) Processing and feature extraction of point cloud data.

[0275] The compressed deflection point cloud voxel-level features are obtained in step (2) of the same embodiment 1, and the compressed vehicle-mounted lidar point cloud voxel-level features are obtained in step (2) of the same embodiment 2.

[0276] (3) Data stitching, data aggregation and target detection of voxel-level features

[0277] The cloud receives the compressed voxel-level features of the vehicle-mounted LiDAR point cloud sent by the autonomous vehicle, and decompresses them to restore the voxel-level features of the vehicle-mounted LiDAR point cloud; the cloud receives the compressed deflection voxel-level features of the point cloud sent by the roadside computing device, and decompresses them to restore the deflection voxel-level features. The subsequent data stitching, data aggregation and target detection steps are the same as in Example 1 (3), until the target detection result is obtained, the cloud sends the target detection result to the autonomous vehicle.

[0278] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for sensory information fusion representation and target detection oriented towards vehicle-road cooperation, the method comprising the following steps: Preparation phase: A. Deploy roadside lidar and equip it with corresponding roadside computing devices; B. Calibrate the external parameters of the roadside lidar; Application phase: C. The roadside computing device calculates the relative pose of the autonomous vehicle with respect to the roadside lidar based on the autonomous vehicle's positioning data and the external parameters of the roadside lidar; D. The roadside computing device deflects the roadside lidar point cloud detected by the roadside lidar into the autonomous vehicle coordinate system according to the relative pose, and obtains the deflected point cloud. E. The roadside computing equipment performs voxelization on the deflection point cloud to obtain a voxelized deflection point cloud. The autonomous vehicle performs voxelization on the point cloud detected by the vehicle-mounted LiDAR to obtain a voxelized vehicle-mounted LiDAR point cloud. F. The roadside computing equipment calculates the voxel-level features of the voxelized deflection point cloud to obtain the voxel-level features of the deflection point cloud; the autonomous vehicle calculates the voxel-level features of the vehicle-mounted LiDAR point cloud to obtain the voxel-level features of the vehicle-mounted LiDAR point cloud. G. The autonomous vehicle compresses the voxel-level features of the on-board LiDAR point cloud to obtain compressed voxel-level features of the on-board LiDAR point cloud, and transmits them to the roadside computing device. The roadside computing device receives the compressed voxel-level features of the on-board LiDAR point cloud and restores them to the voxel-level features of the on-board LiDAR point cloud. H. The roadside computing device performs data stitching and data aggregation on the voxel-level features of the vehicle-mounted lidar point cloud and the voxel-level features of the deflection point cloud to obtain aggregated voxel-level features; I. The roadside computing device aggregates voxel-level features and inputs them into a 3D target detection network model based on voxel-level features to obtain target detection results, and then transmits the target detection results to the autonomous vehicle.

2. A method for sensory information fusion representation and target detection oriented towards vehicle-road cooperation, the method comprising the following steps: Preparation phase: A. Deploy roadside lidar and equip it with corresponding roadside computing devices; B. Calibrate the external parameters of the roadside lidar; Application phase: C. The roadside computing device calculates the relative pose of the autonomous vehicle with respect to the roadside lidar based on the autonomous vehicle's positioning data and the external parameters of the roadside lidar; D. The roadside computing device deflects the roadside lidar point cloud detected by the roadside lidar into the autonomous vehicle coordinate system according to the relative pose, and obtains the deflected point cloud. E. The roadside computing equipment performs voxelization on the deflection point cloud to obtain a voxelized deflection point cloud. The autonomous vehicle performs voxelization on the point cloud detected by the vehicle-mounted LiDAR to obtain a voxelized vehicle-mounted LiDAR point cloud. F. The roadside computing equipment calculates the voxel-level features of the voxelized deflection point cloud to obtain the voxel-level features of the deflection point cloud; the autonomous vehicle calculates the voxel-level features of the vehicle-mounted LiDAR point cloud to obtain the voxel-level features of the vehicle-mounted LiDAR point cloud. G. The roadside computing device compresses the voxel-level features of the deflection point cloud to obtain compressed voxel-level features of the deflection point cloud, and transmits them to the autonomous vehicle; the autonomous vehicle receives the compressed voxel-level features of the deflection point cloud and restores them to voxel-level features of the deflection point cloud. H. Autonomous vehicles perform data stitching and data aggregation on the voxel-level features of the onboard LiDAR point cloud and the voxel-level features of the deflected point cloud to obtain aggregated voxel-level features. I. Autonomous vehicles input aggregated voxel-level features into a 3D target detection network model based on voxel-level features to obtain target detection results.

3. A method for sensory information fusion representation and target detection oriented towards vehicle-road cooperation, the method comprising the following steps: Preparation phase: A. Deploy roadside lidar and equip it with corresponding roadside computing devices; B. Calibrate the external parameters of the roadside lidar; Application phase: C. The roadside computing device calculates the relative pose of the autonomous vehicle with respect to the roadside lidar based on the autonomous vehicle's positioning data and the external parameters of the roadside lidar; D. The roadside computing device deflects the roadside lidar point cloud detected by the roadside lidar into the autonomous vehicle coordinate system according to the relative pose, and obtains the deflected point cloud. E. The roadside computing equipment performs voxelization on the deflection point cloud to obtain a voxelized deflection point cloud. The autonomous vehicle performs voxelization on the point cloud detected by the vehicle-mounted LiDAR to obtain a voxelized vehicle-mounted LiDAR point cloud. F. The roadside computing equipment calculates the voxel-level features of the voxelized deflection point cloud to obtain the voxel-level features of the deflection point cloud. The autonomous vehicle calculates the voxel-level features of the vehicle-mounted LiDAR point cloud to obtain the voxel-level features of the vehicle-mounted LiDAR point cloud. G. The autonomous vehicle compresses the voxel-level features of the onboard LiDAR point cloud to obtain compressed voxel-level features of the onboard LiDAR point cloud, and transmits them to the cloud; the roadside computing device compresses the voxel-level features of the deflection point cloud to obtain compressed deflection point cloud voxel-level features, and transmits them to the cloud; the cloud receives the compressed deflection point cloud voxel-level features and the compressed onboard LiDAR point cloud voxel-level features, restores the compressed deflection point cloud voxel-level features to the deflection point cloud voxel-level features, and restores the compressed onboard LiDAR point cloud voxel-level features to the onboard LiDAR point cloud voxel-level features; H. The cloud performs data stitching and data aggregation on the voxel-level features of the vehicle-mounted LiDAR point cloud and the voxel-level features of the deflected point cloud to obtain aggregated voxel-level features. I. The cloud inputs the aggregated voxel-level features into the 3D target detection network model based on voxel-level features to obtain the target detection results, and then transmits the target detection results to the autonomous vehicle.

4. The method according to any one of claims 1 to 3, characterized in that, The configuration guidelines for roadside lidar are as follows: ① For roadside mechanical rotating lidar and two opposing solid-state lidars mounted on the same pole, the following conditions should be met at least: in: H represents the installation height of the lidar; θ 2 This indicates the angle between the highest elevation beam of the lidar and the horizontal direction; L represents the distance between two adjacent lidar mounting positions; ② For roadside all-solid-state lidar installed on the roadside, at least the following requirements should be met: in: H b Indicates the installation height of the roadside all-solid-state lidar; This indicates the field of view angle of the roadside all-solid-state lidar in the vertical direction; This indicates the angle between the highest elevation beam of the roadside all-solid-state lidar and the horizontal direction; L b This indicates the distance between two adjacent roadside solid-state lidar installation poles.

5. The method according to any one of claims 1 to 3, characterized in that, When calibrating the external parameters of the roadside lidar, the number, location discreteness, and collinearity of the control points are considered when selecting feature points as control points within the roadside lidar scanning area.

6. The method according to any one of claims 1 to 3, characterized in that, The extrinsic parameters of the roadside lidar are calibrated using the following method: the coordinates of the control points in the roadside lidar coordinate system and the coordinates in the reference coordinate system measured by RTK are respectively used as the target point set P and the source point set Q, and the lidar extrinsic parameters are calculated using the ICP algorithm.

7. The method according to any one of claims 1 to 3, characterized in that, The point cloud voxelization process involves expanding the deflected point cloud to ensure that the voxel mesh of the vehicle-mounted LiDAR point cloud (Dc) and the expanded deflected point cloud is consistent. The calculation formula is as follows: in: , The deflection point cloud extended in the K-dimensional dimension The range of starting and ending values; , Deflection point cloud in K dimension The range of starting and ending values; V K Let be the size of the voxel in the K-dimensional dimension.

8. The method according to any one of claims 1 to 3, characterized in that, When extracting voxel-level features from point clouds, the points are supplemented with information by an offset from the center, that is: in: This is the information for the i-th point in the supplemented voxel A; x i y i z i Let be the coordinates of the i-th point in voxel A; r i Let be the reflection intensity at the i-th point in voxel A; v x v y v z The mean of the coordinates of all points within voxel A.

9. The method according to any one of claims 1 to 3, characterized in that, The described voxel-level feature data aggregation method uses max pooling to aggregate voxel-level features with the same coordinates, and its formula is as follows: f k Features of aggregates at the element level The value at position k; f ego_k Voxel-level features of point clouds for vehicle-mounted LiDAR The value at position k; f lidar_k For deflection point cloud voxel-level features The value at position k.

Images