A road drivable area detection method based on fusion of laser radar and millimeter wave radar

Abstract

The present application relates to a kind of road drivable area detection methods based on laser radar and millimeter wave radar fusion, belong to vehicle-road cooperation and intelligent transportation field.The method is by adaptive DBSCAN clustering algorithm to process the point cloud data of laser radar, can improve the consistency in class and difference between classes of clustering result;By adaptive threshold segmentation method based on the law of the big to construct the drivable area electronic fence of road, can avoid the problem that traditional fixed global threshold segmentation method cannot consider the situation of point cloud chart everywhere and thus segmentation effect is poor;By fan-shaped encoding, point cloud coordinate system is converted into polar coordinate system to express the position and direction information of point, and by the selection of radius, the problem of different distance under different sparsity can be solved;By pseudo-image fusion method, attention mechanism is used to adjust weight dynamically to improve environmental adaptability.

Description

Technical Field

[0001] This invention belongs to the field of vehicle-road cooperation and intelligent transportation, and relates to a method for detecting drivable road areas based on the fusion of lidar and millimeter-wave radar. Background Technology

[0002] In recent years, the rapid development of new energy technologies, computer communication technologies, and other automotive-related fields has driven traditional automobiles towards intelligence, electrification, and connectivity. As people's living standards improve and their demands for vehicle performance increase, they are paying more and more attention to vehicle safety and ride comfort, and road environment perception is one of the keys to ensuring vehicle safety and ride comfort.

[0003] Road environment perception is a crucial element in the mature development of vehicle-road cooperative technology. The environment perception layer primarily utilizes various sensors to provide vehicles with traffic environment information, including lane markings, traffic lights, signs, obstacle outlines, and the position and relative distance between the vehicle and obstacles. It represents the primary challenge of vehicle-road cooperation, providing a basis for intelligent vehicles' global path planning, driving behavior decisions, and motion planning, and enabling vehicle control in conjunction with underlying actuators. Simply perceiving the road from the vehicle's perspective has significant limitations. Intelligent vehicle perception hardware systems, whether cameras or radar, are products based on biological senses and are all mounted on the vehicle, inevitably leading to "blind spots." Even the most intelligent systems can only make quick and accurate decisions within their visual range, resulting in phenomena like "ghost pedestrians" (suddenly appearing out of nowhere) that are difficult for human drivers to avoid, and equally difficult for single-vehicle intelligence to prevent. Furthermore, in adverse weather conditions and lane-changing scenarios, the lack of environmental perception by single vehicles can easily lead to traffic accidents.

[0004] Roadside cameras can perform some object recognition, classification, and detection functions well, but due to their lack of depth information, they cannot accurately determine the location and distance of vehicles or obstacles, and therefore cannot establish a more precise drivable road zone. Therefore, a strategy of fusing LiDAR and millimeter-wave radar is used to establish the drivable road zone. The data from LiDAR and millimeter-wave radar can complement each other, thereby improving detection accuracy. LiDAR provides high-precision point cloud data for detecting the shape and position of objects, while millimeter-wave radar provides high-precision velocity and distance measurements for detecting the motion state of targets. Fusing these two types of data allows for more accurate detection of obstacles and pedestrians on the road, and improves the robustness of roadside drivable area perception in different environments. For example, in rainy or snowy weather, LiDAR may be affected by interference and lose accuracy, while millimeter-wave radar can better adapt to such environments.

[0005] Road environment perception, as a fundamental general technology, plays a crucial role in autonomous driving and vehicle-to-everything (V2X) communication. However, to date, onboard intelligent road drivable area perception remains the mainstream solution. Future autonomous vehicles will need to perceive and acquire drivable road areas in real time; the limitations of single-vehicle perception prevent accurate acquisition of drivable areas, hindering the vehicle's ability to effectively generate local paths. Furthermore, in adverse weather conditions and lane-changing scenarios, the inadequacy of single-vehicle environmental perception can easily lead to traffic accidents.

[0006] Therefore, in order to address the shortcomings of single-vehicle systems and meet the needs of vehicle-road cooperation, providing road environment perception services that are not constrained by the environment, ensuring travel safety, and promoting collaborative development, a road drivable area detection technology based on the fusion of lidar and millimeter-wave radar is needed to provide accurate drivable area information for intelligent transportation scenarios under the Internet of Vehicles, thereby realizing vehicle-road cooperation and subsequent autonomous driving. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a road drivable area detection method based on the fusion of lidar and millimeter-wave radar, which improves the recognition rate of road drivable areas and adaptability to different weather conditions by fusing real-time data from lidar and millimeter-wave radar.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for detecting drivable road areas based on the fusion of lidar and millimeter-wave radar, the method comprising the following steps:

[0010] S1. To address the issue that excessive single-frame data of roadside laser point cloud increases the system burden by clustering all points, and that the widely used clustering method, which relies solely on point cloud spatial density to determine whether points belong to the same target, is difficult to adapt to different scene areas, a road area point cloud extraction and adaptive clustering method is designed to extract and cluster the point cloud data of the LiDAR.

[0011] Subsequently, an improved local adaptive threshold segmentation method based on the big rule is used to segment lane lines and road surfaces in the point cloud data, thereby constructing an electronic fence. The process is as follows: First, each scan line in the ground point cloud image is divided into equal-sized neighborhood blocks. Then, all neighborhood blocks are traversed. Finally, based on the reflection intensity distribution of different neighborhood blocks, the big rule is used to determine the optimal threshold for each neighborhood block, thus constructing the electronic fence for the road area. This improves the connection between points and their surroundings, increases the accuracy of threshold selection, obtains more precise road boundaries, and ultimately constructs a reasonable electronic fence.

[0012] S2. To address the problem that single-size voxels cannot adapt to uneven point density distribution at different distances in point clouds, which consumes several times the computational and memory resources, reduces the real-time performance of network detection, and cannot better integrate with millimeter-wave radar to identify object speeds, a fan-shaped point cloud and millimeter-wave coding method is designed, using fan-shaped coding to replace pillar coding.

[0013] S3. To address the issue of insufficient computing power of roadside equipment and the resulting detection distortion due to adverse weather conditions distorting LiDAR point clouds during target-level fusion, which fails to adapt to differences between sensor data, a PointPillar feature layer attention fusion model is designed. This model uses an attention mechanism to calculate the importance of different features in the current scene, then weights and fuses different features according to their importance to obtain the fused features. Finally, the concept of a residual module is used to obtain the final fusion result. Based on the fusion characteristics of LiDAR as the primary detection method and millimeter-wave radar as a secondary method, the weights are dynamically adjusted to improve the model's environmental adaptability.

[0014] Further, in step S1, the extraction and clustering methods are as follows: First, the raw point cloud data from the LiDAR is filtered to extract valid point cloud data; then, the region of interest is selected using the adaptive DBSCAN clustering algorithm; the parameters of the clustering algorithm are corrected using the sigmoid function. The parameter corrected using the sigmoid function is the growth radius of the cluster, and the correction relationship is as follows:

[0015] E′ ps =E ps ×f(r i )

[0016] In the formula, E′ ps E represents the corrected radius parameter. ps Indicates the initial radius parameter;

[0017]

[0018] In the formula, k r b r ε r Both r and r0 are parameters of the algorithm model. The optimal f(r) is obtained by enumerating different values ​​of the parameters. i );r i This indicates the seed point for searching the same cluster.

[0019] Further, in step S1, the improved local adaptive threshold segmentation method based on the big law method specifically involves: classifying the point cloud data obtained after extraction and clustering according to scan lines; performing grayscale conversion on the reflection intensity data within each scan line; calculating the global mean of grayscale values ​​within all scan lines; then finding grayscale values ​​greater than the global mean and calculating the mean of these grayscale values ​​to obtain the intra-class secondary grayscale mean; using the intra-class secondary grayscale mean as the initial threshold to define a threshold selection interval; within the threshold selection interval, there is a threshold th; the threshold th divides the image into foreground and background images; combining the probability and average grayscale value of the foreground and background images, and the global grayscale mean with grayscale values ​​greater than the intra-class secondary grayscale mean, the inter-class variance is calculated; the threshold th corresponding to the maximum inter-class variance is recorded as the optimal threshold; data with grayscale values ​​greater than the optimal threshold are marked as lane line point cloud data, and data with grayscale values ​​less than the optimal threshold are marked as road surface point cloud data.

[0020] The formula for calculating the inter-class variance is as follows:

[0021] σ 2 =P1(μ1-μ G ) 2 +P2(μ2-μ G ) 2

[0022] In the formula, σ 2 Let P1 represent the probability of the background image, P2 represent the probability of the foreground image, μ1 represent the average gray value of the background image, and μ2 represent the average gray value of the foreground image. G This represents the global grayscale mean value, where the grayscale value is greater than the average grayscale value within the class.

[0023] Further, in step S2, sector encoding is performed on both the millimeter-wave radar point cloud data and the lidar point cloud data processed in step S1. The process is as follows: Sector encoding is completed in polar coordinates. A circular detection region is set in the point cloud data, and the circular detection region is divided into several sector regions. Then, the sector regions are divided into several grids radially. The coordinate system of the points in the point cloud data is transformed into polar coordinates, and the points falling within the grid intervals are assigned to the grid, thereby generating polar coordinate cylinder elements in three-dimensional space. The diameter of the grid is equal to the inner arc length, where the diameter represents the radial length of the straight edge of the grid, and the inner arc length represents the length of the shorter arc edge in the grid. After generating polar coordinate cylinder elements in three-dimensional space, the point cloud cylinders are converted into pseudo-images and feature maps are extracted.

[0024] Furthermore, in step S3, after the feature maps of the LiDAR and millimeter-wave radar are input into the feature layer attention fusion model, they are first subjected to convolution and normalization processing respectively, and then attention fusion is performed:

[0025]

[0026] In the formula, W q W v W represents the weight matrix learned by convolution from the lidar feature map X1. k X represents the characteristic map of millimeter-wave radar. r The weight matrices learned through convolution are Q and V, representing the LiDAR matrix and K, respectively, and K, representing the millimeter-wave radar matrix. Multiplying the LiDAR matrix V and the millimeter-wave radar matrix K by matrix multiplication, and then multiplying the result by the LiDAR matrix Q, yields the fusion result O. Multiplying the fusion result O by a scaling factor and then summing it with the original LiDAR feature map X1 gives the final fusion result.

[0027] The beneficial effects of this invention are as follows: By fusing data from lidar and millimeter-wave radar, this invention combines the advantages of both types of data to achieve more accurate detection of drivable areas and improves the robustness of roadside drivable area perception in different environments. In addition, for the fusion of the two types of data, this invention designs a pseudo-image fusion method and uses an attention mechanism to dynamically adjust the weights during fusion, thereby improving the environmental adaptability of the recognition model.

[0028] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0030] Figure 1 This is a structural diagram of the method of the present invention;

[0031] Figure 2 This is a schematic diagram of the through-pass filtering process;

[0032] Figure 3 This is a schematic diagram of the overall structure of the local adaptive threshold segmentation algorithm based on the big law.

[0033] Figure 4 A schematic diagram of point cloud and millimeter-wave coding based on a fan shape;

[0034] Figure 5 This is a schematic diagram of the attention fusion model structure based on the pointpillar feature layer. Detailed Implementation

[0035] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0036] Please see Figures 1-5 This paper presents a method for detecting drivable road areas based on the fusion of lidar and millimeter-wave radar. The method first processes the data acquired by lidar and millimeter-wave radar, and then uses a pointpillar feature layer attention fusion model to fuse the two radar data to detect the drivable roadside area.

[0037] like Figure 2 As shown, for LiDAR-acquired data, this method filters out points outside the corresponding height or radius based on the characteristics of the roadside point cloud to obtain the effective point cloud, and further selects the region of interest (ROI). The effective point cloud set P obtained through radius filtering and height filtering is... pth The following formula must be satisfied:

[0038]

[0039] In the formula, (x i ,y i ,z i P is the effective point cloud set. pth The element in r th h is the radius threshold. th Let x1, x2, y1, and y2 be the height thresholds, and let x1, x2, y1, and y2 be the coordinate ranges in the x and y directions of the road area, respectively, with the x-axis direction being the same as the road direction. Considering that roadside point cloud data has certain blind spots, and that distant point clouds cannot provide sufficient features for object detection, point clouds within a certain range at close range and beyond a certain distance are less significant for clustering and object detection. The point cloud in the central region, from near to far, is the core area for clustering and object detection, requiring adaptive adjustment of the clustering algorithm parameters. The sigmoid function is chosen as the correction coefficient for the density reaching the search radius during cluster growth.

[0040]

[0041] In the formula, k r b r ε rr0 and r0 are the parameters of the algorithm model. The optimal f(r) is obtained by enumerating different values ​​of the parameters. i ), through f(r i The growth radius of the cluster is corrected.

[0042] The sigmoid function shows little change in the smaller and larger ranges, but increases rapidly in the middle range. This is highly advantageous for grouping point cloud clusters belonging to the same target, and it can easily process point cloud data collected by roadside lidar, which has characteristics such as near-range blind spots, no effective features at long distances, and increasing sparsity with distance. The formula for correcting the cluster growth radius is:

[0043] E′ ps =E ps ×f(r i (3)

[0044] In the formula, E′ ps E is the corrected radius parameter. ps r is the initial radius parameter. i To search for seed points of the same cluster, a set of points belonging to the same object is obtained. While being adaptive, this avoids losing points outside the search radius.

[0045] After the roadside point cloud is extracted and clustered, it is segmented using an improved local adaptive threshold segmentation method based on the large law to construct an electronic fence. For example... Figure 3 The diagram shows the overall structure of the improved local adaptive threshold segmentation method based on the large-scale law, which uses adaptive threshold selection of grayscale values ​​for segmentation. In the road surface point cloud extraction and roadside data filtering steps, the final road surface point cloud data is classified according to scan lines. Based on this, the grayscale conversion formula for the reflection intensity data within each scan line is as follows:

[0046]

[0047] In the formula, grayvalue j The intensity is the grayscale value of point j at the midpoint of scan line i after grayscale conversion. j max_intensity represents the reflection intensity within the scan line. i The value representing the maximum reflection intensity within the line is grayscale.

[0048] Calculate the global mean of gray values ​​within all scan lines:

[0049]

[0050] In the formula, clustersize iLet n be the number of points in the scan line, and n represent the number of scan lines. Equation (5) yields the global mean of each scan line after grayscale conversion of the reflection intensity. Find all grayscale values ​​greater than the global mean, calculate the mean ave1 of these grayscale values, and then calculate the probability of each grayscale level:

[0051]

[0052] In the formula, num represents the number of points in the point cloud data.

[0053] The intensity of lane line data is significantly higher than that of road surface data. Furthermore, after filtering out roadside data, the data points carrying the highest intensity in each scan line are essentially the lane line point cloud data. If the threshold selection still starts from 0, it will result in a large amount of useless computation, unnecessarily extending the processing time. Therefore, the intra-class quadratic gray-level mean ave1 is used as the initial threshold, narrowing the threshold selection range to [ave1, 255]. A threshold th divides all pixels in the image into two classes: foreground images with gray-level values ​​greater than th and background images with gray-level values ​​less than th. After setting the initial threshold, the probabilities P1 and P2 of the background and foreground images, the average gray-level values ​​v1 and v2 of the background and foreground images, and the global gray-level mean v for gray-level values ​​greater than ave1 are calculated. G :

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] Finally, the inter-class variance was calculated.

[0060]

[0061] By comparing the variances of all classes within a class and finding the maximum inter-class variance and its corresponding threshold th, the optimal threshold is determined. The grayscale values ​​of the data within the scan line are then filtered. Data with grayscale values ​​greater than the optimal threshold are marked as candidate lane line point cloud data, while data with grayscale values ​​less than the optimal threshold are marked as road surface point cloud data. This process yields the electronic fence.

[0062] To address the issue of varying point sparsity at different distances, a sector-based voxel encoding processing module is designed. By converting the data into polar coordinates through sector encoding, the position and orientation information of the points can be expressed. Furthermore, by selecting the appropriate radius, the problem of varying sparsity at different distances can be resolved.

[0063] The data from lidar and millimeter-wave radar are encoded using a fan-shaped point cloud and millimeter-wave encoding method. First, the inner diameter R is set according to the installation location of the roadside lidar. in To outer diameter R out The circular detection area is defined, and the angle θ of each sector is determined. Then, based on angle θ, the circular detection range is divided into m (m = 2π / θ) sector regions. Each sector region is further divided radially into n grids, such as... Figure 4 As shown, while maintaining the grid's diameter r i Equal to the inner arc length l i The length of the first inner arc, l1, is represented as:

[0064]

[0065] The length of the i-th inner arc is represented as:

[0066]

[0067] Sector encoding divides the point cloud space into an (m, n) grid, where m represents the number of sectors and n represents the number of cells per sector. Because sector encoding is performed in polar coordinates, the planar polar coordinates of the points need to be calculated before assigning voxels to the point cloud.

[0068]

[0069] Points in the Cartesian coordinate system are converted to polar coordinates. After the point cloud data is converted to polar coordinates, the points falling within the polar coordinate range are assigned to a polar coordinate grid. Polar coordinate cylinders are generated in 3D space, and the polar coordinate cylinder containing a point is determined based on the coordinates (ρ, θ). After assigning the point cloud cylinders, the same method as PointPillar is used to convert them into pseudo-images, and point cloud object detection is performed using the same feature extraction and object detection methods.

[0070] The feature maps of LiDAR and millimeter-wave radar are fused using a pointpillar-based feature layer attention fusion model, such as... Figure 5 As shown. First, the feature maps of the LiDAR and millimeter-wave radar are made to the same dimension. Before fusion, both are subjected to 1×1 convolution and normalization. The attention fusion formula for LiDAR and millimeter-wave radar is expressed as:

[0071]

[0072] Among them, W q W v W is the weight matrix learned from the LiDAR feature map X1 through a 1×1 convolution. k It is a millimeter-wave radar feature map X r The weight matrices learned through 1×1 convolution yield LiDAR matrices Q and V, and the millimeter-wave radar matrix K. Multiplying LiDAR matrix V and millimeter-wave radar matrix K, and then multiplying the result with LiDAR matrix Q, yields the fused result O. This alters the numerical distribution of the LiDAR matrices, helping them focus their attention on positive target samples, thus performing attention correction. The attention-corrected LiDAR matrix is ​​then summed with the feature map of the original input LiDAR.

[0073] y=X1+λO (17)

[0074] In the formula, λ is the scaling factor. Borrowing from the concept of residual modules, the fusion result O is multiplied by the scaling factor λ and the LiDAR feature map is added to obtain the final output result y. The initial value of λ is set to 0. This weighting factor is increased through training. Its physical meaning can be regarded as the influence of the attention mechanism being 0 at the beginning, and the influence of attention in the output gradually increasing as training progresses.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for detecting drivable road areas based on the fusion of lidar and millimeter-wave radar, characterized in that: The method includes the following steps: S1. Extract and cluster the point cloud data of the LiDAR, and then use an improved local adaptive threshold segmentation method based on the big law to segment the lane lines and road surface in the point cloud data, thereby constructing an electronic fence. The extraction and clustering methods are as follows: First, the raw point cloud data from the LiDAR is filtered to extract the effective point cloud data. Then, the region of interest is selected using the adaptive DBSCAN clustering algorithm. The growth radius of the clusters in the clustering algorithm is corrected using the sigmoid function, and the correction relationship is as follows: In the formula, This represents the corrected radius parameter. Indicates the initial radius parameter; In the formula, , b r , and These are all parameters of the algorithm model; the optimal value is obtained by enumerating different parameter values. ; This indicates the seed point for searching the same cluster. The improved local adaptive threshold segmentation method based on the big law method is as follows: After extraction and clustering, the point cloud data is classified according to scan lines. The reflection intensity data within each scan line is converted to grayscale. The global mean of grayscale values ​​within all scan lines is calculated. Then, grayscale values ​​greater than the global mean are identified, and the mean of these grayscale values ​​is calculated to obtain the intra-class secondary grayscale mean. The intra-class secondary grayscale mean is used as an initial threshold to define a threshold selection interval. Within this threshold selection interval, a threshold value is defined. th threshold th The image is divided into foreground and background images. The inter-class variance is calculated by combining the probability and average gray value of the foreground and background images, as well as the global gray value whose gray value is greater than the quadratic gray value within each class. The threshold corresponding to the maximum inter-class variance is then determined. th The optimal threshold is defined as the gray value. Data with a gray value greater than the optimal threshold is marked as lane line point cloud data, and data with a gray value less than the optimal threshold is marked as road surface point cloud data. S2. Perform sector coding on the millimeter-wave radar point cloud data and the lidar point cloud data processed in step S1 respectively. The space is divided into grids by sector coding, and polar coordinate cylinder elements are generated in three-dimensional space. Then, the point cloud cylinders are converted into pseudo-images and feature maps are extracted. S3. The feature maps of LiDAR and millimeter-wave radar are weighted and fused using a pointpillar feature layer attention fusion model. The fusion result is multiplied by a scaling factor and then added to the LiDAR feature map to obtain the detection result of the road drivable area.

2. The method for detecting drivable road areas according to claim 1, characterized in that: In step S1, in the improved local adaptive threshold segmentation method based on the large law method, the formula for calculating the inter-class variance is: In the formula, Represents the variance between classes. Represents the probability of the background image. Represents the probability of the foreground image. This represents the average grayscale value of the background image. This represents the average gray value of the foreground image. This represents the global grayscale mean value, where the grayscale value is greater than the average grayscale value within the class.

3. The method for detecting drivable road areas according to claim 1, characterized in that: In step S2, the sector encoding is completed in polar coordinates. The steps are as follows: a circular detection area is set in the point cloud data, the circular detection area is divided into several sector areas, and then the sector areas are divided into several grids along the radial direction; the coordinate system of the points in the point cloud data is converted into polar coordinates, and the points falling within the grid interval are assigned to the grid, thereby generating polar coordinate cylinder elements in three-dimensional space; the radial length of the grid is equal to the inner arc length of the grid.

4. The method for detecting drivable road areas according to claim 1, characterized in that: In step S3, after the feature maps of the LiDAR and millimeter-wave radar are input into the feature layer attention fusion model, they are first subjected to convolution and normalization processing, and then attention fusion is performed: In the formula, , W v These represent the feature maps of the lidar. X 1. The weight matrix learned through convolution. W k Characteristic map of millimeter-wave radar X r The weight matrix learned through convolution, Q , V These represent the lidar matrix, K Represents a millimeter-wave radar matrix; represents a lidar matrix. V With millimeter-wave radar matrix K Perform matrix multiplication, and then multiply the result with the lidar matrix. Q The fusion result is obtained by performing matrix multiplication. O .

Images