A method for estimating the slope of the front road based on multi-source information fusion

Abstract

A method for estimating the slope of the road ahead based on multi-source information fusion. It relates to the field of intelligent driving technology, specifically to a method for estimating the slope of the road ahead based on multi-source information fusion technology. In the present invention, an improved Patchwork++ model is used for road segmentation to exclude the background and interfering point clouds, extract the effective ground point cloud data information, and calculate the road slope value based on the ground point cloud data information. The method includes the following steps: synchronize the time and register the space of the camera and the lidar to obtain a multi-source information data set; construct and improve the Patchwork++ model to obtain an improved Patchwork++ model: process the multi-source information data set through the improved Patchwork++ model to obtain the coordinates #imgabs0# of the ground point cloud; calculate the road slope value according to the coordinates #imgabs1# of the ground point cloud.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving, and particularly to a technical field of a method for estimating the slope of a forward road based on multi-source information fusion. Background Art

[0002] As a core technology for intelligent driving environment perception, the accuracy of road slope estimation directly determines vehicle power distribution, energy recovery efficiency, and driving safety margins. Early slope analysis relied on manual measurement or topographic map calculation, which had problems such as low efficiency, strong subjectivity, and insufficient accuracy. With the development of intelligent driving technology, vehicles need to obtain the road slope in real time to optimize power distribution and braking control. To improve accuracy and robustness, in recent years, heterogeneous data fusion solutions based on cameras and lidar have been studied, which solve the perception blind spots of traditional methods in complex scenarios and improve the road slope estimation ability.

[0003] Existing slope estimation solutions include methods based on dynamic models and Kalman filtering, machine learning and data-driven methods, geographic information systems and remote sensing technology methods, and multi-sensor fusion methods. The following is a comparison of several solutions:

[0004] For the method based on dynamic models and Kalman filtering, a state space model is established through the vehicle longitudinal dynamics equation, and the Kalman filter is used to iteratively optimize the slope estimation value. It has strong real-time performance, but parameter drift leads to error accumulation, and the accuracy decreases under non-steady-state conditions.

[0005] For machine learning and data-driven methods, a feedforward neural network is used to process multi-source sensor data, combined with an extended state observer for dynamic correction, which has strong adaptability, but requires a large amount of labeled data for training, resulting in high costs.

[0006] For geographic information systems and remote sensing technology methods, the regional slope is calculated through spatial analysis based on a digital elevation model, which is highly efficient, but has poor real-time performance and is not suitable for dynamic vehicle control.

[0007] For multi-sensor fusion methods, such as fusing lidar and cameras, ground features are extracted through point cloud segmentation and image semantic analysis. Cross-modal complementarity improves the segmentation accuracy in complex scenarios, effectively solves traditional blind spot problems such as snow cover, and provides a basis for slope-friction coefficient mapping.

[0008] In the doctoral dissertation of the University of Chinese Academy of Sciences, "Research on Key Technologies of Multi-Sensor 3D Environmental Perception System for Autonomous Driving" (Wang Jiarong), a multi-sensor 3D object detection method MCF3D (Multi-Stage Complementary Fusion for Multi-sensor 3D Object Detection) model based on multi-stage complementary fusion is proposed, which can be used to fuse 3D point clouds and RGB images; the MCF3D model includes: data-level fusion, feature-level fusion and decision-level fusion.

[0009] The multi-sensor fusion scheme is significantly superior to other schemes in terms of accuracy, environmental robustness and functionality. Based on multi-sensor data fusion, the present invention proposes a method for estimating the slope of the front road, effectively solving the pain points existing in the traditional slope estimation scheme, and being more sufficient, more efficient and more accurate in estimating the slope of the front road compared with the existing fusion scheme. Summary of the Invention

[0010] Aiming at the problems of large error, low accuracy, high training cost and poor real-time performance of the existing slope estimation methods, the present invention proposes a method for estimating the slope of the front road based on multi-source information fusion. The improved Patchwork++ model is used for road segmentation, excluding background and interfering point clouds, extracting ground effective point cloud data information, and calculating the road slope value based on the ground point cloud data information.

[0011] The method includes the following steps:

[0012] S1. Synchronize the time and register the space of the camera and the lidar to obtain a multi-source information data set;

[0013] S2. Construct and improve the Patchwork++ model to obtain an improved Patchwork++ model:

[0014] S21. Improve the Reflection Noise Removal (RNR) module in the Patchwork++ model: introduce the BiSeNet V2 module and the lossless fine fusion module into the reflection noise removal module;

[0015] S22. After the Cuboid Zone Merging (CZM) module of the Patchwork++ model, introduce the Inverse Perspective Mapping (IPM) module;

[0016] S23. Introduce a comprehensive weight into the Region-based Vertical Plane Fitting (R-VPF) module of the Patchwork++ model;

[0017] Introduce the Histogram of Oriented Gradients (HOG) module for feature extraction and the Support Vector Machine (SVM) module for classification into the geometric constraint filtering module of the Region-based Vertical Plane Fitting (R-VPF) module of the Patchwork++ model.

[0018] S24. In the time ground recovery (TGR) module of the Patchwork++ model, only retain the TGR time reduction mechanism, and use the gradient boosting decision tree (GBDT) module to replace the binary judgment module. At the same time, introduce a joint feature construction module before the gradient boosting decision tree (GBDT) module;

[0019] S25. In the adaptive ground likelihood estimation (A-GLE) module of the Patchwork++ model, use an energy function optimization module to replace the single threshold judgment module, and introduce a dynamic filtering and anti-noise structure after the energy function optimization module;

[0020] S26. Introduce a spatial recovery enhancement module after the adaptive ground likelihood estimation (A-GLE) module in the Patchwork++ model;

[0021] S3. Process the multi-source information dataset through the improved Patchwork++ model to obtain the coordinates of the ground point cloud ;

[0022] S4. Calculate the road slope value according to the coordinates of the ground point cloud , calculate the road slope value.

[0023] Furthermore, the specific method for synchronizing the time and spatially registering the camera and lidar is as follows: On the computer side, drive the camera and lidar sensors based on the ROS platform, and unify the sensor sampling moments through an external trigger signal to achieve time synchronization;

[0024] Independently calibrate and jointly calibrate the lidar and the camera, calculate the internal parameter matrix of each sensor and the external parameter matrix between the sensors; Align the 3D point cloud of the lidar and the pixels of the RGB image of the camera to achieve spatial registration.

[0025] Furthermore, the function of the BiSeNet V2 module is: segment the RGB image to obtain the probability maps of the road surface and non-road surface;

[0026] The function of the lossless fine fusion module is: project the 3D point cloud onto the RGB image plane and suppress the intensity of high-reflection points in the non-ground area.

[0027] Furthermore, the comprehensive weight is specifically: , and both represent undetermined coefficients, , represents the reflection intensity weight, represents the distance weight; The function of the comprehensive weight is: point cloud reflection weight allocation and probability sampling optimization.

[0028] Further, the joint feature construction module sequentially performs image superpixel segmentation, point cloud geometric feature extraction, and joint feature construction from input to output.

[0029] Further, the energy function optimization module constructs a CRF four - element energy function; the CRF four - element energy function includes a unary term, a flatness constraint term, a smooth term, and a reflection intensity constraint term.

[0030] The dynamic filtering and noise - resistant structure dynamically adjusts the reflection intensity variance threshold through the formula: where represents the reflection intensity variance threshold for the th adjustment, represents the critical proportion threshold of the water - logged / snow - covered mask area, represents the control threshold, represents the proportion of the water - logged area, represents the area of the water - logged area, represents the area of the complete area.

[0031] Further, the spatial restoration and enhancement module specifically: first detects the low - density area, and then realizes cross - modal point cloud completion through GAN.

[0032] Further, calculating the road slope value based on the coordinates of the ground point cloud is achieved through the RANSAC algorithm, specifically:

[0033] S801. Initialize the RANSAC parameters;

[0034] S802. Randomly sample and fit the plane equation: where represents the normal vector and represents the constant term of the plane equation;

[0035] S803. Calculate the road slope value according to the formula:

[0036] The beneficial effects of the method of the present invention are as follows:

[0037]

[0038] (1) The method of the present invention uses the improved Patchwork++ model to simultaneously process 3D point clouds and RGB images. The improved Patchwork++ model has a segmentation technology with high precision, low computational complexity, and strong noise - resistant ability, breaking through the technical bottleneck that the existing Patchwork++ model cannot efficiently process 3D point clouds and RGB images simultaneously. Moreover, the method of the present invention has strong real - time performance and is suitable for dynamic vehicle control.

[0038] (2) The method of the present invention uses RANSAC plane fitting for slope estimation, significantly improving the accuracy and robustness of slope estimation. Description of the Drawings

[0039] Figure 1 It is the flowchart of the method described in Embodiment 1 of the present invention;

[0040] Figure 2 It is the schematic diagram of the process of the improved Patchwork++ model for processing multi-source information datasets described in Embodiment 1 of the present invention;

[0041] Figure 3 It is the schematic diagram of the structural comparison before and after the improvement of the regional vertical plane fitting R-VPF module described in Embodiment 1 of the present invention;

[0042] Figure 4 It is the schematic diagram of the structural comparison before and after the improvement of the time ground recovery TGR module described in Embodiment 1 of the present invention;

[0043] Figure 5 It is the schematic diagram of the structural comparison before and after the improvement of the adaptive ground likelihood estimation A-GLE module described in Embodiment 1 of the present invention;

[0044] Figure 6 It is the flowchart of the method for calculating the road slope value by the RANSAC algorithm described in Embodiment 1 of the present invention;

[0045] Figure 7 It is the schematic diagram of data-level fusion described in Embodiment 2 of the present invention. Detailed Embodiments

[0046] Next, the technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0047] This embodiment provides a method for estimating the slope of the road ahead based on multi-source information fusion. The flowchart of the method is as Figure 1 shown, and the method includes the following steps:

[0048] S1. Synchronize the time and register the space of the camera and the lidar to obtain a multi-source information dataset;

[0049] The relevant operations of step S1 will be introduced with specific examples:

[0050] The time synchronization and spatial registration of the camera and lidar are specifically as follows: On the computer side, the camera and lidar sensors are driven based on the ROS platform. In the external trigger mode, the time sampling synchronization of the monocular camera and lidar sensor is achieved in the embedded intelligent control board.

[0051] The lidar and camera are independently calibrated and jointly calibrated to calculate the internal parameter matrix of each sensor and the external parameter matrix between the sensors. The spatial alignment of the lidar 3D point cloud and the camera RGB image pixels is performed to achieve the spatial registration of heterogeneous sensors.

[0052] The spatial registration of the heterogeneous sensors includes the following steps:

[0053] S101. Define four common coordinate systems: the world coordinate system , the camera coordinate system , the image coordinate system and the pixel coordinate system , and calibrate the camera.

[0054] S102. Establish the conversion matrix between each pixel point in the RGB image and the surface point of the spatial object, so that the RGB image and the 3D point cloud are represented in the same coordinate system.

[0055] S103. Given any point, find the corresponding image pixel point according to the internal and external parameter matrices to complete the spatial alignment and registration of the lidar 3D point cloud to the camera RGB image 2D pixels.

[0056] Specifically, it is achieved through the following formulas (1) and (2):

[0057] (1)

[0058] (2)

[0059] where represents the 3D point cloud coordinate in the lidar coordinate system, represents the camera coordinate corresponding to , represents the external parameter matrix for transforming from the lidar coordinate system to the camera coordinate system, which consists of the rotation matrix and the translation matrix , represents the 2D projection matrix for transforming the 3D point cloud in the camera coordinate system into the RGB image in the camera coordinate system.

[0060] S2. Build and improve the Patchwork++ model to obtain the improved Patchwork++ model:

[0061] S21. Improve the PNR module for reflection noise removal in the Patchwork++ model: introduce the BiSeNet V2 module and the lossless fine fusion module into the reflection noise removal module;

[0062] S22. After the CZM module for spatial partitioning in the Patchwork++ model, introduce the inverse perspective mapping (IPM) module;

[0063] S23. Introduce comprehensive weights into the regional vertical plane fitting (R-VPF) module in the Patchwork++ model;

[0064] Introduce the feature extraction HOG module and the classifier SVM module into the geometric constraint filtering module of the regional vertical plane fitting (R-VPF) module in the Patchwork++ model;

[0065] S24. In the time ground recovery (TGR) module of the Patchwork++ model, only retain the TGR time reduction mechanism, replace the binary judgment module with the gradient boosting decision tree (GBDT) module, and introduce a joint feature construction module before the gradient boosting decision tree (GBDT) module;

[0066] S25. In the adaptive ground likelihood estimation (A-GLE) module of the Patchwork++ model, replace the single threshold judgment module with an energy function optimization module, and introduce a dynamic filtering and anti-noise structure after the energy function optimization module;

[0067] S26. Introduce a spatial recovery enhancement module after the adaptive ground likelihood estimation (A-GLE) module in the Patchwork++ model.

[0068] Introduce the relevant operations of step S2 with a specific example:

[0069] As Figure 2 shown, the improved reflection noise removal module is named the multi-modal joint denoising module.

[0070] In the original reflection noise removal (RNR) module of Patchwork++ (eliminating noise points based on the LIDAR reflection model), parallelly add the RGB semantic segmentation model: the BiSeNet V2 module, and perform pixel (lightweight)-level classification on the RGB image through the BiSeNet V2 module to output the road surface / non-road surface probability map.

[0071] The BiSeNet V2 module first uniformly scales the image to a fixed resolution, maintains the aspect ratio to avoid distortion, and normalizes it according to the mean and variance of ImageNet. Secondly, it adjusts the output layer classification to a binary classification (road surface / non-road surface) and outputs a single-channel probability map. Then, it performs model training and fine-tuning, outputs an RGB image, and outputs a single-channel probability map (the value range ), which represents the road surface probability. At the same time, it binarizes the probability map with a default threshold of 0.5 and uses DenseCRF (Dense Conditional Random Field) to refine the boundaries, combining color similarity and spatial smoothness.

[0072] Then, cross-modal noise suppression is carried out: the lossless fine fusion module is used to project the 3D point cloud onto the RGB image plane, align the pixel and point cloud coordinates, and suppress the intensity of the point cloud in the non-road surface area with a reflection intensity > 0.8 to avoid high-reflection noise interfering with ground detection. Calculate the variance or curvature of the local point cloud surface normal to quantify geometric irregularities.

[0073] Perform Canny edge detection on the corresponding image patch and count the proportion of edge pixels.

[0074] Weightedly fuse roughness and edge density to identify outliers.

[0075] As Figure 2 shown, the spatial partitioning CZM module and the IPM module are combined and named the cross-modal concentric circle partitioning module.

[0076] The spatial partitioning CZM module of the Patchwork++ model (divides the point cloud into multiple fan-shaped regions) defines the coordinate system and sets parameters, establishes a polar coordinate system with the vehicle as the center, and divides the BEV space into three annuli: the inner annulus (inner annulus radius ), the middle annulus (middle annulus radius ), and the outer annulus (outer annulus radius ), and partitions them at fixed angular intervals to enhance direction perception, unify the grid resolution of the RGB image and the 3D point cloud, and increase the height limit of the ground point cloud. In this embodiment, , , ;

[0077] Next, perform point cloud annulus partitioning: through formulas (3) and (4), convert the lidar point cloud from the Cartesian coordinate system to the vehicle coordinate system, project it onto the BEV plane ( where represents the height of the BEV plane), and remove non-ground points whose height exceeds the ground range.

[0078] (3)

[0079] (4)

[0080] Among them, , successively represent the inner ring, the middle ring, and the outer ring, represents the center coordinate of the th ring in the BEV plane, represents the horizontal azimuth angle of the point cloud in the th ring relative to the vehicle's due front (positive X-axis direction);

[0081] According to the radius , divide the point cloud into the inner, middle, and outer rings, and divide each ring band into sector blocks to enhance the direction perception ability;

[0082] After CZM partitioning, convert the RGB image to a BEV view through IPM (Inverse Perspective Mapping): Define BEV control points, through the corresponding points on the image plane and the BEV plane , among which, represents the BEV control point coordinates. Through formula (5), calculate the homography matrix H and verify the matrix accuracy;

[0083] (5)

[0084] Use the homography matrix H to perform perspective transformation on the image to generate a BEV view.

[0085] Finally, perform adaptive grid partitioning: Design the grid size, divide the point cloud within each ring band according to the grid size, and count the point cloud attributes within the grid; Divide the image BEV into blocks according to the same grid size, and extract local features (color histogram, texture, edge density, etc.) to ensure that the image grid corresponds one-to-one with the point cloud grid.

[0086] As Figure 2 and 3 shown, the improved regional vertical plane fitting R-VPF module is named the multi-modal plane fitting module.

[0087] First, perform weight assignment (comprehensive weight , reflection intensity weight and distance weight ): Normalize the point cloud reflection intensity (Intensity). The higher the reflection intensity, the lower the weight. Points closer to the vehicle are given higher weights. The comprehensive weight is as shown in formula (6):

[0088] (6)

[0089] Among them, and both represent undetermined coefficients, , represents the reflection intensity weight, and the

[0090] reflection intensity weight is shown in formula (7):

[0091] (7)

[0092] (8)

[0093] represents the maximum value of the point cloud reflection intensity, represents the distance weight; the distance weight is shown in formula (8).

[0094] Perform probability sampling according to the weight distribution, calculate the initial plane equation through the RANSAC algorithm, optimize the plane parameters using the weighted least squares method, and retain the ground with the highest score as the final ground model through repeated sampling and scoring functions;

[0095] Then, perform HOG feature verification on the out-of-plane point cloud: after aligning the point cloud with the image, convert the image to grayscale and enhance the contrast using adaptive histogram equalization; calculate the horizontal and vertical gradients through the Sobel filter , calculate the gradient magnitude and direction of each pixel according to formulas (9) and (10), and discretize the direction into 9 intervals (each

[0096] (9)

[0097] (10)

[0098] Divide the image block into 8×8 cells, calculate the 9-direction gradient histogram for each cell, form blocks of 2×2 cells, perform L2 normalization on the histogram, and output the HOG feature vector generated for each image block.

[0099] Use the SVM classifier to train the HOG features of obstacles (vehicles, pedestrians) and non-obstacles (ground, vegetation), calculate the cosine similarity between the current HOG feature and the obstacle template, and if the similarity is greater than the threshold, determine it as an obstacle; otherwise, mark it as noise or an unknown object (outlier), and adjust the similarity threshold according to the average brightness of the image block, and use higher-resolution image blocks for small targets at a distance.

[0100] Such as Figure 2 and Figure 4As shown, the improved Time Ground Recovery (TGR) module is named: Local Ground Joint Verification Module.

[0101] Based on retaining the TGR time reduction mechanism, the method of the present invention adds the following multi-modal modeling and classification capabilities.

[0102] First, perform image superpixel segmentation and feature extraction: Taking the point cloud projection position as the center, crop the local image patch to cover the area around the point cloud. Using the SLIC (Simple Linear Iterative Clustering) algorithm, divide the image patch into 50 - 100 superpixels to balance color and spatial similarity, and extract color features, texture features, shape features, position features, etc.

[0103] Next, extract point cloud geometric features: The extracted point cloud geometric features include: height residual, local roughness, reflection intensity, and density. Based on the point cloud geometric features, obtain the point cloud spatial distribution: the average distance between the point cloud and the nearest adjacent point cloud and the local surface curvature.

[0104] Construct a joint feature vector, map the point cloud points to the corresponding superpixels, extract all image features of the superpixels, perform feature splicing, normalize the numerical features by mean and variance through Z - score normalization, and normalize the color histogram by sum through histogram normalization.

[0105] Use the Gradient Boosting Decision Tree (GBDT) module to replace the binary judgment in the original scheme to determine whether each point is a real ground point, and synchronously output the ground material type. If the confidence of the GBDT classification result is low, still rely on the flatness statistics of TGR for secondary verification and remove non - ground points. TGR solves the misjudgment of temporary rough ground, and GBDT solves the misjudgment related to materials.

[0106] As Figure 2 and 5 shown, the Adaptive Ground Likelihood Estimation (A - GLE) module is named Adaptive Parameter Optimization Module.

[0107] First, initialize the A - GLE statistics, and then construct the CRF four - element energy function through the energy function optimization module: Calculate each energy term according to the features of the current frame, and obtain the CRF four - element energy function by weighted summation. The CRF four - element energy function consists of four parts: unary term, flatness constraint term, smooth term, and reflection intensity constraint term, replacing the single - threshold judgment in the original scheme.

[0108] The unary term fuses the RGB semantic segmentation confidence (CNN probability map), replacing the pure geometric confidence in the original scheme.

[0109] The flatness constraint term is used to enforce the ground geometry flatness in the lane line area, which only acts on the pixel area covered by the lane line mask, imposing a quadratic penalty on the curvature difference of adjacent pixels. The greater the curvature difference, the higher the energy penalty, and the weight parameter is dynamically adjusted according to the scene;

[0110] The smoothing term is used to ensure the spatial continuity of adjacent pixel labels. If adjacent pixels have similar colors, their labels are forced to be the same, weighted by a Gaussian kernel function. The smaller the color difference, the stronger the smoothing constraint;

[0111] The reflection intensity constraint term is used to suppress the misdetection of lane lines in the reflection anomaly area. The variance of the reflection intensity is calculated for each superpixel area. If it exceeds the reflection intensity variance threshold , a large energy penalty is imposed to prohibit this area from being marked as a lane line. When water accumulation / snow is detected, the threshold is automatically relaxed to allow a higher reflection variance.

[0112] Secondly, a new dynamic filtering and anti-noise structure is added, and the parameter dynamic adjustment stage is entered.

[0113] When the curvature variance in the lane line area shows an upward trend within N consecutive frames, is increased proportionally to enhance the flatness constraint. At the same time, a Kalman filter is added to smooth the weight change to avoid parameter mutations.

[0114] When the area covered by the water accumulation / snow mask exceeds the preset ratio, according to the proportion of the water accumulation area , the reflection intensity variance threshold is adjusted according to formula (11) :

[0115] (11)

[0116] where represents the reflection intensity variance threshold for the th adjustment, represents the critical ratio threshold of the water accumulation / snow mask area, represents the control threshold, which is the parameter (relaxation sensitivity coefficient) controlling the relaxation amplitude of the threshold adjustment, represents the proportion of the water accumulation area, represents the area of the water accumulation area, represents the area of the complete area.

[0117] To avoid drastic jitter of parameters due to single-frame noise, a time consistency protection mechanism is added. The mean and standard deviation of historical parameter values are recorded. If the parameter change in the current frame exceeds the historical mean (standard deviation range): ±3σ, where σ represents the standard deviation, then the historical mean is used to replace the current value, and exponential moving average (EMA) filtering is performed on and .

[0118] Finally, output the final lane detection results and terrain status, record the parameter change curves and detection performance indicators for offline analysis.

[0119] Patchwork++ does not have the function of point cloud completion. To solve this problem, the present invention introduces a spatial recovery enhancement module.

[0120] As Figure 2 shown, first, perform sparse region detection. Divide the point cloud into voxel grids of 0.2m×0.2m, calculate the number of point clouds in each voxel, mark the voxels with density lower than the threshold , perform morphological dilation and merging on adjacent low-density voxels to form candidate regions ; segment the road surface area of the image to obtain a mask , project the point cloud into the image coordinate system through calibration parameters, and extract the corresponding image patches ; calculate the SSIM similarity between the and the typical road surface texture. If SSIM > 0.7 and it is located within the , then it is determined as the region to be completed , output: a set of target regions that are sparse and texture-matched { }; otherwise, it is determined as an irreparable area.

[0121] Secondly, through cross-modal GAN point cloud completion, the generator G inputs the sparse point cloud and the corresponding image patches , extracts geometric features through PointNet++ , extracts texture features through ResNet-50 , generates hybrid features through a cross-modal attention module , and generates a dense point cloud based on the FoldingNet network ; the discriminator D inputs the dense point cloud , judges the rationality of the global scene layout, the accuracy of the mid-range object contours, and the authenticity of the local surface details, calculates the loss function, and realizes the optimization of the generated point cloud.

[0122] S3. Process the multi-source information dataset through an improved Patchwork++ model to obtain the coordinates of the ground point cloud .

[0123] The relevant operations of step S3 are introduced with specific examples:

[0124] Input the multi-source information dataset into the improved Patchwork++ model. The improved Patchwork++ model processes the point cloud and RGB image simultaneously, constructs a concentric cylindrical coordinate system in the BEV space, divides the point cloud into an inner ring, a middle ring, and an outer ring, synchronously converts the RGB image into a BEV view through inverse perspective mapping (IPM), divides the image patches according to the same annulus, adaptively grids the RGB image for each annulus, excludes the background and interfering point clouds, extracts the effective point cloud data information of the ground, and obtains the coordinates of the ground point cloud. 。

[0125] S4. According to the coordinates of the ground point cloud ,calculate the road slope value.

[0126] Introduce the relevant operations of step S3 with a specific example:

[0127] As Figure 6 shown, the calculation of the road slope value according to the coordinates of the ground point cloud is achieved through the RANSAC algorithm, specifically:

[0128] S401. According to the coordinates of the ground point cloud ,organize the point cloud data into an N×3 matrix;

[0129] S402. Initialize the RANSAC parameters: set the maximum number of iterations, the inlier threshold, and the minimum inlier ratio;

[0130] S403. Randomly sample and calculate the plane model: randomly select 3 non-collinear points and calculate the plane equation;

[0131] S404. Through inlier detection and model evaluation: calculate the distance from the inlier to the plane, mark the inliers, record the current number of inliers, and update the optimal plane parameters and inlier set if it exceeds the historical optimal value;

[0132] Repeat steps S403~S404 until the maximum number of iterations is reached or the minimum inlier ratio is satisfied, end the iteration, and obtain the finally fitted plane equation (12) as:

[0133] (12)

[0134] where, represents the normal vector , represents the constant term of the plane equation, and the positive and negative determine the offset of the plane relative to the origin;

[0135] S505. The slope is the angle between the normal vector and the vertical direction (z-axis) (with the unit vector of the vertical direction as For reference), the calculation formula is as follows:

[0136] (13)

[0137] The calculation result needs to be converted into an angle. The conversion formula is as follows:

[0138] (14)

[0139] S506. Visualize the fitted plane and the original point cloud to ensure that the fitting result is reasonable and calculate the average error of the inliers. The calculation formula of the average error is (15): (15)

[0140] in, Indicates the An inner point, Indicates the total number of inliers.

[0141] If the average error is much larger than the inlier threshold, it is necessary to adjust the RANSAC parameters and judge the rationality of the slope.

[0142] Example 2

[0143] This embodiment further limits the first embodiment. In this embodiment, the lossless fine fusion module adopts the data-level fusion module in the MCF3D (Multi-Stage Complementary Fusion for Multi-sensor 3D Object Detection) model to realize the fusion of 3D point cloud and RGB image.

[0144] like Figure 7 As shown in the figure, the adaptive image block allocation method is first used for fusion. The template size is set according to the angular resolution of the image and the lidar. Then, registration is performed to obtain the registered points and the unregistered points. The nearest registration point is found for the unregistered point, and it is classified as the corresponding image block. Each point is combined with the matching image block and expanded into a data piece.

[0145] Next, we use RGB-I data-level fusion to obtain reflection intensity information and the extrinsic matrix of the two sensors to achieve coordinate system 1. Based on the intrinsic matrix, we project the 3D point cloud in the camera coordinate system onto the camera's 2D image. After determining the corresponding position of each LiDAR 3D point in the camera image, we add the reflection intensity RGB-I of the corresponding LiDAR 3D point as its fourth channel.

[0146] Finally, the three-dimensional point cloud is cut into layers by height and projected into a two-dimensional grid from a bird's-eye view. The average height, height difference, and average reflection intensity of each grid are statistically analyzed to obtain a global perspective similar to a "bird's-eye view map", which facilitates the subsequent detection of large-scale targets.

Claims

1. A method for estimating the slope of the road ahead based on multi-source information fusion, characterized in that The method includes the following steps: S1. Synchronize the time and perform spatial registration of the camera and lidar to obtain a multi-source information dataset; S2. Construct and improve the Patchwork++ model to obtain an improved Patchwork++ model: S21. Improve the Reflection Noise Removal (RNR) module in the Patchwork++ model: Introduce the BiSeNet V2 module and the lossless fine fusion module into the reflection noise removal module; S22. After the Spatial Division (CZM) module of the Patchwork++ model, introduce the Inverse Perspective Mapping (IPM) module; S23. Introduce a comprehensive weight into the Regional Vertical Plane Fitting (R-VPF) module of the Patchwork++ model; Introduce the Feature Extraction (HOG) module and the Classifier (SVM) module into the Geometric Constraint Filtering module of the Regional Vertical Plane Fitting (R-VPF) module of the Patchwork++ model; S24. In the Time Ground Recovery (TGR) module of the Patchwork++ model, only retain the TGR time reduction mechanism, replace the binary judgment module with the Gradient Boosting Decision Tree (GBDT) module, and introduce the Joint Feature Construction module before the Gradient Boosting Decision Tree (GBDT) module; S25. In the Adaptive Ground Likelihood Estimation (A-GLE) module of the Patchwork++ model, replace the single threshold judgment module with the Energy Function Optimization module, and introduce the Dynamic Filtering and Anti-Noise Structure after the Energy Function Optimization module; S26. After the Adaptive Ground Likelihood Estimation (A-GLE) module of the Patchwork++ model, introduce the Spatial Recovery Enhancement module; S3. Process the multi-source information dataset through the improved Patchwork++ model to obtain the coordinates of the ground point cloud ; S4. Calculate the road slope value based on the coordinates of the ground point cloud , and calculate the road slope value.

2. The method for estimating the slope of the front road based on multi-source information fusion according to claim 1, wherein The specific process of synchronizing the time and performing spatial registration of the camera and lidar is as follows: On the computer side, drive the camera and lidar sensors based on the ROS platform, and unify the sensor sampling moments through an external trigger signal to achieve time synchronization; Independently calibrate and jointly calibrate the lidar and camera, calculate the internal parameter matrices of each sensor and the external parameter matrix between the sensors; Align the spatial positions of the lidar 3D point cloud and the camera RGB image pixels to achieve spatial registration.

3. The method for estimating the slope of the road ahead based on multi-source information fusion according to claim 2, wherein The role of the BiSeNet V2 module is: Segment the RGB image to obtain the road surface and non-road surface probability maps; The role of the lossless fine fusion module is: Project the 3D point cloud onto the RGB image plane and suppress the intensity of high-reflection points in the non-ground area.

4. The method for estimating the slope of the road ahead based on multi-source information fusion according to claim 3, wherein The specific comprehensive weight is as follows: , and both represent undetermined coefficients, , represents the reflection intensity weight, represents the distance weight; the function of the comprehensive weight is: point cloud reflection weight allocation and probability sampling optimization.

5. The method for estimating the slope of the road ahead based on multi-source information fusion according to claim 4, characterized in that, The Joint Feature Construction module sequentially passes through image superpixel segmentation, point cloud geometric feature extraction, and joint feature construction from input to output.

6. The method for estimating the slope of the road ahead based on multi-source information fusion according to claim 5, characterized in that, The Energy Function Optimization module constructs a CRF four-element energy function; the CRF four-element energy function includes a unary term, a flatness constraint term, a smooth term, and a reflection intensity constraint term; The dynamic filtering and noise reduction structure is based on the formula: dynamically adjusts the variance threshold of the reflection intensity , where represents the variance threshold of the reflection intensity for the th adjustment, represents the critical proportion threshold of the waterlogging / snow cover area, represents the control threshold, represents the proportion of the waterlogging area, represents the area of the waterlogging area, represents the area of the complete area.

7. A method for estimating the slope of the front road based on multi-source information fusion according to claim 6, characterized in that, The Spatial Recovery Enhancement module is specifically: First, detect the low-density area, and then use GAN to achieve cross-modal point cloud completion.

8. A method for estimating the slope of the road ahead based on multi-source information fusion according to claim 7, characterized in that According to the coordinates of the ground point cloud , calculating the road slope value is achieved through the RANSAC algorithm, specifically as follows: S801. Initialize the RANSAC parameters; S802. Random sampling and fitting of the plane equation: , where represents the normal vector , represents the constant term of the plane equation; S803. Calculate the road slope value according to the formula: ​

Images