A laser radar road surface type identification method based on multi-modal information fusion

Abstract

The application discloses a laser radar road surface type identification method based on multi-modal information fusion, and relates to the technical field of intelligent driving environment perception, comprising the following steps: fixing a multi-line laser radar in front of a vehicle to collect height signals and echo intensity signals of a road surface in a preview area; after point cloud preprocessing, height and intensity original matrices are obtained through gridding; a multi-quadratic surface radial basis function is used for interpolation of a null grid surface, and a characteristic image is generated through orthographic projection and resampling; an improved MobileNetV4 is used to extract depth features and perform PCA dimension reduction, and 9-dimensional statistical features such as mean, variance, skewness and kurtosis are extracted from two original matrices; four groups of features are normalized and concatenated into a fusion vector, and the fusion vector is stacked into a time sequence matrix by a sliding window; the RoadFormer model is input, multi-layer Transformer coding and time sequence semantic modeling are performed, and a road surface type is output, so that all-weather high-precision and high-robust real-time road surface identification is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving environment perception technology, specifically to a lidar road type recognition method based on multimodal information fusion. Background Technology

[0002] Currently, road surface recognition technologies both domestically and internationally are mainly divided into two categories. One category is sensor-based active perception methods, which directly detect the physical characteristics of the road surface using sensors such as vision and LiDAR. Among them, vision-based solutions acquire road surface images and automatically mine high-level semantic features by combining image statistical features or convolutional neural networks to achieve high-precision type recognition. However, this approach is highly sensitive to lighting conditions, and its perception performance is severely degraded in environments such as darkness, strong light, rain, and fog. LiDAR-based solutions can overcome the susceptibility of vision to environmental influences, but existing methods mostly rely solely on road surface echo intensity data for single-modal recognition. Furthermore, due to the ultra-sparse nature of point clouds, it is difficult to capture fine geometric features at close range, resulting in an adaptation bottleneck between low-level signals and high-level semantics. Another category is identification methods based on vehicle dynamics response, including acoustic and vibration schemes. Acoustic schemes identify road roughness by collecting the spectral characteristics of tire noise signals, but are susceptible to wind noise, ambient horns, and changes in vehicle speed, resulting in limited generalization ability. Vibration schemes use accelerometers to measure the vibration response of unsprung mass, and then infer road smoothness from this. These methods suffer from significant perception lag, failing to meet the requirements of pre-aiming control. Furthermore, existing identification models are mostly based on instantaneous sampling, ignoring the temporal evolution of perceived information. When encountering local potholes, momentary obstructions, or vehicle vibrations, the identification results frequently change, and the system's robustness needs improvement. Summary of the Invention

[0003] To address the shortcomings of existing technologies in terms of environmental adaptability, feature sparsity, and robustness, this invention discloses a LiDAR road surface type recognition method based on multimodal information fusion. This method first utilizes LiDAR to actively detect and acquire road surface height and echo intensity signals. Then, a continuous reconstruction technique based on surface fitting reshapes the discrete point cloud samples into a high-density, high-fidelity two-dimensional feature image. A lightweight semantic mapping network and statistical analysis methods are then combined to extract fused feature vectors representing the road surface's geometric shape and physical material. Subsequently, a temporal feature sequence is constructed using a dynamic sliding window mechanism, and the RoadFormer deep learning architecture with global correlation capabilities is introduced to transform static perceived features into a time-series classification problem. This invention fully leverages the advantages of LiDAR in environmental interference resistance and multi-dimensional perception. Through physical-level reconstruction and temporal-dimensional logical verification, it significantly improves the accuracy and robustness of road surface recognition under all-weather and dynamic conditions.

[0004] This invention proposes a lidar road surface type identification method based on multimodal information fusion, comprising: Acquire road surface point cloud data collected by vehicle-mounted lidar, wherein the road surface point cloud data includes height information and echo intensity information; The road surface point cloud data is rasterized to generate a road surface height matrix and a road surface echo intensity matrix; Radial basis functions are used to perform surface interpolation on the null grids in the road height matrix and the road echo intensity matrix, respectively. Orthophoto projection and resampling are then performed on the interpolated surfaces to generate road height feature images and road echo intensity feature images. The road height feature vector of the road height feature image and the road echo intensity feature vector of the road echo intensity feature image are extracted using a pre-trained deep neural network. Statistical features are extracted from the road height matrix and the road echo intensity matrix respectively to obtain the road height statistical feature vector and the road echo intensity statistical feature vector. The road height feature vector, road echo intensity feature vector, road height statistical feature vector, and road echo intensity statistical feature vector are fused to obtain a fused feature vector; A time-series feature matrix is ​​constructed by fusing feature vectors corresponding to multiple consecutive frames of point cloud data through a sliding window; the time-series feature matrix is ​​then input into a time-series classification model based on the Transformer architecture to obtain road surface type recognition results.

[0005] Furthermore, the acquisition of road point cloud data collected by the vehicle-mounted lidar includes: The multi-line lidar is horizontally and centrally fixed to the front bumper of the vehicle, so that the radar's emission axis forms a negative angle with the horizontal plane, in order to cover a pre-aiming area of ​​1.0 meter to 4.8 meters in front of the vehicle. Based on the standard road width, a lateral cutoff threshold is set to retain point clouds within the range of -1.9 meters to +1.9 meters in the X-axis direction and 1.0 meters to 4.8 meters in the Y-axis direction, while filtering out interference from non-own lanes.

[0006] Further, the road surface point cloud data is subjected to rasterization processing, including: Map the point cloud within the region to the first quadrant, calculate the matrix index coordinates of each sampling point according to the preset spatial resolution, and classify the point cloud into multiple grid cells. For each grid cell, the average height and average echo intensity of all the sampling points it contains are calculated as the value of that grid cell, and blank grid cells that do not contain sampling points are marked as null values, thus forming the original matrix of road height and the original matrix of road echo intensity with a fixed row and column index relationship.

[0007] Furthermore, radial basis functions are used to perform surface interpolation on the null grids in the road surface height matrix and the road surface echo intensity matrix, including: A spatial reconstruction model is constructed using multiple quadratic radial basis functions. Shape parameters are determined through cross-validation. The null grid in the matrix is ​​then filled by interpolation to obtain a continuous physical surface. The expression for the radial basis function of the multi-quadratic surface type is as follows: , r is the Euclidean distance between the point to be interpolated and the known sampling point, and c is the shape parameter.

[0008] Further, using a pre-trained deep neural network, the road height feature vector of the road surface height feature image and the road echo intensity feature vector of the road echo intensity feature image are extracted, including: An improved MobileNetV4 network is used as the deep neural network. The improvements include: modifying the initial convolutional layer to a single-channel input, uniformly setting the expansion ratio of the general inverse bottleneck module to 2.0, adjusting the spatial downsampling step size of the key-value mapping in the multi-head self-attention module to 2, and replacing the classification head with an output layer corresponding to the number of road surface types. The improved network was trained using height image sample sets and intensity image sample sets respectively, resulting in height classification model and intensity classification model; The road surface height feature image is input into the height classification model, the original depth features after global average pooling are extracted, and the road surface height feature vector is obtained after dimensionality reduction by principal component analysis. The road surface echo intensity feature image is input into the intensity classification model, the original depth features after global average pooling are extracted, and the road surface echo intensity feature vector is obtained after dimensionality reduction by principal component analysis.

[0009] Furthermore, the statistical feature vector of road height and the statistical feature vector of road echo intensity both include the following nine statistics: mean, variance, standard deviation, median, maximum value, minimum value, skewness, kurtosis and range.

[0010] Furthermore, the road surface height feature vector, road surface echo intensity feature vector, road surface height statistical feature vector, and road surface echo intensity statistical feature vector are fused, including: The road surface height feature vector and the road surface echo intensity feature vector are respectively L2 normalized; The statistical feature vectors of road height and road echo intensity are both standardized to zero mean. The processed vectors are concatenated in the order of geometric height depth features, reflection texture intensity depth features, height statistical features, and intensity statistical features to form a fused feature vector.

[0011] Furthermore, a temporal feature matrix is ​​constructed by using a sliding window to gather the fused feature vectors corresponding to multiple consecutive frames of point cloud data, including: Set the sliding window length to N frames and the sliding step size to 1 frame. Stack the fused feature vectors of N consecutive time steps into an N×D temporal feature matrix, where D is the dimension of the fused feature vector. The temporal feature matrix is ​​subjected to layer normalization, and temporal breakpoints caused by missing frames or anomalies are filled with linear interpolation or historical mean.

[0012] Furthermore, the time-series classification model based on the Transformer architecture is called RoadFormer, which includes: A temporal encoder consisting of stacked Transformer blocks, each Transformer block including a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization; And a multi-level hierarchical classification head connected to the encoder output, wherein the classification head sequentially includes an input fully connected layer, at least one intermediate compressed fully connected layer and an end discriminative fully connected layer, and the end is connected to a Softmax activation layer to output the probability of various road surface types.

[0013] Furthermore, it also includes: The trained RoadFormer model is transformed into an inference engine through operator fusion and quantization and deployed on the vehicle edge computing platform. New road surface point cloud data is acquired at preset time intervals, the temporal feature matrix is ​​updated, and the road surface type recognition result is output in real time by the inference engine.

[0014] Compared with the prior art, the present invention has the following beneficial technical effects: This invention actively detects and acquires the height signal and echo intensity signal of the road surface using lidar, fundamentally overcoming the defect of visual solutions being highly sensitive to lighting conditions. It can maintain stable perception performance even in harsh environments such as darkness, strong light, rain, and fog, and has the significant advantage of working in all weather conditions.

[0015] This invention utilizes a spatial reconstruction model based on radial basis functions of multiple quadratic surfaces to perform global surface interpolation on the null-valued grid formed by sparse point clouds, and generates a dense feature image through orthophoto projection and resampling. This physical reconstruction process effectively compensates for the non-uniformity of the LiDAR scan line distribution, preserves the fine features of the road surface micromorphology and material reflection, and successfully overcomes the bottleneck of cross-modal adaptation between the low-level sparse signal and the high-level semantic recognition network.

[0016] This invention constructs a multimodal fusion feature vector that incorporates geometric shape information, physical reflection characteristics, and data distribution patterns by integrating deep visual features and multivariate statistical features. This enables multidimensional and complementary characterization of road surface type attributes, significantly improving the completeness and discriminative power of the feature space.

[0017] This invention introduces a dynamic sliding window mechanism to construct temporal feature sequences and employs the RoadFormer deep learning architecture with global correlation capabilities to transform static perception into a temporal classification problem. The model can automatically identify and suppress signal distortion caused by local potholes, momentary occlusion, or vehicle body vibrations, and smoothly output recognition results through logical verification of temporal background. This completely solves the problem of frequent jumps in recognition results in traditional instantaneous sampling methods, and significantly improves the robustness and stability of recognition under dynamic driving conditions.

[0018] Furthermore, by specifically tailoring and improving the MobileNetV4 architecture, this invention significantly reduces model complexity while ensuring feature extraction accuracy, enabling the lightweight inference engine to be deployed on in-vehicle edge computing platforms, meeting the low latency requirements of real-time perception, and possessing extremely high engineering practical value. Attached Figure Description

[0019] Figure 1 This is a flowchart of a lidar road surface type identification method based on multimodal information fusion, as per this invention patent. Figure 2 A schematic diagram of lidar installation; Figure 3 A schematic diagram of image reconstruction of lidar signals; Figure 4 A schematic diagram of time series sensing information segmentation; Figure 5 This is a schematic diagram of the RoadFormer model architecture. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention proposes a road surface type recognition method that fully utilizes multimodal information from LiDAR. The method converts the road surface point cloud information acquired by LiDAR into a high-density, high-fidelity image through surface fitting, and combines the advantages of deep learning and the RoadFormer architecture for road surface recognition, achieving high recognition accuracy and generalization ability. The entire invention patent process is as follows: Figure 1 As shown, the main steps include the following: Step 1: Road surface data preprocessing and radar signal reconstruction Step 1.1: Acquisition of raw lidar data First, a multi-line lidar is horizontally and centrally fixed to the front bumper of the vehicle. The lidar's attitude is adjusted so that its emission axis forms a negative 7-degree angle with the horizontal plane, ensuring that the laser beam vertically covers the core pre-aiming area from 1.0 meter to 4.8 meters in front, thereby acquiring the original road feature point cloud. Continuous scanning is performed on road surfaces of different materials and working conditions, simultaneously acquiring height and echo intensity signals, and classifying and labeling the sequence samples according to the actual physical characteristics of the road surface. Then, based on the standard width of 3.75 meters for Class I highways in my country, a lateral coordinate truncation threshold is set, retaining only the point cloud within the area of ​​-1.9 meters to +1.9 meters in the X-axis direction and 1.0 meter to 4.8 meters in the Y-axis direction, filtering out environmental interference from outside the vehicle's lane, ensuring that feature extraction is focused on the road surface of the vehicle. Finally, preprocessing operations are performed on the received original road surface height signal and echo intensity signal. The original point cloud is resampled using the statistical outlier operator. By measuring the mean and standard deviation of the local point set distribution, outlier noise that deviates from the global distribution pattern is removed. At the same time, the echo energy of heterogeneous road surfaces is unified to a normalized numerical range through linear mapping, thereby achieving intensity normalization.

[0022] Step 1.2: Reconstruction of LiDAR signal image based on surface fitting The preprocessed road height and road echo intensity information are rasterized into original road height and original road echo intensity matrices, respectively. These original matrices are then used to generate two-dimensional dense feature images through 3D surface fitting and orthophoto projection techniques, as follows: Figure 3 As shown. Specifically, it includes the following steps: Step 1.2.1: Rasterization of road surface height signal and road surface echo intensity signal A local Cartesian coordinate system with the lidar projection point as the origin is established, and the discrete point cloud in the 3.8m×3.8m detection area is mapped to the first quadrant. The matrix index coordinates corresponding to each sampling point are calculated based on the spatial resolution of 0.017m, so that the point cloud height values ​​are classified into 224×224 grid cells. The average height and average echo intensity of the sampling points in each grid cell are calculated to establish their characterization values. Null value labeling is performed for blank grid cells where no signal is acquired, thereby transforming the discrete point cloud distribution into a 224×224 original road surface height matrix and original road surface echo intensity matrix with a fixed row and column index relationship.

[0023] Step 1.2.2: Generation of road surface height and road surface echo intensity images based on signal reconstruction To address the technical challenge of point cloud sparseness caused by the increasing beam spacing of lidar in specific areas, leading to the loss of microscopic road features such as pothole edges and gravel texture, this invention constructs a spatial reconstruction model based on multi-quadratic surface radial basis functions to perform global physical surface reshaping on the null-valued grid in the original 224×224 matrix. The radial basis function used in this invention is a multi-quadratic surface, and its analytical expression is: ,in, This represents the Euclidean distance between the point to be interpolated and the known sampling points. The preset shape parameter (ShapeParameter).

[0024] This invention determines the shape parameter c=1.0 based on the actual spatial distribution density of road feature point clouds using cross-validation, with a normalized value of 0.017 meters per unit length for the grid spacing. Experiments verified that under this parameter setting, the root mean square errors of the height matrix reconstruction for four types of road surfaces—asphalt, cement, gravel, and grass—were 0.062 mm, 0.078 mm, 0.124 mm, and 0.153 mm, respectively, demonstrating that multi-quadratic surface interpolation has good accuracy and applicability in the road reconstruction task of this invention. Utilizing the radial symmetry and smooth mapping capability of this basis function, the non-uniformity of the lidar scan line distribution is compensated, thereby transforming regions with missing signals into smooth, continuous physical surfaces.

[0025] Subsequently, orthophoto projection and resampling are performed. The completed 3D geometric surface is orthophoto-mapped along a direction perpendicular to the XY plane, and the height and echo intensity values ​​of each sampling point on the surface are mapped to the pixel amplitude of the image, strictly following the 0.017-meter equidistant resampling rule. This process transforms the geometric height and reflection intensity of the physical dimension into a standardized dense feature image, achieving cross-modal adaptation between the low-level physical signal and the high-level semantic recognition network RoadFormer, ultimately generating a road height feature image and a road echo intensity feature image with dense spatial semantics.

[0026] Step 2: Extraction of multimodal features of road surface Step 2.1: Image Feature Extraction Based on Lightweight Semantic Mapping Network The improved MobileNetV4 architecture is used to perform deep feature mining on the reconstructed road height feature image and echo intensity feature image, respectively, and feature dimensionality reduction is performed using the PCA algorithm. Specifically, the following steps are included: Step 2.1.1: Model building and transfer learning pre-training.

[0027] The specific improvements of this invention to the original MobileNetV4 network structure are as follows: Input layer adaptation: The original MobileNetV4-Hybrid-Medium's initial convolutional layer (Stem) consists of a 3×3 convolutional kernel with a stride of 2 and 3 input channels (corresponding to an RGB three-channel image). This invention modifies the number of input channels of this convolutional kernel to 1 to adapt to the input format of single-channel road height feature images and echo intensity feature images, while keeping the number of output channels unchanged (32 channels). This reduces the number of initial layer parameters by about 66% while maintaining the integrity of subsequent feature extraction pathways.

[0028] Customized Improvement of the Universal Inverse Bottleneck (UIB) Module Expansion Ratio: The original MobileNetV4 uses an expansion ratio of 4.0 to 6.0 for UIB modules in different stages, with the number of intermediate activation channels reaching up to 6 times the number of input channels, introducing a large number of redundant features unrelated to road surface texture. This invention uniformly fixes the expansion ratio of all UIB modules at 2.0, making the number of intermediate activation channels exactly twice the number of input channels. This significantly reduces the number of model parameters while forcing feature channels to focus on expressing two core discriminative information types: road surface smoothness gradient and echo reflectivity distribution.

[0029] Multi-head Self-Attention (MHSA) Module Input Resolution Downsampling Adjustment: The original MobileNetV4-Hybrid integrated the Mobile MHSA module in Stages 4 and 5, calculating the attention weight matrix by default at a feature map resolution of 14×14. Since the effective discrimination scale of road texture features is concentrated in coarse-grained spatial regions, such as sand and gravel particles with a diameter of about 2-8cm corresponding to a grid scale of about 1-5 grids, this invention adjusts the spatial downsampling step size of the Key-Value mapping in the MHSA module from the original 1 to 2, reducing the effective attention calculation resolution to 7×7. While compressing the attention calculation workload to 1 / 4 of the original, it forces the attention head to focus on capturing the macroscopic consistency features of road surface types within a larger receptive field, effectively suppressing the attention weight dispersion problem caused by local minor noise on the road surface, such as sand and gravel scattering points and grass occlusion.

[0030] Classification head reconstruction: The original 1000-dimensional fully connected output layer is replaced with a 4-dimensional output layer (corresponding to four types of road surfaces: asphalt, cement, grass, and gravel), and a Dropout layer with a dropout rate of 0.2 is inserted before the Softmax activation layer to enhance the model's generalization robustness when the image is blurred due to vehicle bumps.

[0031] The combined effect of the above four improvements reduces the number of parameters in the improved MobileNetV4 model from the original 9.7M to approximately 4.2M, and reduces the single-frame inference latency on the road surface recognition test set of this invention to 12ms (RTX4090 GPU), meeting the deployment requirement of less than 50ms latency at the edge in step 5.

[0032] The fine-tuned network was retrained using both height and intensity image sets. The training process employed a single-channel RTX4090 GPU, with a Mini-Batch Size of 64 and a maximum iteration cycle of 20.

[0033] Step 2.1.2: Road surface height image feature extraction and dimensionality reduction First, a single-channel road surface height feature image with a resolution of 224×224 is input into a trained height classification model, and a 2048-dimensional original depth feature vector is extracted from the global average pooling layer. Then, the PCA algorithm is used to reduce the dimensionality of the original height feature vector, retaining the principal component features with a cumulative contribution rate of more than 98%, and finally obtaining a 72-dimensional road surface height feature vector.

[0034] Step 2.1.3: Feature Extraction and Dimensionality Reduction of Road Echo Intensity Image First, a single-channel echo intensity feature image with a resolution of 224×224 is input into the intensity classification model. Similarly, a 2048-dimensional original intensity feature vector is extracted from the back of the global average pooling layer. Then, the PCA algorithm is used to reduce the dimensionality of the original intensity features to obtain a 72-dimensional road echo intensity feature vector.

[0035] Step 2.2: Statistical Feature Extraction Step 2.2.1: Extraction of statistical eigenvectors from the road surface height matrix A 9-dimensional statistical feature vector for road height was extracted from the processed original road height matrix using statistical analysis. This vector includes: mean, variance, standard deviation, median, maximum, minimum, skewness, kurtosis, and range. The specific mathematical definition of this 9-dimensional statistical feature vector is as follows: Let the set of height values ​​of all valid sampling points in the road height matrix be denoted as . There are a total of One effective point, Let the height value be the height of the i-th valid sampling point. Then, the statistical measures are calculated as follows: mean. ;variance Standard deviation The median (Med) is the middle value after the valid point height values ​​are sorted in ascending order; the maximum value... Minimum value Skewness = Kurt's kurtosis reflects the degree of asymmetry in the height distribution. Range reflects the thickness of the tail portion of the height distribution; Range = This reflects the overall magnitude of high-level changes.

[0036] Taking the measured values ​​of statistical features of four typical road surface height matrices collected by this invention as an example (height unit: mm): The mean thickness of the asphalt pavement is approximately 0.83 mm, and the variance is approximately 0.42 mm. 2 The standard deviation is approximately 0.65 mm, the median is approximately 0.91 mm, the maximum value is approximately 2.14 mm, the minimum value is approximately 0.18 mm, the skewness is approximately 0.31, the kurtosis is approximately 2.64, and the range is approximately 1.96 mm.

[0037] The mean thickness of the cement pavement is approximately 1.21 mm, and the variance is approximately 0.58 mm. 2 The standard deviation is approximately 0.76 mm, the median is approximately 1.13 mm, the maximum value is approximately 3.24 mm, the minimum value is approximately 0.27 mm, the skewness is approximately 0.52, the kurtosis is approximately 3.17, and the range is approximately 2.97 mm.

[0038] The mean thickness of the gravel road surface is approximately 2.63 mm, and the variance is approximately 2.31 mm. 2 The standard deviation is approximately 1.52 mm, the median is approximately 2.35 mm, the maximum value is approximately 6.82 mm, the minimum value is approximately 0.48 mm, the skewness is approximately 0.78, the kurtosis is approximately 3.85, and the range is approximately 6.34 mm.

[0039] The mean thickness of the grass surface is approximately 3.41 mm, and the variance is approximately 3.87 mm. 2 The standard deviation is approximately 1.97 mm, the median is approximately 3.12 mm, the maximum value is approximately 9.18 mm, the minimum value is approximately 0.43 mm, the skewness is approximately 1.12, the kurtosis is approximately 4.32, and the range is approximately 8.75 mm.

[0040] The above data shows that different road surface types are significantly separable in terms of mean and variance. For example, the mean of asphalt (0.83 mm) differs from that of grassland (3.41 mm) by about four times. Skewness and kurtosis further characterize the differences in the height distribution of various road surfaces. For example, the skewness of asphalt pavement (0.31) is close to a symmetrical distribution, while the skewness of grassland pavement (1.12) is obviously right-skewed, reflecting the morphological characteristics of randomly protruding grass blades. This provides statistical inputs with clear physical meaning and high discriminative power for subsequent multimodal feature fusion.

[0041] Step 2.2.2: Extraction of statistical eigenvectors from the road surface echo intensity matrix The original pavement echo intensity matrix after processing was subjected to statistical analysis to extract a 9-dimensional pavement echo intensity statistical feature vector, which includes: mean, variance, standard deviation, median, maximum value, minimum value, skewness, kurtosis, and range.

[0042] The calculation methods for each statistic are consistent with the definitions in step 2.2.1, where the height value sequence... Replace with the normalized echo intensity value sequence Its value ranges from [0,1]. Taking the measured statistical characteristics of four typical road surface echo intensity matrices collected by this invention as an example: The mean of the asphalt pavement is approximately 0.31, the variance is approximately 0.018, the standard deviation is approximately 0.134, the median is approximately 0.29, the maximum value is approximately 0.67, the minimum value is approximately 0.12, the skewness is approximately 0.41, the kurtosis is approximately 2.81, and the range is approximately 0.55.

[0043] The mean of the cement pavement is approximately 0.54, the variance is approximately 0.023, the standard deviation is approximately 0.152, the median is approximately 0.52, the maximum value is approximately 0.89, the minimum value is approximately 0.21, the skewness is approximately 0.28, the kurtosis is approximately 2.52, and the range is approximately 0.68.

[0044] The mean of the gravel road surface is approximately 0.43, the variance is approximately 0.047, the standard deviation is approximately 0.217, the median is approximately 0.41, the maximum value is approximately 0.92, the minimum value is approximately 0.08, the skewness is approximately 0.59, the kurtosis is approximately 3.14, and the range is approximately 0.84.

[0045] The mean of the grass surface is approximately 0.22, the variance is approximately 0.031, the standard deviation is approximately 0.176, the median is approximately 0.19, the maximum value is approximately 0.71, the minimum value is approximately 0.04, the skewness is approximately 0.84, the kurtosis is approximately 3.62, and the range is approximately 0.67.

[0046] The above data shows that the statistical characteristics of echo intensity can effectively distinguish the laser reflection characteristics of different road materials: cement road surface is smooth and light in color, so the mean (0.54) is significantly higher than that of asphalt road surface (0.31); gravel road surface has the highest variance (0.047) due to the uneven particle size, resulting in a discrete echo intensity distribution; grassland road surface has the lowest mean intensity (0.22) but the largest skewness (0.84) due to the diffuse reflection characteristics of vegetation, showing a significant difference in statistical distribution pattern from the other three types of road surfaces.

[0047] By combining the statistical feature vectors of road height and road echo intensity, the two sets of 9-dimensional statistical feature vectors are spliced ​​together to form 18-dimensional physical statistical features, which provide complementary geometric shape and material reflection information for multimodal fusion.

[0048] Step 3: Multimodal feature fusion and vectorized reconstruction This step aligns and aggregates the depth visual features and physical statistical features obtained from PCA dimensionality reduction in step 2 at the physical level, constructing a 162-dimensional fused feature vector containing full-dimensional road surface information.

[0049] Step 3.1: Subspace Feature Normalization Processing Before feature aggregation, it is necessary to eliminate the differences in numerical dimensions generated by different extraction algorithms: First, L2 normalization is performed on the depth image features, that is, L2 normalization is performed on the 72-dimensional height image features and 72-dimensional intensity image features generated in steps 2.1.2 and 2.1.3 respectively; then, the statistical features are standardized, that is, the 9-dimensional road height statistical feature vector and the 9-dimensional road echo intensity statistical feature vector obtained in steps 2.2.1 and 2.2.2 are zero-mean processed.

[0050] Step 3.2: Concatenation of multidimensional semantic features and construction of concatenated vectors The road surface height feature vector , road surface echo intensity eigenvector Statistical feature vector of road surface height and statistical eigenvectors of road surface echo intensity The fusion is performed to obtain the fused feature vector. : ; Step 3.3: Time Series Reconstruction under Dynamic Sliding Window After completing multimodal feature fusion, the system is initialized by setting a sliding window with a sampling frequency of 10Hz and a length of 9 frames to ensure that the window can completely cover the physical cycle of the vehicle passing over the local interference source within 0.9s. Figure 4 As shown, this step employs a sliding segmentation mechanism with a high overlap rate, by setting a sliding step size of 1 frame to make the first frame... The time series feature vector set and the first Maintaining an 8-frame data overlap between sequences, i.e., producing... Figure 4 The 0.8s overlapping time series is labeled to ensure a smooth transition of recognition results during dynamic driving.

[0051] Finally, the 162-dimensional fused feature vectors generated in step 3.2 are matrix-reassembled in chronological order. (Time step defined) The fusion feature vector is By using a sliding window mechanism, the vectors of nine consecutive time points are... Stacked into a 9×162 temporal feature matrix : ; This transforms discrete instantaneous features into a continuous perception matrix with spatiotemporal consistency, which serves as the input unit for the RoadFormer architecture.

[0052] Step 3.4: Spatiotemporal Consistency Calibration and Robustness Compensation Spatiotemporal consistency calibration: for time series matrices Layer normalization is performed to eliminate instantaneous changes in signal strength at different times caused by vehicle bumps or speed fluctuations.

[0053] Inter-frequency missing compensation mechanism: For timing breaks caused by high-speed vehicle travel or radar frame loss, linear interpolation or historical mean filling method is used to smooth the abnormal rows in the matrix to ensure the logical continuity of the 9-frame sequence.

[0054] Step 4: Build a new RoadFormer for real-time road surface type monitoring This step leverages the global temporal correlation capabilities of the RoadFormer architecture to perform in-depth analysis of the 9×162-dimensional temporal feature matrix constructed in step 3.3. By identifying consistent features in the sequence and suppressing anomalous instantaneous fluctuations, highly robust real-time monitoring of road surface types is achieved.

[0055] Step 4.1, RoadFormer Model Architecture The RoadFormer model independently constructed in this invention is a deep learning architecture specifically designed for vehicle-mounted LiDAR road surface recognition tasks. It consists of a deep temporal encoder composed of three vertically stacked Transformer blocks and a multi-level hierarchical classification head. Its core improvement lies in solving the uncertainty problem in road attribute perception through a specific network topology. The model can automatically identify and suppress weights caused by local occlusion or transient signal changes, ensuring the smoothness and global robustness of the recognition results during dynamic driving. Each Transformer block integrates an 8-head self-attention mechanism, a feedforward neural network containing two linear mapping layers and a GELU activation function, as well as residual connections and layer normalization modules. A multi-level hierarchical classification head is sequentially connected to the encoder output. Internally, it includes a primary mapping fully connected layer with an input dimension of 256 and an output dimension of 128, a mid-level compression fully connected layer with an output dimension of 64, and a terminal discriminative fully connected layer that aligns 72-dimensional features and compresses them to 4 dimensions. ReLU activation functions and Dropout layers with a dropout rate of 0.3 are embedded between each fully connected layer. The terminal connections are softmax activation layers used to output the probability distributions of the four road types. The specific model structure is as follows: Figure 5 As shown.

[0056] Step 4.2, Road surface type identification based on RoadFormer First, temporal semantic modeling is performed. The temporal feature matrix is ​​input into the RoadFormer encoder, and the contribution weights between features at different times are automatically calculated using a multi-head self-attention mechanism to extract global road features with temporal contextual semantics. Then, feature mapping and classification are performed. The feature vector output by the encoder is fed into the classification head for nonlinear mapping. Through three layers of neurons, the road is filtered and compressed layer by layer. Finally, the output layer determines the road type label within the current time window, such as asphalt or concrete.

[0057] Step 4.3, Model Training The experimental data was divided into training and test sets in an 8:2 ratio. During training, reinforcement learning was applied to samples containing transient interference to improve the model's tolerance to sudden signal changes. The optimal model was derived if the accuracy on the test set consistently exceeded 95% and the class switching latency was below a preset threshold.

[0058] Step 5: Edge deployment and online real-time monitoring This step aims to transform the trained RoadFormer model into a high-efficiency online inference engine and deploy it on an in-vehicle edge computing platform to achieve real-time closed-loop monitoring of road surface types based on a single multi-line LiDAR source.

[0059] Step 5.1: Model Optimization and Sampling Planning The TensorRT optimizer is used to perform operator fusion and quantization on the RoadFormer model to build a lightweight inference engine, which makes the single-frame processing latency less than 50ms. At the same time, the fixed sampling frequency of the multi-line lidar is set to 10Hz, i.e., the sampling period is 0.1s. The internal timestamp of the lidar is used to ensure the modal alignment of the height image and the echo intensity image in each period.

[0060] Step 5.2: Dynamic Monitoring and Interference Removal The system performs sliding window sampling at 0.1-second intervals, synchronously updating a 9×162-dimensional temporal feature matrix that covers information from the past 0.9 seconds. The inference engine parses this matrix every 0.1 seconds, using RoadFormer's self-attention mechanism to compare feature consistency within 9 frames. If a frame is detected to have signal distortion due to potholes or occlusions, the weight of that frame is automatically reduced, and the recognition result is corrected based on the temporal background, ultimately outputting a smooth road surface type monitoring category.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0062] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A lidar road surface type identification method based on multimodal information fusion, characterized in that, include: Acquire road surface point cloud data collected by vehicle-mounted lidar, wherein the road surface point cloud data includes height information and echo intensity information; The road surface point cloud data is rasterized to generate a road surface height matrix and a road surface echo intensity matrix; Radial basis functions are used to perform surface interpolation on the null grids in the road height matrix and the road echo intensity matrix, respectively. Orthophoto projection and resampling are then performed on the interpolated surfaces to generate road height feature images and road echo intensity feature images. The road height feature vector of the road height feature image and the road echo intensity feature vector of the road echo intensity feature image are extracted using a pre-trained deep neural network. Statistical features are extracted from the road height matrix and the road echo intensity matrix respectively to obtain the road height statistical feature vector and the road echo intensity statistical feature vector. The road height feature vector, road echo intensity feature vector, road height statistical feature vector, and road echo intensity statistical feature vector are fused to obtain a fused feature vector; A time-series feature matrix is ​​constructed by fusing feature vectors corresponding to multiple consecutive frames of point cloud data through a sliding window; the time-series feature matrix is ​​then input into a time-series classification model based on the Transformer architecture to obtain road surface type recognition results.

2. The method according to claim 1, characterized in that, The acquisition of road point cloud data collected by the vehicle-mounted lidar includes: The multi-line lidar is horizontally and centrally fixed to the front bumper of the vehicle, so that the radar's emission axis forms a negative angle with the horizontal plane, in order to cover a pre-aiming area of ​​1.0 meter to 4.8 meters in front of the vehicle. Based on the standard road width, a lateral cutoff threshold is set to retain point clouds within the range of -1.9 meters to +1.9 meters in the X-axis direction and 1.0 meters to 4.8 meters in the Y-axis direction, while filtering out interference from non-own lanes.

3. The method according to claim 2, characterized in that, The road surface point cloud data is rasterized, including: Map the point cloud within the region to the first quadrant, calculate the matrix index coordinates of each sampling point according to the preset spatial resolution, and classify the point cloud into multiple grid cells. For each grid cell, the average height and average echo intensity of all the sampling points it contains are calculated as the value of that grid cell, and blank grid cells that do not contain sampling points are marked as null values, thus forming the original matrix of road height and the original matrix of road echo intensity with a fixed row and column index relationship.

4. The method according to claim 3, characterized in that, Surface interpolation is performed on the null grids in the road height matrix and road echo intensity matrix using radial basis functions, including: A spatial reconstruction model is constructed using multiple quadratic radial basis functions. Shape parameters are determined through cross-validation. The null grid in the matrix is ​​then filled by interpolation to obtain a continuous physical surface. The expression for the radial basis function of the multi-quadratic surface type is as follows: , r is the Euclidean distance between the point to be interpolated and the known sampling point, and c is the shape parameter.

5. The method according to claim 1, characterized in that, The road height feature vector of the road surface height feature image and the road echo intensity feature vector of the road echo intensity feature image are extracted using a pre-trained deep neural network, including: An improved MobileNetV4 network is used as the deep neural network. The improvements include: modifying the initial convolutional layer to a single-channel input, uniformly setting the expansion ratio of the general inverse bottleneck module to 2.0, adjusting the spatial downsampling step size of the key-value mapping in the multi-head self-attention module to 2, and replacing the classification head with an output layer corresponding to the number of road surface types. The improved network was trained using height image sample sets and intensity image sample sets respectively, resulting in height classification model and intensity classification model; The road surface height feature image is input into the height classification model, the original depth features after global average pooling are extracted, and the road surface height feature vector is obtained after dimensionality reduction by principal component analysis. The road surface echo intensity feature image is input into the intensity classification model, the original depth features after global average pooling are extracted, and the road surface echo intensity feature vector is obtained after dimensionality reduction by principal component analysis.

6. The method according to claim 1, characterized in that, The statistical feature vectors of road height and road echo intensity both include the following nine statistics: mean, variance, standard deviation, median, maximum, minimum, skewness, kurtosis, and range.

7. The method according to claim 1, characterized in that, The road surface height feature vector, road surface echo intensity feature vector, road surface height statistical feature vector, and road surface echo intensity statistical feature vector are fused, including: The road surface height feature vector and the road surface echo intensity feature vector are respectively L2 normalized; The statistical feature vectors of road height and road echo intensity are both standardized to zero mean. The processed vectors are concatenated in the order of geometric height depth features, reflection texture intensity depth features, height statistical features, and intensity statistical features to form a fused feature vector.

8. The method according to claim 1, characterized in that, A time-series feature matrix is ​​constructed by using a sliding window to fuse the feature vectors corresponding to multiple consecutive frames of point cloud data, including: Set the sliding window length to N frames and the sliding step size to 1 frame. Stack the fused feature vectors of N consecutive time steps into an N×D temporal feature matrix, where D is the dimension of the fused feature vector. The temporal feature matrix is ​​subjected to layer normalization, and temporal breakpoints caused by missing frames or anomalies are filled with linear interpolation or historical mean.

9. The method according to any one of claims 1 to 8, characterized in that, The time-series classification model based on the Transformer architecture is called RoadFormer, which includes: A temporal encoder consisting of stacked Transformer blocks, each Transformer block including a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization; And a multi-level hierarchical classification head connected to the encoder output, wherein the classification head sequentially includes an input fully connected layer, at least one intermediate compressed fully connected layer and an end discriminative fully connected layer, and the end is connected to a Softmax activation layer to output the probability of various road surface types.

10. The method according to claim 9, characterized in that, Also includes: The trained RoadFormer model is transformed into an inference engine through operator fusion and quantization and deployed on the vehicle edge computing platform. New road surface point cloud data is acquired at preset time intervals, the temporal feature matrix is ​​updated, and the road surface type recognition result is output in real time by the inference engine.

Images