An end-to-end off-road terrain feature prediction method based on suspension travel decoupling

Abstract

The present invention relates to an end-to-end off-road ground feature prediction method based on suspension stroke decoupling, which belongs to the field of data prediction technology and solves the problem of inaccurate ground feature prediction caused by the low accuracy of existing extracted features. The method includes collecting and preprocessing environmental information and vehicle response information, using the SLAM method to splice multiple frames of point cloud data in the environmental information and projecting them into a BEV map; extracting multiple statistics of the vehicle response information as vehicle response features, where the suspension stroke in the vehicle response information is the suspension stroke of each suspension point after stroke decoupling; constructing a sample set based on the preprocessed camera mask image, BEV map and vehicle response features in the environmental information, and training a multi-branch fusion classification network; obtaining the camera mask image, BEV map and vehicle response features at the current moment and passing them into the trained multi-branch fusion prediction network to predict the road material type and bumpiness level at the next moment. This achieves accurate prediction of off-road ground features at future moments.

Description

Technical Field

[0001] The present invention relates to the field of data prediction technology, and in particular to an end-to-end off-road ground feature prediction method based on suspension travel decoupling. Background Art

[0002] With the rapid development of unmanned driving technology, for ground unmanned platforms in off-road scenarios, fully and accurately understanding and predicting ground features is a key prerequisite for guiding planning and control modules, and is also a key technology for realizing unmanned driving. This technology includes predicting the material type and degree of bumpiness of the ground where the vehicle body will be located at future moments.

[0003] Currently, there are many methods for identifying ground features. However, existing methods for identifying ground features based on environmental information are difficult to accurately identify ground features in complex off-road ground scenarios with a variety of materials. For example, in a scenario where the vehicle is located on a cement paved road and the road ahead is a dirt road, lidar and a forward-looking camera are needed to capture the road area ahead. However, the ground information at the vehicle's location is inconsistent with the ground information ahead, making it difficult to accurately identify ground features.

[0004] Furthermore, existing ground feature recognition methods that consider vehicle response characteristics fail to filter out the dynamic responses of vehicle maneuverability caused by driving behavior, making it difficult to fully and accurately identify ground features. Furthermore, most methods only recognize ground features at the current vehicle location and lack the ability to predict ground features at future vehicle locations, thus failing to guide the planning and control module to achieve optimal results. Summary of the Invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide an end-to-end off-road terrain feature prediction method based on suspension travel decoupling, so as to solve the problem of inaccurate terrain feature prediction caused by the low feature accuracy of the existing environmental information and vehicle response information extraction.

[0006] An embodiment of the present invention provides an end-to-end off-road terrain feature prediction method based on suspension travel decoupling, comprising the following steps:

[0007] Collect and pre-process the environmental information and vehicle response information of the unmanned vehicle, use the SLAM method to stitch multiple frames of point cloud data in the environmental information and project them into a BEV map;

[0008] extracting multiple statistics of the vehicle body response information as vehicle body response features based on a time window, wherein the suspension stroke in the vehicle body response information is the suspension stroke of each suspension point after stroke decoupling;

[0009] Constructing a sample set based on the preprocessed camera mask image in the environmental information, the BEV map, and the vehicle body response features;

[0010] Constructing a multi-branch fusion prediction network, and training the multi-branch fusion classification network based on the sample set;

[0011] The camera mask image, BEV map and vehicle response features at the current moment are obtained and passed into the trained multi-branch fusion prediction network to predict the road material type and bumpiness level at the next moment where the unmanned vehicle will be located.

[0012] Based on a further improvement of the above method, the pre-processing of the environmental information and vehicle body response information of the unmanned vehicle includes:

[0013] The environmental information and vehicle response information are exported frame by frame in time sequence, and the environmental information and vehicle response information are synchronized with each other based on the acquisition time of the original camera image in the environmental information;

[0014] The original camera image is corrected according to the multi-scale smoothing function and illumination unevenness, and then the corrected camera image is subjected to data enhancement and road segmentation annotation to obtain a camera mask image.

[0015] Based on a further improvement of the above method, the correction of the camera original image according to the multi-scale smoothing function and the illumination unevenness includes:

[0016] A multi-scale Gaussian function is used as a multi-scale smoothing function, and a first illumination component of each pixel in the camera original image is extracted according to the multi-scale smoothing function;

[0017] An image gradient is calculated based on the first illumination component of each pixel as the illumination unevenness of each pixel; whether the illumination unevenness of each pixel is less than a threshold is identified in sequence; if so, a multi-scale hyperbolic tangent function is used as a multi-scale smoothing function to calculate the second illumination component of the pixel; otherwise, a multi-scale Laplace Gaussian function is used as a multi-scale smoothing function to calculate the second illumination component of the pixel;

[0018] The brightness enhancement index of each pixel is calculated according to the second illumination component of each pixel; the corrected brightness value of each pixel is calculated using a two-dimensional gamma function according to the brightness enhancement index of each pixel to obtain a corrected camera image.

[0019] Based on a further improvement of the above method, the vehicle body response characteristics are obtained by extracting multiple statistics of the longitudinal acceleration, vertical acceleration, longitudinal velocity, lateral velocity, vertical velocity, steering curvature and suspension travel of each suspension point after travel decoupling from the vehicle body response information and normalizing them; the multiple statistics include: mean, standard deviation, root square amplitude, root mean square value and maximum absolute value.

[0020] Based on a further improvement of the above method, the suspension stroke of each suspension point after the stroke decoupling is obtained by subtracting the suspension stroke estimated value of each suspension point estimated according to the vehicle vertical motion dynamics model from the suspension stroke measurement value of each suspension point collected by the wire displacement sensor.

[0021] Based on the further improvement of the above method, the multi-branch fusion prediction network includes in sequence: a three-branch feature extraction module, a feature splicing module, a state feature discarding module and a prediction module; wherein, two branches in the three-branch feature extraction module are used to extract and fuse the features of the camera mask image and the BEV map to obtain an environmental feature vector; the third branch is used to extract the feature vector of the vehicle body response feature; the feature splicing module splices the environmental feature vector and the feature vector of the vehicle body response feature and passes it to the state feature discarding module; the feature discarding module applies different discarding rates to the spliced ​​features and passes them to the prediction module; the prediction module outputs the probability distribution of road material type and bumpiness level through a lightweight Transformer network.

[0022] Based on the further improvement of the above method, the two branches are used to extract and fuse the features of the camera mask image and the BEV map to obtain the environment feature vector. The ResNet network is used as a feature extractor, and the multi-head attention mechanism in the Transformer network is used to fuse the intermediate feature maps extracted from each layer of the two branches. After upsampling, the intermediate feature maps are re-input into their respective branches for residual summation.

[0023] Based on the further improvement of the above method, the third branch is used to extract the feature vector of the vehicle body response characteristics. The vehicle body response characteristics are first subjected to Fourier feature mapping to obtain a high-dimensional feature vector, which is then passed into a multi-layer MLP network for dimensionality reduction and feature extraction.

[0024] Based on the further improvement of the above method, the multi-branch fusion prediction network updates the network parameters by calculating the weighted multi-task loss function value; the weighted multi-task loss function value is obtained by weighted summing the cross entropy loss function value for road material classification, the cross entropy loss function value for bumpiness level classification, and the regularization loss function value.

[0025] Based on a further improvement of the above method, the bumpiness level is divided into severe bumpiness, moderate bumpiness, and mild bumpiness based on a comparison of the root mean square value of the unsprung mass vertical acceleration in the vehicle body response information with a bumpiness threshold range; and the road material types include: paved cement road, dirt road, gravel road, and sand and gravel road.

[0026] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0027] 1. Extract features from environmental information and vehicle response information, fuse multimodal data features through a multi-branch fusion prediction network, and directly predict the ground features of the unmanned vehicle's future location. This achieves end-to-end information prediction and provides more effective guidance for planning and vehicle control systems in the field of autonomous driving.

[0028] 2. By introducing illumination unevenness and adaptively selecting multi-scale smoothing functions, it adapts to different types of image features, improves the accuracy of environmental information extraction, and reduces the situation where ground features are unclear or even covered due to illumination factors.

[0029] 3. Decoupling suspension travel eliminates the impact of inertial forces caused by sprung mass acceleration on ground feature recognition during sudden acceleration, braking, or steering. Feature values ​​are extracted from multiple dimensions and statistical perspectives. This not only fully captures the dynamic characteristics of the vehicle's response, but also simplifies and abstracts the original data, making it easier to extract key features and improving prediction accuracy.

[0030] 4. By performing Fourier feature mapping on the vehicle response characteristics, the network pays more attention to changes in high-frequency signals, thereby improving its ability to respond to changes in speed and other factors, and avoiding excessive reliance on high-dimensional environmental inputs.

[0031] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0033] Figure 1 This is a flow chart of an end-to-end off-road terrain feature prediction method based on suspension travel decoupling in an embodiment of the present invention;

[0034] Figure 2 Schematic diagram of a vehicle vertical motion dynamics model in an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of the multi-branch fusion prediction network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0037] A specific embodiment of the present invention discloses an end-to-end off-road terrain feature prediction method based on suspension travel decoupling, such as Figure 1 As shown, the following steps are included:

[0038] S1. Collect and pre-process the environmental information and vehicle body response information of the unmanned vehicle, use the SLAM method to splice multiple frames of point cloud data in the environmental information and project it into a BEV map.

[0039] It should be noted that a controllable off-road test environment is pre-set, and the unmanned vehicle is manually controlled to follow a planned route and driving behavior. The planned route includes different surface types and levels of bumpiness, and the planned driving behaviors include rapid acceleration and deceleration, steady speed driving, and sharp turns. During the driving process, the unmanned vehicle uses a variety of sensors to collect environmental information and vehicle response information.

[0040] Specifically, the various sensors include: cameras, lidars, integrated navigation systems, vehicle underlying electronic control systems, cable displacement sensors and zero-frequency acceleration sensors, among which the environmental information includes camera images captured by the camera and point cloud data captured by the lidar; the vehicle response information includes: the vehicle's pitch angle, roll angle, heading angle, GPS information, vehicle acceleration and speed data collected by the integrated navigation system, the left and right rotation speeds of the underlying motor collected by the vehicle's underlying electronic control system, the suspension travel measurement values ​​of each suspension point in the vehicle suspension system collected by the cable displacement sensor, and the vertical acceleration of the unsprung mass in the vehicle suspension system collected by the zero-frequency acceleration sensor.

[0041] Furthermore, the environmental information and vehicle response information of the unmanned vehicle are pre-processed, including:

[0042] The environmental information and vehicle response information are exported frame by frame in time sequence, and the environmental information and vehicle response information are synchronized with each other based on the acquisition time of the original camera image in the environmental information;

[0043] The original camera image is corrected according to the uneven illumination, and then data enhancement and road segmentation annotation are performed on the corrected camera image to obtain a camera mask image.

[0044] Specifically, the camera raw image is corrected according to the uneven illumination, including:

[0045] ① Extract the first illumination component of each pixel in the camera original image based on the multi-scale smoothing function and the multi-scale Gaussian function.

[0046] It should be noted that, considering that the Gaussian function has a strong smoothing effect on most stable or slowly changing image areas and is suitable for capturing global illumination changes, this embodiment uses a multi-scale Gaussian function as a multi-scale smoothing function through the following formula:

[0047]

[0048] Among them, G p (x,y) represents the smooth function of the pth scale, represents the Gaussian function of the pth scale, c p Indicates the value of the pth scale factor, which controls the degree of smoothing. λ is the normalization constant. (x, y) represents the pixel point on the camera's original image. p = 1, 2, ..., N, where N represents the number of scales.

[0049] Preferably, N=3, and the values ​​of the three scale factors are: 20, 90 and 260 respectively.

[0050] Furthermore, the first illumination component of each pixel in the camera original image is extracted using the following formula:

[0051]

[0052] Where I(x,y) represents the first illumination component extracted and weighted by the multi-scale smoothing function at the pixel point (x,y) in the original camera image, F(x,y) represents the brightness value of the pixel point (x,y) in the original camera image, ω p The weight coefficients representing the illumination components extracted by the smoothing function at the p-th scale are all set to 1 / 3, for example.

[0053] ② Calculate the image gradient based on the first illumination component of each pixel as the illumination unevenness of each pixel; identify in turn whether the illumination unevenness of each pixel is less than a threshold; if so, calculate the second illumination component of the pixel using a multiscale hyperbolic tangent function as a multiscale smoothing function; otherwise, calculate the second illumination component of the pixel using a multiscale Laplace Gaussian function as a multiscale smoothing function.

[0054] Considering that the Gaussian function cannot effectively capture the lighting details of the image in some areas, such as areas with strong edges and large local light variations, in order to improve the extraction accuracy, this embodiment uses lighting unevenness as a local feature to adaptively select a suitable multi-scale smoothing function.

[0055] Specifically, the image gradient of each pixel is calculated by the following formula as the illumination unevenness of each pixel:

[0056]

[0057] Among them, U(x,y) represents the illumination unevenness at the (x,y) pixel in the camera's original image.

[0058] If the illumination uniformity is less than the threshold, it indicates a low-complexity area. The multi-scale hyperbolic tangent function is selected to preserve local details and avoid oversmoothing. This function moderately compresses the low-frequency portion of the image while not causing oversmoothing in areas with stable illumination changes, thus preserving detail information well. The formula is as follows:

[0059]

[0060] If the illumination uniformity is greater than or equal to the threshold, it indicates a highly complex area, and the multi-scale Laplace-Gaussian function is selected to remove large-scale illumination variations. This function combines the Laplace operator and Gaussian filtering to effectively smooth large-scale illumination fluctuations and remove large-scale illumination variations while maintaining edge and structural information. For areas with large illumination non-uniformity, the Laplace-Gaussian function can effectively suppress illumination fluctuations, remove noise, and optimize the global brightness distribution of the image. The formula is as follows:

[0061]

[0062] in, represents the Laplace operator.

[0063] After selecting an adaptive multi-scale smoothing function according to the illumination unevenness, the second illumination component of each pixel is calculated according to formula (2).

[0064] ③ Calculate the brightness enhancement index of each pixel point according to the second illumination component of each pixel point; calculate the corrected brightness value of each pixel point using a two-dimensional gamma function according to the brightness enhancement index of each pixel point to obtain a corrected camera image.

[0065] It should be noted that the second illumination component mean of the entire camera original image is calculated based on the second illumination component of each pixel. The brightness enhancement index of each pixel is calculated using the following formula:

[0066]

[0067] Where γ(x,y) represents the brightness enhancement index of the pixel (x,y) in the original camera image.

[0068] Then, the 2D gamma function is used to calculate the corrected brightness value of each pixel using the following formula:

[0069]

[0070] Where O(x,y) represents the corrected brightness value of the pixel (x,y) in the original camera image.

[0071] According to the above formula, the intensity of the brightness correction of each pixel varies with different lighting conditions.

[0072] This embodiment improves the extraction accuracy by introducing illumination unevenness and adaptively selecting a multi-scale smoothing function to adapt to different types of image features. The brightness of the corrected image decreases in areas with excessive illumination and increases in areas with insufficient illumination, further highlighting the characteristics of the image itself, thereby reducing the situation where ground features are unclear or even covered due to illumination factors.

[0073] Furthermore, the corrected image data is enhanced by random cropping, mirroring, and noise addition to improve the generalization of the model.

[0074] By inputting the text "road" into the automatic annotation software Grounded-SAM for the enhanced image, the text information can be used to generate accurate annotation boxes and mask annotations for the unstructured road areas in the enhanced image. The annotation boxes and mask annotation contents are then used to generate a mask map based on the original image, in which the road area mask is 1 and the background area mask is 0.

[0075] Furthermore, the enhanced camera image is segmented using the mask map to obtain a camera mask image. That is, the mask map is applied to the enhanced camera image through mathematical operations (bitwise AND operations). In the camera mask image, the road area with a mask of 1 is retained, and the background area with a mask of 0 is set to transparent.

[0076] It should be noted that due to the low density of point clouds collected by mechanical lidar, the amount of information that can be reflected by single-frame point cloud data is not detailed enough, and the density is not high enough. It is easy to fail to extract related features during network training, leading to problems such as difficulty in convergence. Therefore, first of all, SLAM is used to perform three-dimensional point cloud splicing on single-frame point cloud data. By splicing multiple frames of point clouds, point cloud densification is achieved, reducing the noise or data missing of single-frame point cloud data, improving the density and integrity of the point cloud, enhancing the ability to express data features, and obtaining richer ground information. This can improve the stability and accuracy of the network model in extracting ground features, reduce the difficulty of convergence, and better ensure the continuity and smoothness of the ground in subsequent scene parameter estimation.

[0077] Furthermore, based on the acquisition time of the camera image, the multi-frame point cloud image obtained after SLAM processing is projected into a BEV (Bird's Eye View) image containing height and intensity information, which corresponds one-to-one to the camera image at each moment.

[0078] It should be noted that the projection area selected in this embodiment is 32 meters in front of the vehicle and 16 meters to either side of the vehicle. Each point cloud in the projection area is assigned to the nearest pixel in the image. Each pixel corresponds to a number of point cloud data points, referred to as a point cloud cluster. The sum of the mean height and mean reflection intensity of the point cloud cluster in each pixel is calculated. This sum is then rescaled to an integer in the range of 0-255 based on the minimum and maximum sums of all data in the projection area. Areas with missing point clouds are filled in by linear interpolation of the pixel blocks surrounding the projection area to produce a BEV map.

[0079] S2. Extracting multiple statistics of the vehicle body response information as vehicle body response features based on a time window, wherein the suspension stroke in the vehicle body response information is the suspension stroke of each suspension point after stroke decoupling.

[0080] It should be noted that suspension travel is essentially the suspension system's response to ground excitation. Uneven ground excitation is much greater than that of smooth ground, resulting in greater suspension travel. This embodiment decouples the time-synchronized suspension travel of the four suspension points on the front and rear axles to eliminate the impact of suspension travel changes on ground feature recognition caused by inertial forces resulting from sprung mass acceleration during sudden acceleration, braking, or steering.

[0081] The suspension stroke of each suspension point after the stroke decoupling is obtained by subtracting the suspension stroke estimated value of each suspension point estimated according to the vehicle vertical motion dynamics model from the suspension stroke measurement value of each suspension point collected by the cable displacement sensor.

[0082] In order to effectively reflect the influence of vehicle roll stiffness on suspension travel, the 1 / 2 vehicle vertical motion dynamics model is used to estimate the suspension travel for the front and rear axles respectively, as shown in the following example: Figure 2 shown.

[0083] Since the 1 / 2 vehicle model calculates the front and rear axle suspension masses separately, i is used to represent the front axle or rear axle. When i=f, it represents the front axle, and when i=r, it represents the rear axle. Figure 2 The force and moment balance equations for the model are listed as follows:

[0084]

[0085] Among them, z il and z irRespectively represent the vertical travel of the left and right suspension points of the single axis (i.e., the estimated value of the suspension travel to be solved), and Respectively represent z il and z ir Solve for the first derivative; z ib Indicates the vertical travel of the sprung mass of the front and rear axles, z ib =(z il +z ir ) / 2, Indicates z ib Solve for second-order derivatives; Indicates the vehicle roll angle, Express Solving for the second-order derivative, Indicates the vehicle's single-axis roll angle, ΔF ibl and ΔF ibr Represents the single-axis left and right suspension forces respectively; I ixx represents the vehicle's moment of inertia in the x direction, Expressed as the roll stiffness of the vehicle's front axle and rear axle roll stiffness B is the wheelbase; c il and c ir Respectively represent the single-axis left and right suspension damping; k il and k ir Respectively represent the single-axis left and right suspension stiffness; q il and q ir Represents the ground elevation excitation input of the left and right wheels of a single axle respectively; m i Indicates the equivalent concentrated mass of the front and rear axles, ΔF il and ΔF ir Represent the dynamic loads of the single-axle left and right wheels respectively.

[0086] It should be noted that since uneven ground and vehicle acceleration will affect the suspension travel respectively, this embodiment takes them into consideration separately. Therefore, when estimating the suspension travel caused by vehicle acceleration, it is assumed that the ground is completely flat, that is, the ground elevation excitation input q of the left and right wheels is il =q ir =0.

[0087] Furthermore, m i The corresponding m f and m r They represent the equivalent concentrated mass of the front and rear axles of the vehicle, respectively, and are calculated using the following formula:

[0088]

[0089] Where M represents the total mass of the vehicle; L represents the wheelbase; a represents the distance from the center of mass of the vehicle to the front axle; and b represents the distance from the center of mass of the vehicle to the rear axle.

[0090] ΔF il Corresponding front axle left wheel vehicle dynamic load ΔF fl and the vehicle dynamic load ΔF on the right wheel of the front axle fr , ΔF ir Corresponding vehicle dynamic load ΔF on the left rear axle wheel rl and the vehicle dynamic load on the right wheel of the rear axle ΔF rr They are calculated by the following formulas:

[0091]

[0092] Where h is the height of the vehicle's center of mass; e is the distance from the center of mass of the sprung mass to the roll center; g is the acceleration due to gravity; a x Indicates the longitudinal acceleration of the vehicle; a y Indicates the lateral acceleration of the vehicle.

[0093] Assuming that the vehicle load state remains unchanged, the above formula is solved by combining the measurement values ​​collected by various sensors on the vehicle to obtain the estimated suspension travel value z of the four suspension points on the front and rear axles. ir and z il , then subtract the corresponding suspension stroke estimation value from the suspension stroke measurement value of each suspension point collected by the cable displacement sensor to obtain the suspension stroke of each suspension point after stroke decoupling, that is, the suspension stroke caused only by ground excitation.

[0094] It should be noted that the camera raw images and point cloud data in the environmental information are low-frequency signals, while the GPS and IMU-related data in the vehicle response information are high-frequency signals. When extracting features from the vehicle response information, the vehicle response information within the corresponding time window is selected according to the acquisition time of the environmental information, thereby better reflecting the continuity of the vehicle response characteristics.

[0095] Specifically, the feature dimensions extracted from the vehicle response information include longitudinal acceleration, vertical acceleration, longitudinal velocity, lateral velocity, vertical velocity, steering curvature, and suspension travel of each suspension point after travel decoupling.

[0096] The acceleration information is directly output by the integrated navigation system, and the longitudinal acceleration and vertical acceleration are mainly selected for analysis. The longitudinal acceleration reflects the acceleration or deceleration of the vehicle, and the vertical acceleration describes the acceleration of the vehicle in the vertical direction. It should be noted that the vertical acceleration must eliminate the gravity acceleration g = 9.8m / s 2 The vertical acceleration can represent the vibration degree of the vehicle during driving and reflect the current ground properties to a certain extent.

[0097] Velocity information is obtained by integrating the acceleration of the integrated navigation system, combining it with the velocity collected by GPS, and performing velocity correction using a Kalman filter. This embodiment analyzes three-axis velocity: longitudinal velocity represents the vehicle's travel speed, lateral velocity represents steering and other vehicle operations, and vertical velocity represents the vehicle's vertical ascent.

[0098] The steering curvature is the real-time steering curvature of the vehicle calculated by combining the left and right speeds of the underlying motor with the integrated navigation system.

[0099] Because the time window contains vehicle response information at multiple moments, multiple statistics from the aforementioned feature dimensions are extracted and normalized to form the vehicle response features. These statistics include mean, standard deviation, root mean square amplitude, root mean square value, and maximum absolute value. Specifically, this embodiment extracts 5 statistics from each of the 10 feature dimensions to obtain 50 values. After calculating the mean and variance, each value is normalized to obtain feature values ​​with a standard normal distribution, forming the vehicle response features.

[0100] S3. Construct a sample set according to the preprocessed camera mask image in the environmental information, the BEV map, and the vehicle body response features.

[0101] It should be noted that the off-road ground characteristics in this embodiment include road material type and bumpiness level, wherein the road material type includes: paved cement road, dirt road, gravel road and sand and gravel road; the bumpiness level includes: severe bumpiness, moderate bumpiness and mild bumpiness.

[0102] In this embodiment, based on the actual application scenario, the bump degree level is divided according to the comparison between the root mean square value of the vertical acceleration of the unsprung mass and the bump threshold range. Specifically, when the root mean square value of the vertical acceleration of the unsprung mass is less than 0.4m / s 2 When the root mean square value of the vertical acceleration of the unsprung mass is within the range of [0.4,1.5], it is considered to be a mild bump; when the root mean square value of the vertical acceleration of the unsprung mass is greater than 1.5m / s 2 It is considered severe turbulence.

[0103] Based on the camera mask image and BEV map at each moment obtained in step S1, the vehicle body response characteristics corresponding to the same moment in step S2 are obtained as a set of data. Each set of data is put into the sample set as a sample, and the road material type and bumpiness level are marked for each sample.

[0104] It should be noted that this embodiment predicts the off-road ground characteristics of the sample at the next moment based on the sample at the current moment. Therefore, the true label of the sample at the current moment corresponds to the road material type and bumpiness level marked by the sample at the next moment.

[0105] S4. Construct a multi-branch fusion prediction network, and train the multi-branch fusion prediction network based on the sample set.

[0106] It should be noted that if Figure 3 As shown, the multi-branch fusion prediction network includes, in sequence: a three-branch feature extraction module, a feature splicing module, a state feature discarding module and a prediction module; wherein, two branches in the three-branch feature extraction module are used to extract and fuse the features of the camera mask image and the BEV map to obtain an environmental feature vector; the third branch is used to extract the feature vector of the vehicle body response feature; the feature splicing module splices the environmental feature vector and the feature vector of the vehicle body response feature and passes it to the state feature discarding module; the feature discarding module applies different discarding rates to the spliced ​​features and passes them to the prediction module; the prediction module outputs the probability distribution of road material type and bumpiness level through a lightweight Transformer network.

[0107] Specifically, the two branches are used to extract and fuse the features of the camera mask image and the BEV map to obtain the environment feature vector. The ResNet network is used as a feature extractor, and the multi-head attention mechanism in the Transformer network is used to fuse the intermediate feature maps of each layer on the two branches. After upsampling, the intermediate feature maps are re-input into their respective branches for residual summation.

[0108] It should be noted that the camera mask image and the global background of the BEV image are complementary. Combining these two types of images and utilizing the multi-head attention mechanism in the Transformer network can obtain more comprehensive environmental features and enhance the prediction network's ability to understand the scene.

[0109] In the three-branch feature extraction module, the branches of camera mask image and BEV map input both use the ResNet network to extract features. The ResNet network includes multiple layers of convolutional pooling layers to perform multi-layer feature extraction of different resolutions on each input image to obtain multiple intermediate feature maps. The feature vector is obtained after average pooling and flattening in the last layer.

[0110] The intermediate feature maps of the same layer in both branches are 3D tensors of dimensions H×W×C. The intermediate feature maps of the two branches are stacked together, and after adding position and velocity embeddings, the element-by-element summation is performed to obtain a sequence of dimensions (2*H*W)×C. This sequence is then passed to the Transformer network between the two branches. Velocity embedding involves projecting the scalar velocity at the corresponding moment into a C-dimensional vector using a linear layer. In the Transformer network between the two branches, a multi-head attention mechanism is used to capture the relationship between features for feature fusion. The upsampled output feature map is then fed back to the two branches, where the residual summation is performed with the intermediate feature maps from the branches.

[0111] For example, the camera mask image and the BEV map are both converted to 256×256 pixel size. Figure 3 The ResNet network used in the two branches includes 4 layers of convolutional pooling layers, and the dimensions of the intermediate feature maps output are: 64×64×64, 32×32×128, 16×16×256, and 8×8×512. After the average pooling of the last layer, the dimension is reduced to 1×1×512, and after flattening, a 512-dimensional feature vector is obtained.

[0112] In the three-branch feature extraction module, the third branch is used to extract the feature vector of the vehicle body response characteristics. The vehicle body response characteristics are first subjected to Fourier feature mapping to obtain a high-dimensional feature vector, which is then passed into the multi-layer MLP network for dimensionality reduction and feature extraction.

[0113] Specifically, the Fourier feature mapping is performed on each eigenvalue in the vehicle body response characteristics using the following formula:

[0114]

[0115] Among them, l(d k ) represents the kth eigenvalue d k The eigenvector obtained after Fourier eigenmapping, k = 1, 2, ..., 50; b j ~N(0,σ 2 ) is a parameter randomly sampled from a Gaussian distribution with an adjustable scale σ, used to control the frequency of mapping, j = 1, 2, ..., n, preferably, n = 10.

[0116] It should be noted that if you want to obtain high-frequency features to reflect subtle changes, set a larger scale σ, such as the mean feature for velocity; if you want to obtain low-frequency features to reflect macroscopic changes, set a smaller scale σ, such as the mean feature for acceleration. This embodiment uses Fourier feature mapping to make the network pay more attention to changes in high-frequency signals, thereby improving its responsiveness to changes in velocity and other factors, and avoiding over-reliance on high-dimensional environmental inputs.

[0117] The vehicle body response characteristics of this embodiment are subjected to Fourier feature mapping to obtain a high-dimensional feature vector of 50×20=1000 dimensions, and then the high-dimensional feature vector is subjected to dimensionality reduction and feature extraction through a multi-layer MLP network.

[0118] Preferably, a three-layer MLP network is used to gradually reduce the 1000-dimensional input features to 256 dimensions to facilitate subsequent feature fusion and scene understanding.

[0119] Specifically, the first layer performs preliminary dimensionality reduction and feature extraction: the first MLP layer receives a 1000-dimensional input feature vector after Fourier transformation and compresses the feature dimensions to 512 dimensions using 512 neurons. The ReLU activation function enhances the nonlinear representation of features, enabling the network to capture preliminary relationships between high-dimensional features. This layer's primary function is to remove redundant information through preliminary dimensionality reduction while retaining the essential information of the features, ensuring efficient subsequent computations and information integrity.

[0120] The second layer is used for deep feature extraction and further dimensionality reduction: The second MLP layer receives a 512-dimensional feature vector and further reduces it to 256 dimensions using 256 neurons. Using the ReLU activation function, the model more effectively fits and expresses nonlinear relationships among multidimensional features. This layer performs deep feature extraction on the initially compressed features, capturing complex feature combinations and ensuring that key feature information is retained even after reducing the dimension to 256.

[0121] The third layer is used for final feature representation and output: The input and output dimensions of the third MLP layer are both 256, consistent with the previous layer. However, the 256-dimensional features output by the second layer are further transformed by neurons through nonlinear transformations, further abstracting the existing feature information. Although the dimensionality is not further reduced, the nonlinear activation enables this layer to discover deep patterns in the features, thereby improving the model's discriminative ability and the quality of feature representation. This layer's role is to generate a final, deeply fused and compressed feature representation, providing high-quality input data for subsequent modules.

[0122] The final output is a low-dimensional feature vector (256 dimensions) after MLP dimensionality reduction and feature extraction. It contains a fusion of acceleration, velocity, steering curvature, and suspension travel. These features are further concatenated into the environment feature vector (global context) and fed into the feature concatenation module to enhance the model's robustness to global and local state information.

[0123] It should be noted that due to hardware limitations and noise in the sensors on unmanned vehicles, the observation and perception of the surrounding environment may be inaccurate, resulting in an inability to fully reflect the actual situation. To enable the model to deeply understand and infer environmental information without clear perception, this embodiment uses the feature concatenation module to concatenate the environmental feature vector and the feature vector of the vehicle response feature before passing it to the state feature discarding module. The state feature discarding module first adds a learnable position embedding to the concatenated feature vector and then applies different discard rates to the feature vectors of the environmental feature vector and the feature vector of the vehicle response feature.

[0124] It is worth noting that a relatively low dropout rate is assigned to the feature vectors of the vehicle response characteristics to avoid excessive loss of fused perception information and thus degrading overall performance. In other words, by masking certain environmental and vehicle response characteristics, overall performance is enhanced, thereby improving generalization and robustness. Preferably, the dropout rate for the environmental feature vector is set to 0.3, while the dropout rate for the feature vector of the vehicle response characteristic is set to 0.15.

[0125] Finally, the feature vector output by the state feature discarding module is passed to the prediction module, which uses a lightweight Transformer network to output the probability distribution of road material type and bumpiness level. The lightweight Transformer network consists of an input layer, a transformer layer, and an output layer. The output layer is used for multi-task prediction (simultaneously predicting road material type and bumpiness level). It first passes through a fully connected layer and then uses a softmax activation function to predict the probability of each category in each task.

[0126] It should be noted that when training a multi-branch fusion prediction network, the network parameters are updated by calculating the weighted multi-task loss function value; the weighted multi-task loss function value L total is the cross entropy loss function value L for road material classification material , the cross entropy loss function value L for the turbulence level classification bump And the regularized loss function value L reg The weighted summation is obtained as follows:

[0127]

[0128] Among them, α, β, and γ represent the weight hyperparameters of the loss function values, which are used to balance the impact of different task losses; JN represents the number of samples, SN represents the number of road material categories, and TN represents the number of bumpiness levels; Represents the one-hot encoding of the sth category in the true road material category distribution of the jth sample; is the probability of the sth road material category predicted by the jth sample; Represents the one-hot encoding of the t-th category in the true bumpiness level distribution of the j-th sample; is the probability of the tth level of bumpiness predicted by the jth sample; Represents the L2 norm of the parameters of the multi-branch fusion prediction network, λ reg represents the regularization coefficient.

[0129] S5. Obtain the camera mask image, BEV map, and vehicle body response features at the current moment, and pass them into the trained multi-branch fusion prediction network to predict the road material type and bumpiness level at the next moment where the unmanned vehicle is located.

[0130] During implementation, based on the environmental information and vehicle response information to be predicted collected by the unmanned vehicle in the off-road scene, the camera mask image, BEV map and vehicle response features at the current moment are obtained according to the methods of steps S1 and S2, and are transmitted to the multi-branch fusion prediction network trained in step S4 to predict the road material type and bumpiness level at the next moment where the unmanned vehicle will be located.

[0131] Furthermore, the driving path and driving strategy are adjusted based on the prediction results to provide more effective guidance for planning and vehicle control systems in the field of autonomous driving. For example, a better driving path can be planned to improve driving efficiency and safety; and the vehicle can be decelerated in advance on severely bumpy roads to improve ride comfort.

[0132] Compared with the existing technology, the end-to-end off-road ground feature prediction method based on suspension travel decoupling provided in this embodiment extracts features from environmental information and vehicle response information, fuses multimodal data features through a multi-branch fusion prediction network, and directly predicts the ground features of the unmanned vehicle's position at the future moment, thereby realizing end-to-end information prediction, and thus providing more effective guidance for planning and vehicle control systems in the field of autonomous driving; by introducing illumination unevenness and adaptively selecting multi-scale smoothing functions, it adapts to different types of image features, improves the extraction accuracy of environmental information, and reduces the unevenness of ground features caused by illumination factors. It decouples the suspension travel to eliminate the influence of the inertial force caused by the acceleration of the sprung mass when the vehicle accelerates, brakes or turns on the recognition of ground features, and extracts characteristic values ​​from the perspective of multiple statistics in multiple dimensions, which not only fully captures the dynamic characteristics of the vehicle response, but also simplifies and abstracts the original data, making it easier to extract key features and improving prediction accuracy. By performing Fourier feature mapping on the vehicle response characteristics, the network pays more attention to changes in high-frequency signals, thereby improving its response to changes in speed and other factors, and avoiding excessive reliance on high-dimensional environmental inputs.

[0133] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0134] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. An end-to-end off-road terrain feature prediction method based on suspension travel decoupling, characterized in that: The following steps are involved: Collect and pre-process the environmental information and vehicle response information of the unmanned vehicle, use the SLAM method to stitch multiple frames of point cloud data in the environmental information and project them into a BEV map; extracting multiple statistics of the vehicle body response information as vehicle body response features based on a time window, wherein the suspension travel in the vehicle body response information is the suspension travel of each suspension point after travel decoupling; the vehicle body response features are obtained by extracting multiple statistics of longitudinal acceleration, vertical acceleration, longitudinal velocity, lateral velocity, vertical velocity, steering curvature, and suspension travel of each suspension point after travel decoupling from the vehicle body response information and performing normalization processing; The plurality of statistics include: mean, standard deviation, root square amplitude, root mean square value and maximum absolute value; Constructing a sample set based on the preprocessed camera mask image in the environmental information, the BEV map, and the vehicle body response features; Construct a multi-branch fusion prediction network, and train the multi-branch fusion classification network based on the sample set; the multi-branch fusion prediction network includes: a three-branch feature extraction module, a feature splicing module, a state feature discarding module and a prediction module in sequence; wherein, two branches in the three-branch feature extraction module are used to extract and fuse the features of the camera mask image and the BEV map to obtain an environmental feature vector; the third branch is used to extract a feature vector of a vehicle body response feature; the feature splicing module splices the environmental feature vector and the feature vector of the vehicle body response feature and passes them to the state feature discarding module; the state feature discarding module applies different discarding rates to the spliced ​​features and then passes them to the prediction module, specifically, first adding a learnable position embedding to the spliced ​​feature vector, and then applying different discarding rates to the feature vector of the environmental feature vector and the feature vector of the vehicle body response feature respectively; the prediction module outputs the probability distribution of road material type and bumpiness level through a lightweight Transformer network; The camera mask image, BEV map and vehicle response features at the current moment are obtained and passed into the trained multi-branch fusion prediction network to predict the road material type and bumpiness level at the next moment where the unmanned vehicle will be located.

2. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The pre-processing of the environmental information and the vehicle body response information of the unmanned vehicle includes: The environmental information and vehicle response information are exported frame by frame in time sequence, and the environmental information and vehicle response information are synchronized with each other based on the acquisition time of the original camera image in the environmental information; The original camera image is corrected according to the multi-scale smoothing function and illumination unevenness, and then the corrected camera image is subjected to data enhancement and road segmentation annotation to obtain a camera mask image.

3. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 2, characterized in that: The correcting of the camera original image according to the multi-scale smoothing function and the illumination unevenness includes: A multi-scale Gaussian function is used as a multi-scale smoothing function, and a first illumination component of each pixel in the camera original image is extracted according to the multi-scale smoothing function; An image gradient is calculated based on the first illumination component of each pixel as the illumination unevenness of each pixel; whether the illumination unevenness of each pixel is less than a threshold is identified in sequence; if so, a multi-scale hyperbolic tangent function is used as a multi-scale smoothing function to calculate the second illumination component of the pixel; otherwise, a multi-scale Laplace Gaussian function is used as a multi-scale smoothing function to calculate the second illumination component of the pixel; The brightness enhancement index of each pixel is calculated according to the second illumination component of each pixel; the corrected brightness value of each pixel is calculated using a two-dimensional gamma function according to the brightness enhancement index of each pixel to obtain a corrected camera image.

4. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The suspension stroke of each suspension point after the stroke decoupling is obtained by subtracting the suspension stroke estimated value of each suspension point estimated according to the vehicle vertical motion dynamics model from the suspension stroke measurement value of each suspension point collected by the cable displacement sensor.

5. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The two branches are used to extract and fuse the features of the camera mask image and the BEV map to obtain the environment feature vector. The ResNet network is used as a feature extractor, and the multi-head attention mechanism in the Transformer network is used to fuse the intermediate feature maps extracted from each layer of the two branches. After upsampling, the intermediate feature maps are re-input into their respective branches for residual summation.

6. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The third branch is used to extract the feature vector of the vehicle body response feature, which is obtained by first performing Fourier feature mapping on the vehicle body response feature to obtain a high-dimensional feature vector, and then passing it into a multi-layer MLP network for dimensionality reduction and feature extraction.

7. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The multi-branch fusion prediction network updates the network parameters by calculating the weighted multi-task loss function value; the weighted multi-task loss function value is obtained by weighted summing the cross entropy loss function value for road material classification, the cross entropy loss function value for bumpiness level classification, and the regularization loss function value.

8. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The bumpiness level is divided into severe bumpiness, moderate bumpiness, and mild bumpiness based on a comparison between a root mean square value of the unsprung mass vertical acceleration in the vehicle body response information and a bumpiness threshold range; The road material types include: paved cement road, dirt road, gravel road and sand and gravel road.

Images