End-to-end cross-country ground feature prediction method based on suspension travel decoupling

Abstract

The invention relates to an end-to-end cross-country ground feature prediction method based on suspension stroke decoupling, belongs to the technical field of data prediction, and solves the problem of inaccurate ground feature prediction caused by low accuracy of extracted features in the prior art. Comprising the following steps: acquiring and preprocessing environment information and vehicle body response information, splicing multi-frame point cloud data in the environment information by using an SLAM method, and projecting the multi-frame point cloud data into a BEV map; extracting a plurality of statistics of the vehicle body response information as vehicle body response characteristics, wherein the suspension stroke in the vehicle body response information is the suspension stroke of each suspension point after stroke decoupling; constructing a sample set according to the preprocessed camera mask image, BEV map and vehicle body response characteristics in the environment information, and training a multi-branch fusion classification network; and acquiring a camera mask image, a BEV map and a vehicle body response characteristic at the current moment, transmitting the camera mask image, the BEV map and the vehicle body response characteristic into the trained multi-branch fusion prediction network, and predicting a road material type and a bumping degree grade at the next moment. And accurate prediction of off-road ground features at future moments is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data prediction, and in particular to an end-to-end off-road ground feature prediction method based on suspension travel decoupling. Background Art

[0002] With the rapid development of unmanned driving technology, for ground unmanned platforms in off-road scenarios, fully and accurately understanding and predicting ground features is a key prerequisite for guiding planning and control modules, and is also a key technology for realizing unmanned driving. This technology includes predicting the material type and degree of bumpiness of the ground where the vehicle will be located at future moments.

[0003] Currently, there are many methods for identifying ground features. However, the existing methods for identifying ground features based on environmental information are not suitable for complex off-road ground scenes with multiple materials. For example, when the vehicle is located on a cement paved road and the road ahead is a dirt road, it is necessary to use lidar and a forward-looking camera to capture the road area ahead. At this time, the ground information at the vehicle's location is inconsistent with the ground information ahead, making it difficult to accurately identify ground features.

[0004] In addition, the existing ground feature recognition methods that consider the vehicle response characteristics fail to filter out the dynamic response of the vehicle's mobility caused by driving behavior, making it difficult to fully and accurately recognize ground features. Moreover, most of them only recognize the ground features at the current location of the vehicle, lacking the prediction of the ground features at the future location of the vehicle, thus failing to guide the planning control module to achieve better results. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide an end-to-end off-road terrain feature prediction method based on suspension travel decoupling, so as to solve the problem of inaccurate terrain feature prediction caused by low feature accuracy of existing environmental information and vehicle response information extraction.

[0006] An embodiment of the present invention provides an end-to-end off-road terrain feature prediction method based on suspension travel decoupling, comprising the following steps:

[0007] Collect and pre-process the environmental information and body response information of the unmanned vehicle, and use the SLAM method to splice multiple frames of point cloud data in the environmental information and project them into a BEV map;

[0008] Extracting multiple statistics of the vehicle body response information as vehicle body response features based on a time window, wherein the suspension stroke in the vehicle body response information is the suspension stroke of each suspension point after stroke decoupling;

[0009] Constructing a sample set according to the preprocessed camera mask image in the environmental information, the BEV map and the vehicle body response feature;

[0010] Constructing a multi-branch fusion prediction network, and training the multi-branch fusion classification network based on the sample set;

[0011] The camera mask image, BEV map and vehicle body response features at the current moment are obtained and passed into the trained multi-branch fusion prediction network to predict the road material type and bumpiness level at the next moment where the unmanned vehicle is located.

[0012] Based on the further improvement of the above method, the preprocessing of the environmental information and the vehicle body response information of the unmanned vehicle includes:

[0013] Export the environmental information and vehicle response information frame by frame in time series, and synchronize the environmental information and vehicle response information based on the acquisition time of the original camera image in the environmental information;

[0014] The original camera image is corrected according to the multi-scale smoothing function and the illumination unevenness, and then the corrected camera image is subjected to data enhancement and road segmentation annotation to obtain a camera mask image.

[0015] Based on the further improvement of the above method, the camera original image is corrected according to the multi-scale smoothing function and the illumination unevenness, including:

[0016] A multi-scale Gaussian function is used as a multi-scale smoothing function, and a first illumination component of each pixel in the original image of the camera is extracted according to the multi-scale smoothing function;

[0017] Calculate the image gradient according to the first illumination component of each pixel as the illumination unevenness of each pixel; identify in turn whether the illumination unevenness of each pixel is less than a threshold value, if so, calculate the second illumination component of the pixel using a multiscale hyperbolic tangent function as a multiscale smoothing function; otherwise, calculate the second illumination component of the pixel using a multiscale Laplace Gaussian function as a multiscale smoothing function;

[0018] The brightness enhancement index of each pixel is calculated according to the second illumination component of each pixel; the corrected brightness value of each pixel is calculated using a two-dimensional gamma function according to the brightness enhancement index of each pixel, and a corrected camera image is obtained.

[0019] Based on the further improvement of the above method, the vehicle body response characteristics are obtained by extracting multiple statistics of the longitudinal acceleration, vertical acceleration, longitudinal velocity, lateral velocity, vertical velocity, steering curvature and suspension travel of each suspension point after stroke decoupling in the vehicle body response information and standardizing them; the multiple statistics include: mean, standard deviation, root square amplitude, root mean square value and maximum absolute value.

[0020] Based on the further improvement of the above method, the suspension stroke of each suspension point after the stroke decoupling is obtained by subtracting the suspension stroke estimated value of each suspension point estimated according to the vehicle vertical motion dynamics model from the suspension stroke measurement value of each suspension point collected by the cable displacement sensor.

[0021] Based on the further improvement of the above method, the multi-branch fusion prediction network includes in sequence: a three-branch feature extraction module, a feature splicing module, a state feature discarding module and a prediction module; wherein, two branches in the three-branch feature extraction module are used to extract and fuse the features of the camera mask image and the BEV image to obtain an environmental feature vector; the third branch is used to extract a feature vector of a vehicle body response feature; the feature splicing module splices the environmental feature vector and the feature vector of the vehicle body response feature and passes them to the state feature discarding module; the feature discarding module applies different discarding rates to the spliced ​​features and passes them to the prediction module; the prediction module outputs the probability distribution of road material type and bumpiness level through a lightweight Transformer network.

[0022] Based on the further improvement of the above method, the two branches are used to extract and fuse the features of the camera mask image and the BEV map to obtain the environmental feature vector, which is obtained by using the ResNet network as a feature extractor, and utilizing the multi-head attention mechanism in the Transformer network to fuse the intermediate feature maps extracted from each layer of the two branches and then re-input them into their respective branches after upsampling to sum the residuals.

[0023] Based on the further improvement of the above method, the third branch is used to extract the feature vector of the vehicle body response characteristics, which is obtained by first performing Fourier feature mapping on the vehicle body response characteristics to obtain a high-dimensional feature vector, and then passing it into a multi-layer MLP network for dimensionality reduction and feature extraction.

[0024] Based on the further improvement of the above method, the multi-branch fusion prediction network updates the network parameters by calculating the weighted multi-task loss function value; the weighted multi-task loss function value is obtained by weighted summing the cross entropy loss function value for road material classification, the cross entropy loss function value for bumpiness level classification, and the regularization loss function value.

[0025] Based on further improvement of the above method, the bumpiness level is divided into severe bumpiness, moderate bumpiness and mild bumpiness according to comparison between the root mean square value of the unsprung mass vertical acceleration in the vehicle body response information and the bumpiness threshold range; the road material types include: paved cement road, dirt road, gravel road and sand and gravel road.

[0026] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0027] 1. Extract features from environmental information and vehicle response information, fuse multimodal data features through a multi-branch fusion prediction network, and directly predict the ground features of the unmanned vehicle's location at future times, achieving end-to-end information prediction, thereby providing more effective guidance for planning and vehicle control systems in the field of autonomous driving.

[0028] 2. By introducing illumination unevenness and adaptively selecting multi-scale smoothing functions, it adapts to different types of image features, improves the accuracy of environmental information extraction, and reduces the situation where ground features are unclear or even covered due to illumination factors.

[0029] 3. Decouple the suspension travel to eliminate the influence of the suspension travel change caused by the inertial force caused by the acceleration of the sprung mass when the vehicle accelerates, brakes or turns suddenly on the recognition of ground features, and extract characteristic values ​​from the perspective of multiple dimensions and multiple statistics, which not only fully captures the dynamic characteristics of the vehicle body response, but also simplifies and abstracts the original data, making it easier to extract key features and improve prediction accuracy.

[0030] 4. By performing Fourier feature mapping on the vehicle response characteristics, the network can pay more attention to the changes in high-frequency signals, thereby improving the ability to respond to changes in speed and other factors, and avoiding excessive reliance on high-dimensional environmental inputs.

[0031] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0033] Figure 1 This is a flow chart of an end-to-end off-road terrain feature prediction method based on suspension travel decoupling in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of a vehicle vertical motion dynamics model in an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of the multi-branch fusion prediction network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0037] A specific embodiment of the present invention discloses an end-to-end off-road terrain feature prediction method based on suspension travel decoupling, such as Figure 1 As shown, the following steps are included:

[0038] S1. Collect and pre-process the environmental information and body response information of the unmanned vehicle, use the SLAM method to splice multiple frames of point cloud data in the environmental information and project it into a BEV map.

[0039] It should be noted that a controllable off-road test environment is pre-set, and the unmanned vehicle is manually controlled to drive according to the planned route and driving behavior. The planned route includes different types of materials and ground bumps, and the planned driving behaviors include rapid acceleration and straight driving, rapid deceleration and straight driving, constant speed and rapid turning. During the driving process, various sensors installed on the unmanned vehicle are used to collect environmental information and vehicle response information.

[0040] Specifically, the various sensors include: cameras, lidars, integrated navigation systems, vehicle underlying electronic control systems, cable displacement sensors and zero-frequency acceleration sensors, among which environmental information includes camera images captured by the camera and point cloud data captured by the lidar; vehicle response information includes: vehicle pitch angle, roll angle, heading angle, GPS information, vehicle acceleration and speed data collected by the integrated navigation system, left and right rotation speeds of the underlying motor collected by the vehicle underlying electronic control system, suspension travel measurement values ​​of each suspension point in the vehicle suspension system collected by the cable displacement sensor, and vertical acceleration of the unsprung mass in the vehicle suspension system collected by the zero-frequency acceleration sensor.

[0041] Furthermore, the environmental information and vehicle response information of the unmanned vehicle are pre-processed, including:

[0042] Export the environmental information and vehicle response information frame by frame in time series, and synchronize the environmental information and vehicle response information based on the acquisition time of the original camera image in the environmental information;

[0043] The original camera image is corrected according to the uneven illumination, and then the corrected camera image is subjected to data enhancement and road segmentation annotation to obtain a camera mask image.

[0044] Specifically, the camera raw image is corrected according to the uneven illumination, including:

[0045] ① Extract the first illumination component of each pixel in the original camera image based on the multi-scale smoothing function and the multi-scale Gaussian function.

[0046] It should be noted that, considering that the Gaussian function has a strong smoothing effect on most stable or slowly changing image areas and is suitable for capturing global illumination changes, this embodiment uses a multi-scale Gaussian function as a multi-scale smoothing function through the following formula:

[0047]

[0048] Among them, G p (x,y) represents the smooth function of the pth scale, represents the Gaussian function of the pth scale, c p It represents the value of the pth scale factor, which controls the degree of smoothing. λ represents the normalization constant. (x, y) represents the pixel point on the original image of the camera. p = 1, 2, ..., N, where N represents the number of scales.

[0049] Preferably, N=3, and the values ​​of the three scale factors are: 20, 90 and 260 respectively.

[0050] Furthermore, the first illumination component of each pixel in the camera original image is extracted by the following formula:

[0051]

[0052] Where I(x,y) represents the first illumination component extracted and weighted by the multi-scale smoothing function at the pixel point (x,y) in the original camera image, F(x,y) represents the brightness value of the pixel point (x,y) in the original camera image, ω p The weight coefficients representing the illumination components extracted by the smoothing function of the p-th scale are all set to 1 / 3, for example.

[0053] ② Calculate the image gradient according to the first illumination component of each pixel as the illumination unevenness of each pixel; identify in turn whether the illumination unevenness of each pixel is less than a threshold value, if so, calculate the second illumination component of the pixel using a multi-scale hyperbolic tangent function as a multi-scale smoothing function; otherwise, calculate the second illumination component of the pixel using a multi-scale Laplace Gaussian function as a multi-scale smoothing function.

[0054] Considering that the Gaussian function cannot effectively capture the illumination details of the image in some areas, such as areas with strong edges and large local light changes, in order to improve the extraction accuracy, this embodiment uses illumination non-uniformity as a local feature to adaptively select a suitable multi-scale smoothing function.

[0055] Specifically, the image gradient of each pixel is calculated by the following formula as the illumination unevenness of each pixel:

[0056]

[0057] Among them, U(x,y) represents the illumination unevenness at the pixel (x,y) in the original camera image.

[0058] If the illumination uniformity is less than the threshold, it indicates a low-complexity area, and the multi-scale hyperbolic tangent function is selected to maintain local details and avoid over-smoothing. This function moderately compresses the low-frequency part of the image, and at the same time does not cause over-smoothing for areas with stable illumination changes, and can better maintain detail information. The formula is as follows:

[0059]

[0060] If the illumination uniformity is greater than or equal to the threshold, it indicates a highly complex area, and the multi-scale Laplace Gaussian function is selected to remove large-scale illumination changes. This function combines the Laplace operator and Gaussian filtering, which can effectively smooth large-scale illumination fluctuations and remove large-scale illumination changes while maintaining edge and structural information. For areas with large illumination unevenness, the Laplace Gaussian function can better suppress illumination fluctuations, remove noise and optimize the global brightness distribution of the image. The formula is as follows:

[0061]

[0062] in, represents the Laplace operator.

[0063] After selecting an adaptive multi-scale smoothing function according to the illumination unevenness, the second illumination component of each pixel is calculated according to formula (2).

[0064] ③ Calculate the brightness enhancement index of each pixel point according to the second illumination component of each pixel point; calculate the corrected brightness value of each pixel point using a two-dimensional gamma function according to the brightness enhancement index of each pixel point to obtain a corrected camera image.

[0065] It should be noted that, according to the second illumination component of each pixel, the second illumination component mean of the entire camera original image is calculated. The brightness enhancement index of each pixel is calculated by the following formula:

[0066]

[0067] Among them, γ(x,y) represents the brightness enhancement index of the pixel (x,y) in the original image of the camera.

[0068] Then, the corrected brightness value of each pixel is calculated using the two-dimensional gamma function through the following formula:

[0069]

[0070] Among them, O(x,y) represents the corrected brightness value of the pixel (x,y) in the original camera image.

[0071] According to the above formula, the strength of the brightness correction of each pixel varies with different lighting conditions.

[0072] This embodiment improves the extraction accuracy by introducing illumination non-uniformity and adaptively selecting a multi-scale smoothing function to adapt to different types of image features. The brightness of the corrected image is reduced in areas with excessive illumination, and increased in areas with insufficient illumination, thereby further highlighting the characteristics of the image itself, thereby reducing the situation where ground features are unclear or even covered due to illumination factors.

[0073] Furthermore, the corrected image data is enhanced by random cropping, mirroring and noise addition to improve the generalization of the model.

[0074] By inputting the text "road" into the automatic annotation software Grounded-SAM for the enhanced image, the text information can be used to generate accurate annotation boxes and mask annotations for the unstructured road areas in the enhanced image. The annotation boxes and mask annotation contents can be used to generate a mask map with the road area mask being 1 and the background area mask being 0 based on the original image.

[0075] Furthermore, the enhanced camera image is segmented using the mask map to obtain a camera mask image, that is, the mask map is applied to the enhanced camera image through mathematical operations (bitwise AND operations), and in the camera mask image, the road area with a mask of 1 is retained, and the background area with a mask of 0 is set to be transparent.

[0076] It should be noted that due to the low density of point clouds collected by mechanical laser radar, the amount of information that can be reflected by single-frame point cloud data is not detailed enough, and the density is not high enough. It is easy to fail to extract related features during network training, resulting in problems such as difficulty in convergence. Therefore, first of all, SLAM is used to perform three-dimensional point cloud splicing on single-frame point cloud data. By splicing multiple frames of point clouds, point cloud densification is achieved, the noise or data missing of single-frame point cloud data is reduced, the density and integrity of point clouds are improved, the expression ability of data features is enhanced, and richer ground information is obtained. It can improve the stability and accuracy of ground feature extraction by the network model, reduce the difficulty of convergence, and better ensure the continuity and smoothness of the ground in subsequent scene parameter estimation.

[0077] Furthermore, based on the acquisition time of the camera image, the multi-frame point cloud image obtained after SLAM processing is projected into a BEV (Bird's Eye View) image containing height and intensity information, which corresponds one-to-one to the camera image at each moment.

[0078] It should be noted that the projection area selected in this embodiment is the area 32 meters in front of the car and 16 meters on both sides of the car. In the projection area, each point cloud is assigned to the pixel closest to it in the image, and each pixel unit corresponds to a number of point cloud data points, which are called point cloud clusters. The sum of the mean height and the mean reflection intensity of the point cloud cluster in each pixel is calculated, and then the sum of the mean height and the mean reflection intensity of the point cloud cluster in each pixel is rescaled to an integer in the range of 0-255 according to the minimum and maximum sum values ​​of all data in the projection area. For areas where point clouds are missing, linear interpolation is performed through the pixel blocks around the projection area to obtain a BEV map.

[0079] S2. Extracting multiple statistics of the vehicle body response information as vehicle body response features based on a time window, wherein the suspension stroke in the vehicle body response information is the suspension stroke of each suspension point after stroke decoupling.

[0080] It should be noted that the suspension travel is essentially the response of the suspension system under the ground excitation. The excitation of uneven ground is much greater than that of good ground, so the suspension travel caused by uneven ground is also larger. This embodiment performs travel decoupling processing on the suspension travel of the four suspension points of the front and rear axles after time synchronization to eliminate the influence of the suspension travel change caused by the inertial force caused by the sprung mass acceleration on the ground feature recognition when the vehicle accelerates, brakes or turns suddenly.

[0081] The suspension stroke of each suspension point after the stroke decoupling is obtained by subtracting the estimated suspension stroke value of each suspension point estimated according to the vehicle vertical motion dynamics model from the suspension stroke measurement value of each suspension point collected by the cable displacement sensor.

[0082] In order to effectively reflect the influence of vehicle roll stiffness on suspension travel, the 1 / 2 vehicle vertical motion dynamics model is used to estimate the suspension travel for the front and rear axles respectively, as follows: Figure 2 shown.

[0083] Since the 1 / 2 vehicle model calculates the front and rear axle suspension masses separately, i is used to represent the front axle or rear axle. When i=f, it represents the front axle, and when i=r, it represents the rear axle. Figure 2 The force and moment balance equations for the model are listed as follows:

[0084]

[0085] Among them, z il and z irThey represent the vertical travel of the left and right suspension points of the single axis (i.e., the estimated value of the suspension travel to be solved), and Respectively represent z il and z ir Solve for the first-order derivative; z ib represents the vertical travel of the sprung mass of the front and rear axles, z ib =(z il +z ir ) / 2, Indicates z ib Solve for second-order derivatives; Indicates the vehicle roll angle, Express Solving for the second-order derivative, Indicates the vehicle's single-axis roll angle, ΔF ibl and ΔF ibr Respectively represent the single-axis left and right suspension forces; I ixx represents the moment of inertia of the vehicle in the x direction, Expressed as the roll stiffness of the vehicle's front axle and the roll stiffness of the rear axle B is the wheelbase; c is the wheelbase il and c ir Respectively represent the single-axis left and right suspension damping; k il and k ir Respectively represent the single-axis left and right suspension stiffness; q il and q ir Respectively represent the ground elevation excitation input of the left and right wheels of a single axle; m i Denotes the equivalent concentrated mass of the front and rear axles, ΔF il and ΔF ir Represent the dynamic loads of the left and right wheels of a single-axle vehicle respectively.

[0086] It should be noted that since uneven ground and vehicle acceleration will affect the suspension travel respectively, this embodiment takes them into consideration respectively. Therefore, when estimating the suspension travel caused by vehicle acceleration, it is assumed that the ground is completely flat, that is, the ground elevation excitation input q of the left and right wheels at this time is il =q ir =0.

[0087] Furthermore, m i The corresponding m f and m r Respectively represent the equivalent concentrated mass of the front axle and rear axle of the vehicle, which are calculated by the following formula:

[0088]

[0089] Among them, M represents the total mass of the vehicle; L represents the wheelbase; a represents the distance from the center of mass of the vehicle to the front axle; b represents the distance from the center of mass of the vehicle to the rear axle.

[0090] ΔF il Corresponding front axle left wheel vehicle dynamic load ΔF fl and the front axle right wheel vehicle dynamic load ΔF fr , ΔF ir Corresponding rear axle left wheel vehicle dynamic load ΔF rl and the rear axle right wheel vehicle dynamic load ΔF rr They are calculated by the following formulas:

[0091]

[0092] Where h is the height of the vehicle's center of mass; e is the distance from the center of mass of the sprung mass to the roll center; g is the acceleration due to gravity; a x Indicates the longitudinal acceleration of the vehicle; a y Indicates the lateral acceleration of the vehicle.

[0093] Assuming that the vehicle load state remains unchanged, the above formula is solved in combination with the measurement values ​​collected by the sensors on the vehicle to obtain the estimated suspension travel value z of the four suspension points on the front and rear axles. ir and z il , and then subtract the corresponding estimated suspension stroke value from the suspension stroke measurement value of each suspension point collected by the cable displacement sensor to obtain the suspension stroke of each suspension point after stroke decoupling, that is, the suspension stroke caused only by ground excitation.

[0094] It should be noted that the camera original images and point cloud data in the environmental information are low-frequency signals, and the GPS and IMU related data in the vehicle response information are high-frequency signals. When extracting features from the vehicle response information, the vehicle response information in the corresponding time window is selected according to the acquisition moments of the environmental information, thereby better reflecting the continuity of the vehicle response characteristics.

[0095] Specifically, the feature dimensions extracted from the vehicle response information include longitudinal acceleration, vertical acceleration, longitudinal velocity, lateral velocity, vertical velocity, steering curvature, and suspension travel of each suspension point after travel decoupling.

[0096] Among them, the acceleration information is directly output by the integrated navigation system, and the longitudinal acceleration and vertical acceleration are mainly selected for analysis. The longitudinal acceleration reflects the acceleration or deceleration of the vehicle, and the vertical acceleration describes the acceleration of the vehicle in the vertical direction. It should be noted that the vertical acceleration must eliminate the gravity acceleration g = 9.8m / s 2 The vertical acceleration can represent the vibration degree of the vehicle during driving and reflect the current ground properties to a certain extent.

[0097] The speed information is obtained by integrating the acceleration of the integrated navigation system to obtain the speed, and then combining it with the speed collected by GPS, and correcting the speed through Kalman filtering. In this embodiment, three-axis speed is selected for analysis, where the longitudinal speed represents the driving speed of the vehicle, the lateral speed represents the steering operation of the vehicle, and the vertical speed represents the vertical height of the vehicle.

[0098] The steering curvature is the real-time steering curvature of the vehicle calculated by combining the left and right rotation speeds of the underlying motor with the integrated navigation system.

[0099] Since the time window contains the vehicle body response information at multiple moments, multiple statistics of the above-mentioned feature dimensions are extracted and standardized as the vehicle body response features, and the multiple statistics include: mean, standard deviation, root amplitude, root mean square value and maximum absolute value. That is to say, in this embodiment, 5 statistics are extracted from 10 feature dimensions to obtain 50 values, and each value is standardized after the mean and variance are calculated to obtain feature values ​​with a standard normal distribution to form the vehicle body response features.

[0100] S3. Constructing a sample set according to the preprocessed camera mask image in the environmental information, the BEV map and the vehicle body response features.

[0101] It should be noted that the off-road ground features in this embodiment include road material types and bumpiness levels, wherein the road material types include: paved cement roads, dirt roads, gravel roads and sandstone roads; the bumpiness levels include: severe bumps, moderate bumps and mild bumps.

[0102] In this embodiment, in combination with actual application scenarios, the bump degree level is divided according to the comparison between the root mean square value of the vertical acceleration of the unsprung mass and the bump threshold range. Specifically, when the root mean square value of the vertical acceleration of the unsprung mass is less than 0.4m / s 2 When the root mean square value of the vertical acceleration of the unsprung mass is within the range of [0.4,1.5], it is considered as mild bumping; when the root mean square value of the vertical acceleration of the unsprung mass is greater than 1.5m / s 2 It is considered severe turbulence.

[0103] According to the camera mask image and BEV map at each moment obtained in step S1, the vehicle body response characteristics corresponding to the same moment in step S2 are respectively obtained as a group of data, each group of data is put into the sample set as a sample, and the road material type and bumpiness level are marked for each sample.

[0104] It should be noted that this embodiment predicts the off-road ground characteristics of the sample at the next moment based on the sample at the current moment. Therefore, the true label of the sample at the current moment corresponds to the road material type and bumpiness level annotated by the sample at the next moment.

[0105] S4. Construct a multi-branch fusion prediction network, and train the multi-branch fusion prediction network based on the sample set.

[0106] It should be noted that if Figure 3 As shown, the multi-branch fusion prediction network includes: a three-branch feature extraction module, a feature splicing module, a state feature discarding module and a prediction module in sequence; wherein, two branches in the three-branch feature extraction module are used to extract and fuse the features of the camera mask image and the BEV image to obtain an environmental feature vector; the third branch is used to extract a feature vector of a vehicle body response feature; the feature splicing module splices the environmental feature vector and the feature vector of the vehicle body response feature and passes them to the state feature discarding module; the feature discarding module applies different discarding rates to the spliced ​​features and passes them to the prediction module; the prediction module outputs the probability distribution of road material type and bumpiness level through a lightweight Transformer network.

[0107] Specifically, the two branches are used to extract and fuse the features of the camera mask image and the BEV map to obtain the environmental feature vector. The ResNet network is used as a feature extractor, and the multi-head attention mechanism in the Transformer network is used to fuse the intermediate feature maps of each layer on the two branches and re-input them into their respective branches after upsampling to sum the residuals.

[0108] It should be noted that the camera mask image and the global background of the BEV image are complementary. These two types of images are combined to utilize the multi-head attention mechanism in the Transformer network to obtain more comprehensive environmental features and enhance the prediction network's ability to understand the scene.

[0109] In the three-branch feature extraction module, the branches of camera mask image and BEV map input both use ResNet network to extract features. The ResNet network includes multiple layers of convolutional pooling layers to perform multi-layer feature extraction of different resolutions on each input image to obtain multiple intermediate feature maps, and the feature vector is obtained after average pooling and flattening in the last layer.

[0110] The intermediate feature maps of the same layer on the two branches are 3D tensors of dimensions H×W×C. The intermediate feature maps of the two branches are stacked together, and the position embedding and speed embedding are added and summed element by element to obtain a (2*H*W)×C dimensional sequence, which is passed to the Transformer network between the two branches. The speed embedding is to project the scalar value of the speed at the corresponding moment into a C-dimensional vector using a linear layer. In the Transformer network between the two branches, a multi-head attention mechanism is used to capture the relationship between features for feature fusion, and the up-sampled output feature map is fed back to the two branches, and the residual summation is performed with the intermediate feature map on the branch.

[0111] For example, the camera mask image and the BEV map are both converted to 256×256 pixel size. Figure 3 The ResNet network used in the two branches includes 4 layers of convolutional pooling layers, and the dimensions of the intermediate feature maps output are: 64×64×64, 32×32×128, 16×16×256, 8×8×512. After the last layer of average pooling, the dimension is reduced to 1×1×512, and a 512-dimensional feature vector is obtained after flattening.

[0112] In the three-branch feature extraction module, the third branch is used to extract the feature vector of the vehicle body response feature. The vehicle body response feature is first subjected to Fourier feature mapping to obtain a high-dimensional feature vector, which is then passed into the multi-layer MLP network for dimensionality reduction and feature extraction.

[0113] Specifically, each eigenvalue in the vehicle body response characteristic is mapped to a Fourier feature using the following formula:

[0114]

[0115] Among them, l(d k ) represents the kth eigenvalue d k The feature vector obtained after Fourier feature mapping, k = 1, 2, ..., 50; b j ~N(0,σ 2 ) is a parameter randomly sampled from a Gaussian distribution with an adjustable scale σ, used to control the frequency of mapping, j = 1, 2, ..., n, preferably, n = 10.

[0116] It should be noted that if you want to obtain high-frequency features to reflect subtle changes, set a larger scale σ, such as the mean feature of speed; if you want to obtain low-frequency features to reflect macro changes, set a smaller scale σ, such as the mean feature of acceleration. This embodiment uses Fourier feature mapping to make the network pay more attention to changes in high-frequency signals, thereby improving the responsiveness to changes in speed and other factors, and avoiding excessive reliance on high-dimensional environmental inputs.

[0117] The vehicle body response characteristics of this embodiment are subjected to Fourier feature mapping to obtain a high-dimensional feature vector of 50×20=1000 dimensions, and then the high-dimensional feature vector is subjected to dimensionality reduction and feature extraction through a multi-layer MLP network.

[0118] Preferably, a three-layer MLP network is used to gradually reduce the 1000-dimensional input features to 256 dimensions to facilitate subsequent feature fusion and scene understanding.

[0119] Specifically, the first layer is used for preliminary dimensionality reduction and feature extraction: the first layer MLP receives the 1000-dimensional input feature vector after Fourier mapping, and compresses the feature dimension to 512 dimensions through 512 neurons. The ReLU activation function is used to enhance the nonlinear expression ability of the features, so that the network can capture the preliminary relationship between high-dimensional features. The main function of this layer is to remove redundant information through preliminary dimensionality reduction while retaining the main information of the features to ensure the efficiency of subsequent calculations and the integrity of information.

[0120] The second layer is used for deep feature extraction and further dimensionality reduction: The second layer MLP receives a 512-dimensional feature vector and further reduces the feature dimension to 256 dimensions through 256 neurons. The ReLU activation function is used to enable the model to more effectively fit and express the nonlinear relationship of multidimensional features. The role of this layer is to perform deep feature extraction on the features after preliminary compression, capture complex feature combination relationships, and ensure that key feature information is retained after reducing to 256 dimensions.

[0121] The third layer is used for final feature expression and output: The input and output dimensions of the third layer MLP are both 256 dimensions, which is consistent with the previous layer. The 256-dimensional features output by the second layer are transformed by neurons again to further abstract the existing feature information. Although the dimension is not further reduced, this layer can discover deep patterns in the features through nonlinear activation, thereby improving the model's discriminative ability and the quality of feature expression. The role of this layer is to generate the final feature expression after deep fusion and compression, providing high-quality input data for subsequent modules.

[0122] The final output is a low-dimensional feature vector (256 dimensions) after MLP dimensionality reduction and feature extraction, which contains a fusion understanding of acceleration, velocity, steering curvature and suspension travel. These features will be further spliced ​​into the environment feature vector (global context) and fed into the feature splicing module to enhance the model's robustness to global and local state information.

[0123] It should be noted that due to hardware limitations and noise in the sensors on the unmanned vehicle, the observation and perception of the surrounding environment may be inaccurate, resulting in an inability to fully reflect the real situation. In order to enable the model to have a deep understanding and reasoning of environmental information without clear perception, this embodiment uses a feature splicing module to splice the environmental feature vector and the feature vector of the vehicle body response feature and then pass it to the state feature discarding module. In the state feature discarding module, a learnable position embedding is first added to the spliced ​​feature vector, and different discard rates are applied to the feature vectors of the environmental feature vector and the feature vector of the vehicle body response feature.

[0124] It is worth noting that a relatively low discard rate is assigned to the feature vector of the vehicle body response feature to avoid excessive loss of fused perception information and reduce the overall performance, that is, to enhance the overall performance by masking certain environmental and vehicle body response features, thereby improving generalization and robustness. Preferably, the discard rate of the environmental feature vector is set to 0.3, while the discard rate of the feature vector of the vehicle body response feature is set to 0.15.

[0125] Finally, the feature vector output by the state feature discarding module is passed to the prediction module, which outputs the probability distribution of road material type and bumpiness level through a lightweight Transformer network. The lightweight Transformer network includes an input layer, a transformer layer, and an output layer in sequence. The output layer is used for multi-task prediction (predicting road material type and bumpiness level at the same time), first passing through a fully connected layer, and then predicting the probability of each category in each task through a softmax activation function.

[0126] It should be noted that when training a multi-branch fusion prediction network, the network parameters are updated by calculating the weighted multi-task loss function value; the weighted multi-task loss function value L total is the cross entropy loss function value L for road material classification material , the cross entropy loss function value L for the classification of the degree of bumpiness bump And the regularized loss function value L reg The weighted sum is obtained as follows:

[0127]

[0128] Among them, α, β, and γ represent the weight hyperparameters of the loss function values, which are used to balance the impact of different task losses; JN represents the number of samples, SN represents the number of road material categories, and TN represents the number of bumpiness levels; Represents the one-hot encoding of the sth category in the real road material category distribution of the jth sample; is the probability of the sth road material category predicted by the jth sample; Represents the one-hot encoding of the t-th category in the true bumpiness level distribution of the j-th sample; is the probability of the tth level of bumpiness predicted by the jth sample; Represents the L2 norm of the parameters of the multi-branch fusion prediction network, λ reg represents the regularization coefficient.

[0129] S5. Obtain the camera mask image, BEV map and vehicle body response features at the current moment, and pass them into the trained multi-branch fusion prediction network to predict the road material type and bumpiness level at the location of the unmanned vehicle at the next moment.

[0130] During implementation, based on the environmental information and vehicle response information to be predicted collected by the unmanned vehicle in the off-road scene, the camera mask image, BEV map and vehicle response characteristics at the current moment are obtained according to the methods of step S1 and step S2, and are transmitted to the multi-branch fusion prediction network trained in step S4 to predict the road material type and bumpiness level of the unmanned vehicle's location at the next moment.

[0131] Furthermore, the driving path and driving strategy are adjusted according to the prediction results to provide more effective guidance for planning and vehicle control systems in the field of autonomous driving. For example, a better driving path can be planned to improve driving efficiency and safety; and the vehicle can be decelerated in advance on severely bumpy roads to improve ride comfort.

[0132] Compared with the prior art, the end-to-end off-road ground feature prediction method based on suspension travel decoupling provided in this embodiment extracts features from environmental information and vehicle response information, fuses multimodal data features through a multi-branch fusion prediction network, and directly predicts the ground features of the unmanned vehicle's location at the future moment, thereby realizing end-to-end information prediction, and thus providing more effective guidance for planning and vehicle control systems in the field of autonomous driving; by introducing illumination unevenness and adaptively selecting multi-scale smoothing functions, it adapts to different types of image features, improves the extraction accuracy of environmental information, and reduces the ground feature unevenness caused by illumination factors. It can detect obvious or even covered situations; decouple the suspension travel to eliminate the influence of the suspension travel change caused by the inertial force caused by the acceleration of the sprung mass when the vehicle accelerates, brakes or turns suddenly on the ground feature recognition, and extract characteristic values ​​from the perspective of multiple statistics in multiple dimensions, which not only fully captures the dynamic characteristics of the vehicle body response, but also simplifies and abstracts the original data, making it easier to extract key features and improving the prediction accuracy; by performing Fourier feature mapping on the vehicle body response characteristics, the network pays more attention to the changes in high-frequency signals, thereby improving the response ability to changes in speed and other factors, and avoiding the network from being overly dependent on high-dimensional environmental inputs.

[0133] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0134] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An end-to-end off-road terrain feature prediction method based on suspension travel decoupling, characterized in that: The following steps are involved: Collect and pre-process the environmental information and body response information of the unmanned vehicle, and use the SLAM method to splice multiple frames of point cloud data in the environmental information and project them into a BEV map; Extracting multiple statistics of the vehicle body response information as vehicle body response features based on a time window, wherein the suspension stroke in the vehicle body response information is the suspension stroke of each suspension point after stroke decoupling; Constructing a sample set according to the preprocessed camera mask image in the environmental information, the BEV map and the vehicle body response feature; Constructing a multi-branch fusion prediction network, and training the multi-branch fusion classification network based on the sample set; The camera mask image, BEV map and vehicle body response features at the current moment are obtained and passed into the trained multi-branch fusion prediction network to predict the road material type and bumpiness level at the next moment where the unmanned vehicle is located.

2. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The preprocessing of the environmental information and the vehicle body response information of the unmanned vehicle includes: Export the environmental information and vehicle response information frame by frame in time series, and synchronize the environmental information and vehicle response information based on the acquisition time of the original camera image in the environmental information; The original camera image is corrected according to the multi-scale smoothing function and the illumination unevenness, and then the corrected camera image is subjected to data enhancement and road segmentation annotation to obtain a camera mask image.

3. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 2, characterized in that: The correcting of the camera original image according to the multi-scale smoothing function and the illumination unevenness includes: A multi-scale Gaussian function is used as a multi-scale smoothing function, and a first illumination component of each pixel in the original image of the camera is extracted according to the multi-scale smoothing function; Calculate the image gradient according to the first illumination component of each pixel as the illumination unevenness of each pixel; identify in turn whether the illumination unevenness of each pixel is less than a threshold value, if so, calculate the second illumination component of the pixel using a multiscale hyperbolic tangent function as a multiscale smoothing function; otherwise, calculate the second illumination component of the pixel using a multiscale Laplace Gaussian function as a multiscale smoothing function; The brightness enhancement index of each pixel is calculated according to the second illumination component of each pixel; the corrected brightness value of each pixel is calculated using a two-dimensional gamma function according to the brightness enhancement index of each pixel, and a corrected camera image is obtained.

4. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The vehicle body response characteristics are obtained by extracting multiple statistics of the longitudinal acceleration, vertical acceleration, longitudinal speed, lateral speed, vertical speed, steering curvature and suspension travel of each suspension point after travel decoupling from the vehicle body response information and performing normalization processing; The multiple statistics include: mean, standard deviation, root square amplitude, root mean square value and maximum absolute value.

5. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The suspension stroke of each suspension point after the stroke decoupling is obtained by subtracting the estimated suspension stroke value of each suspension point estimated according to the vehicle vertical motion dynamics model from the suspension stroke measurement value of each suspension point collected by the cable displacement sensor.

6. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The multi-branch fusion prediction network includes in sequence: a three-branch feature extraction module, a feature splicing module, a state feature discarding module and a prediction module; wherein, two branches in the three-branch feature extraction module are used to extract and fuse the features of the camera mask image and the BEV image to obtain an environmental feature vector; the third branch is used to extract a feature vector of a vehicle body response feature; the feature splicing module splices the environmental feature vector and the feature vector of the vehicle body response feature and passes them to the state feature discarding module; the feature discarding module applies different discarding rates to the spliced ​​features and passes them to the prediction module; the prediction module outputs the probability distribution of road material type and bumpiness level through a lightweight Transformer network.

7. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 6, characterized in that: The two branches are used to extract and fuse the features of the camera mask image and the BEV image to obtain the environment feature vector, which is obtained by using the ResNet network as a feature extractor, and utilizing the multi-head attention mechanism in the Transformer network to fuse the intermediate feature maps extracted from each layer of the two branches and then re-input them into their respective branches after upsampling to sum the residuals.

8. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 6, characterized in that: The third branch is used to extract the feature vector of the vehicle body response feature, which is obtained by first performing Fourier feature mapping on the vehicle body response feature to obtain a high-dimensional feature vector, and then passing it into a multi-layer MLP network for dimensionality reduction and feature extraction.

9. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 6, characterized in that: The multi-branch fusion prediction network updates the network parameters by calculating the weighted multi-task loss function value; the weighted multi-task loss function value is obtained by weighted summing the cross entropy loss function value for road material classification, the cross entropy loss function value for bumpiness level classification, and the regularization loss function value.

10. The end-to-end off-road terrain feature prediction method based on suspension travel decoupling according to claim 1, characterized in that: The bumping degree level is divided into severe bumping, moderate bumping and mild bumping according to the comparison between the root mean square value of the vertical acceleration of the unsprung mass in the vehicle body response information and the bumping threshold range; The road material types include: paved cement road, dirt road, gravel road and sand and gravel road.

Images