Intelligent vehicle passable map construction method based on multi-sensor fusion

Abstract

The application relates to the technical field of robots, in particular to a smart car passable map construction method based on multi-sensor fusion.The application provides a smart car passable map construction method based on multi-sensor fusion, which can accurately perceive and identify terrain information and passable areas in a complex unstructured environment, and improves the driving safety of the smart car in a complex scene.The process is as follows: a height cost map and an uncertainty map are obtained;an image processing model trained is obtained;image data is input into the trained image processing model, and a category label corresponding to each pixel point of the image is output, which is converted into a binary navigability value;point cloud is obtained in combination with the binary navigability value; a semantic cost map is obtained based on the point cloud; the uncertainty map and the semantic cost map are used as a constraint mask of the height cost map, and a final passable map is generated.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to environmental perception technology for mobile robots, and particularly to a method for constructing a smart vehicle accessibility map based on multi-sensor fusion. Background Technology

[0002] In unstructured scenarios, such as complex terrain environments like wilderness roads, intelligent vehicles face challenges such as diverse terrain types, irregular environmental elements, and high uncertainty in perceived information. The safe and efficient operation of intelligent vehicles relies on accurate perception and assessment of complex terrain. The slope, flatness, and obstacles of the terrain directly affect the vehicle's stability and safety. Therefore, constructing a terrain model that accurately reflects the traversable characteristics of the environment is a crucial foundation for achieving autonomous decision-making and path planning in intelligent vehicles.

[0003] In existing technologies, intelligent vehicles typically rely on single or multiple sensors, such as LiDAR, cameras, or millimeter-wave radar, to acquire environmental information and perform terrain recognition and accessibility assessment through point cloud processing and image analysis. However, in unstructured environments, single sensors are susceptible to changes in lighting, occlusion interference, and terrain complexity, leading to unstable perception results. Furthermore, while some multi-sensor fusion methods improve terrain recognition accuracy to some extent, they often neglect the constraints of the vehicle's dynamic driving performance, making it difficult to reasonably assess terrain accessibility.

[0004] Furthermore, with the improvement of sensor accuracy and the continuous increase in the scale of perceived data, existing environmental perception and terrain analysis algorithms generally suffer from high computational complexity and insufficient real-time performance, making it difficult to meet the actual needs of intelligent vehicles for rapid response and efficient decision-making in complex environments. Existing technologies often fail to fully consider the dynamic driving performance constraints of the vehicle itself when performing terrain recognition and accessibility assessment, resulting in poor adaptability to different models of intelligent vehicles. Therefore, how to effectively improve the computational efficiency of algorithms while ensuring terrain recognition accuracy remains a key technical problem that needs to be solved in the field of intelligent vehicle environmental perception. Summary of the Invention

[0005] This invention provides a method for constructing a smart vehicle accessibility map based on multi-sensor fusion, which can accurately perceive and identify terrain information and passable areas in complex unstructured environments, thereby improving the driving safety of smart vehicles in complex scenarios.

[0006] The specific process of constructing a smart vehicle accessibility map based on multi-sensor fusion is as follows:

[0007] Step 1: Based on lidar point cloud data, fuse a sparse Gaussian process model to obtain an elevation cost map and an uncertainty map;

[0008] Step 2: Obtain the dataset. Each data point in the dataset includes an image and its corresponding label.

[0009] The images and labels in the data are remapped to obtain the remapped images and labels.

[0010] The remapped image is used as the input to the image processing model, and the label corresponding to the remapped image is used as the output of the input image processing model. The image processing model is trained until the total loss function converges, and the trained image processing model is obtained.

[0011] Step 3: Input the image data acquired by the vehicle-mounted RGB-D camera into the image processing model trained in Step 2. The image processing model outputs the class label values ​​of each pixel in the image, which range from [0,5].

[0012] Based on the remapping rules, the terrain category labels are converted into binary navigability values. ;

[0013] Combined with binary navigability value Obtain the point cloud; obtain the semantic cost map based on the point cloud;

[0014] Step 4: Use the uncertainty map and semantic cost map as constraint masks for the elevation cost map to generate the final drivability map.

[0015] The beneficial effects of this invention are as follows:

[0016] To address the issue of single-sensor susceptibility to environmental interference during traditional unstructured environment map construction, this invention integrates multi-source sensor data, effectively reducing the impact of environmental changes on perception results and improving the stability of environmental perception and driving safety. This invention fully considers the differences in dynamic driving capabilities among different types of intelligent vehicles, introducing weighting coefficients and critical thresholds such as the maximum slope, minimum flatness, and maximum elevation change that intelligent vehicles can withstand when constructing the elevation cost map to evaluate the terrain. When constructing the semantic cost map, terrain categories are remapped based on vehicle driving capabilities, enabling different intelligent vehicles to achieve good adaptability and traffic performance in various unstructured environments.

[0017] To address the issue of high computational complexity caused by the massive scale of LiDAR point cloud data in constructing elevation cost maps, this invention introduces a sparse Gaussian process model. This model uses a small number of representative induced points to approximate the original point cloud data, significantly reducing computational complexity and improving LiDAR data processing efficiency while maintaining modeling accuracy. Simultaneously, in the semantic cost map construction process, this invention first improves semantic recognition and inference speed by reducing feature resolution. Furthermore, it introduces an intra-group self-attention mechanism to alleviate the decrease in model accuracy caused by the reduced resolution. The synergistic effect of these techniques enables this invention to meet real-time requirements while ensuring the accuracy and reliability of the constructed semantic cost map. Attached Figure Description

[0018] Figure 1 A flowchart outlining the overall methodology for constructing intelligent vehicle accessibility maps based on multi-sensor fusion.

[0019] Figure 2 Flowchart for constructing an elevation cost map;

[0020] Figure 3 This is a flowchart of the semantic segmentation process;

[0021] Figure 4 Flowchart for constructing a semantic cost map; Detailed Implementation

[0022] Specific implementation method one: Combining Figure 1 , Figure 2 , Figure 3 , Figure 4 This embodiment describes the specific process of constructing an intelligent vehicle accessibility map based on multi-sensor fusion as follows:

[0023] Step 1: Based on lidar point cloud data, fuse a sparse Gaussian process model to obtain an elevation cost map and an uncertainty map;

[0024] Step 2: Obtain the dataset. Each data point in the dataset includes an image and its corresponding label.

[0025] The images and labels in the data are remapped to obtain remapped images and labels, which are used for subsequent semantic segmentation results.

[0026] The remapped image is used as the input to the image processing model, and the label corresponding to the remapped image is used as the output of the input image processing model. The image processing model is trained until the total loss function converges, and the trained image processing model is obtained.

[0027] Step 3: Input the image data acquired by the vehicle-mounted RGB-D camera into the image processing model trained in Step 2. The image processing model outputs the category label (6 types) corresponding to each pixel of the image, with a value range of [0,5].

[0028] Based on the remapping rules, the terrain category labels are converted into binary navigability values. ;

[0029] Combined with binary navigability value Obtain the point cloud; obtain the semantic cost map based on the point cloud;

[0030] Step 4: Use the uncertainty map and semantic cost map as constraint masks for the elevation cost map to generate the final drivability map.

[0031] Specific Implementation Method Two: Combining Figure 2 This embodiment differs from Specific Embodiment 1 in that: in step 1, an elevation cost map and an uncertainty map are obtained by fusing a sparse Gaussian process model based on lidar point cloud data.

[0032] The specific process is as follows:

[0033] Step 1.1: Using the node-based programming method of the ROS system, create an action service object. The action service object receives point cloud data / points output by the vehicle-mounted LiDAR (the LiDAR is installed on the intelligent vehicle chassis). The point cloud data is located in the LiDAR coordinate system.

[0034] The lidar coordinate system uses a right-handed coordinate system, where The axis points in the direction the intelligent vehicle is moving. The axis points to the left of the vehicle. Axis perpendicular to Flat and pointing upwards;

[0035] Step 1.2: Based on the current pitch and roll angles of the lidar, transform the point cloud data in the lidar coordinate system to the "radar-horizontal" coordinate system to obtain the point cloud data in the "radar-horizontal" coordinate system;

[0036] The timestamps of the LiDAR and RGB-D camera are aligned;

[0037] "Radar-Horizontal" Coordinate System The axis points in the direction the intelligent vehicle is moving. The axis points to the left of the vehicle. The plane is parallel to the ground. Axis perpendicular to Flat and pointing upwards;

[0038] Step 1.3: Calculate the distance between all point clouds in the "radar-horizontal" coordinate system and the origin of the "radar-horizontal" coordinate system. Select point clouds with a distance less than 5, and use all selected point clouds with a distance less than 5 as the local observation range (the local observation range...). and The values ​​are all between +5 and -5 (inclusive).

[0039] Step 1.4: To reduce computational complexity, 1000 point clouds are uniformly selected within the local observation range obtained in Step 1.3 (for example, if the total number of point clouds after filtering in Step 1.3 is 10000, one point cloud is selected every 10 point clouds, resulting in 1000 point clouds). The selected 1000 point clouds are used as the training sample set; the selected 1000 point cloud data are called induced points.

[0040] Step 1.5: Each induced point is decomposed into its position on a two-dimensional plane. and the corresponding elevation values A dataset containing 1000 observation points was constructed based on 1000 point cloud data. ;

[0041] in, Represents the first in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane Represents the first in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane Represents the first in the "radar-horizontal" coordinate system Elevation values ​​of point clouds; ;

[0042] Step 1.6: Using the dataset The regression model of the two-dimensional sparse Gaussian process is trained to obtain the trained regression model of the two-dimensional sparse Gaussian process.

[0043] Step 1.7: Discretize the local observation area to obtain a rasterized map; specifically:

[0044] The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.2, resulting in 2500 ( Point cloud positions on a two-dimensional plane (2500 points, 2500 small grids);

[0045] The point cloud positions on each two-dimensional plane obtained in step 1.7 Input the regression model of the two-dimensional sparse Gaussian process trained in step 1.6. The regression model of the two-dimensional sparse Gaussian process trained in step 1.6 outputs the elevation value, gradient value and uncertainty of the corresponding point cloud.

[0046] Step 1.8: Based on the 2500 point cloud locations on the two-dimensional plane obtained in Step 1.7 The elevation values ​​of the corresponding point cloud output by the trained two-dimensional sparse Gaussian process regression model constitute a set of 2500 points. Elevation map ;

[0047] Based on the 2500 point cloud locations on the two-dimensional plane obtained in step 1.7 The gradient values ​​of the corresponding point cloud output by the trained 2D sparse Gaussian process regression model constitute a set of 2500 points. Slope map ;

[0048] Based on the 2500 point cloud locations on the two-dimensional plane obtained in step 1.7 The uncertainty of the corresponding point cloud output by the regression model of the trained two-dimensional sparse Gaussian process, and the uncertainty of the corresponding point cloud, constitute a set of 2500 points. Uncertainty map ;

[0049] Step 1.9: Construct a flatness map and elevation change map and elevation cost map ;

[0050] Step 1.10: Use the elevation cost map constructed in Step 1.9 The elevation cost values ​​in the map are normalized (using values ​​in the [0,1] interval to represent drivability), resulting in a normalized elevation cost map. ;

[0051] The elevation cost refers to the elevation cost map. Point cloud location The third element outside;

[0052] Step 1.11: Using action communication, transfer the normalized elevation cost map obtained in Step 1.10. And the uncertainty map obtained in step 1.8 The information is sent in the form of topics and received by the client.

[0053] The other steps and parameters are the same as in Specific Implementation Method 1.

[0054] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that step 1.6 uses a dataset. The regression model of the two-dimensional sparse Gaussian process is trained to obtain the trained regression model of the two-dimensional sparse Gaussian process.

[0055] The specific process is as follows:

[0056] Dataset The first radar in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane The dataset is used as input to the regression model of a two-dimensional sparse Gaussian process. The first radar in the "radar-horizontal" coordinate system The elevation values ​​of the point cloud are used as the output of the regression model of the two-dimensional sparse Gaussian process. The regression model of the two-dimensional sparse Gaussian process is trained to obtain the trained regression model of the two-dimensional sparse Gaussian process.

[0057] The regression model of a two-dimensional sparse Gaussian process is expressed as:

[0058]

[0059] in

[0060] SGP stands for Sparse Gaussian process regression notation;

[0061] To show obedience;

[0062] Represents the dataset The first radar in the "radar-horizontal" coordinate system The elevation values ​​corresponding to each point cloud on a two-dimensional plane;

[0063] Represents the dataset The first radar in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane;

[0064] Represents the dataset The first radar in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane ;

[0065] Represents the mean function;

[0066] Represents the squared exponent kernel;

[0067] Length scale and signal variance These are the hyperparameters of the regression model for a two-dimensional sparse Gaussian process;

[0068] Based on length scale and signal variance Obtain the squared exponent kernel ;

[0069] In Gaussian process regression, an observation noise term that follows a zero-mean normal distribution is typically introduced. Reflecting the noise in actual measurements, This indicates that the mean is 0 and the variance is... The Gaussian distribution.

[0070] Other steps and parameters are the same as in specific implementation method one or two.

[0071] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step 1.9, a flatness map is constructed. and elevation change map and elevation cost map ;

[0072] The specific process is as follows:

[0073] Step 1.9.1: Construct a rasterized map, which serves as a flatness map. Specifically:

[0074] The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.2. The rasterized map serves as a flatness map. ;

[0075] Step 1.9.2: Construct a space of size Sliding window;

[0076] In elevation map The step size is 1, and the size is... The sliding window is scrolled to obtain the point cloud within each sliding window area;

[0077] Step 1.9.3: Solve for the fitting plane of the point cloud within each sliding window region using the least squares method, and obtain the value of the plane normal vector of the corresponding sliding window region based on each fitting plane;

[0078] Calculate the angle between the plane normal vector of each sliding window region and the z-axis of the world coordinate system, and store the angle in the flatness map corresponding to the center point cloud of the current sliding window region. middle;

[0079] Step 1.9.4: Construct a rasterized map, which will serve as an elevation change map. Specifically:

[0080] The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.2. The rasterized map serves as an elevation change map. ;

[0081] Step 1.9.5: On the elevation map The step size is 1, and the size is... The sliding window is scrolled to obtain the point cloud within each sliding window area;

[0082] Step 1.9.6: Find the maximum height difference between all point clouds in each sliding window and the current intelligent vehicle, and store the maximum height difference in the elevation change map corresponding to the center point cloud within the current sliding window area. middle;

[0083] Step 1.9.7: Slope map based on Step 1.8 The flatness map obtained in step 1.9.3 Elevation change map obtained in step 1.9.6 Constructing an elevation cost map , is represented as:

[0084]

[0085] in, , and These are weighting coefficients. ; , and These represent the critical thresholds for the maximum gradient, minimum flatness, and maximum elevation change that an intelligent vehicle can withstand before reaching an unsafe state.

[0086] These values ​​can be obtained from the manufacturer's manual for intelligent vehicles. For different types of intelligent vehicles, adjustments can be made appropriately based on their driving capabilities on different terrains. , and The value of .

[0087] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0088] Specific Implementation Method 5: This implementation method differs from one of the specific implementation methods one to four in that, in step 2, a dataset is obtained, and each data in the dataset includes an image and the label corresponding to the image;

[0089] The images and labels in the data are remapped to obtain remapped images and labels, which are used for subsequent semantic segmentation results.

[0090] The remapped image is used as the input to the image processing model, and the label corresponding to the remapped image is used as the output of the image processing model. The image processing model is trained until the total loss function converges, and the trained image processing model is obtained.

[0091] The specific process is as follows:

[0092] Step 2.1: Obtain the dataset from the RUGD public dataset. Each data point in the dataset includes an image and its corresponding label (labels are [0,24], 25 labels).

[0093] The images and labels in each dataset are remapped to obtain remapped images and labels, which are then used for subsequent semantic segmentation results.

[0094] The size of the remapped image is ;

[0095] in, The height of the remapped image. The width of the remapped image. Indicates the number of channels;

[0096] Step 2.2: Segment the remapped image into... Image blocks;

[0097] Each Image block input block 1, block 1 output feature map Feature map Input block 2, output feature map of block 2 Feature map Input block 3, output feature map of block 3 Feature map Input block 4, output feature map of block 4 ;

[0098] Step 2.3:

[0099] Output feature map of block 1 Resampling to shape The resampled feature map is obtained. ;

[0100] Output feature map of block 2 Resampling to shape The resampled feature map is obtained. ;

[0101] Output feature map of block 3 Resampling to shape The resampled feature map is obtained. ;

[0102] Output feature map of block 4 Resampling to shape The resampled feature map is obtained. ;

[0103] in, In order to be in , , or The height corresponding to any selected channel; In order to be in , , or The width corresponding to any selected channel;

[0104] for example, In order to be in The height corresponding to the selected channel ; In order to be in The width corresponding to the selected channel Then the feature map of block 1 will be output. Resampling to shape The resampled feature map is obtained. ; Output feature map of block 2 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 3 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 4 Resampling to shape The resampled feature map is obtained. ;

[0105] In order to be in The height corresponding to the selected channel ; In order to be in The width corresponding to the selected channel Then the feature map of block 1 will be output. Resampling to shape The resampled feature map is obtained. ; Output feature map of block 2 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 3 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 4 Resampling to shape The resampled feature map is obtained. ;

[0106] In order to be in The height corresponding to the selected channel ; In order to be in The width corresponding to the selected channel Then the feature map of block 1 will be output. Resampling to shape The resampled feature map is obtained. ; Output feature map of block 2 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 3 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 4 Resampling to shape The resampled feature map is obtained. ;

[0107] In order to be in The height corresponding to the selected channel ; In order to be in The width corresponding to the selected channel Then the feature map of block 1 will be output. Resampling to shape The resampled feature map is obtained. ; Output feature map of block 2 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 3 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 4 Resampling to shape The resampled feature map is obtained. ;

[0108] Resampled feature map Feature maps after resampling Feature maps after resampling Feature maps after resampling The features are concatenated along the channel dimension to obtain the concatenated features. ;

[0109] Features after splicing Size: ;

[0110] This step effectively integrates information from multiple scales, enabling the model to better understand contextual information at different scales;

[0111] Step 2.4: Combine the spliced ​​features Input a multi-head self-attention model, and the multi-head self-attention model outputs new features. ;

[0112] The spliced ​​features and new features Concatenate, then input a... Convolutional layers further compress features and extract important information to obtain feature maps. ;

[0113] The number of heads is 6;

[0114] In a multi-head self-attention model, each single-head self-attention model outputs one attention matrix, and a 6-head self-attention model outputs 6 attention matrices.

[0115] Step 2.5: Based on feature maps The cross-entropy loss is calculated based on the category result of each pixel and the category label of the corresponding pixel in the remapped image obtained in step 2.1, and is denoted as the principal loss.

[0116] Step 2.6: Calculate the auxiliary branch loss based on the six attention matrices output by the six single-head self-attention models and the labels of the corresponding pixels in the remapped image obtained in Step 2.1;

[0117] Step 2.7: The total loss is calculated by weighted summation of the main loss calculated in Step 2.5 and the auxiliary branch loss calculated in Step 2.6;

[0118] ; ; , As weight;

[0119] By introducing this auxiliary supervision mechanism, the problem of accuracy reduction caused by reduced resolution is effectively mitigated without significantly increasing computational complexity, thus achieving a better balance between training speed and recognition accuracy.

[0120] Step 2.8: Use the remapped image as the input to the image processing model and the corresponding label of the remapped image as the output of the image processing model. Train the image processing model until the total loss function converges to obtain the trained image processing model.

[0121] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0122] Specific Implementation Method Six: This implementation method differs from one of Specific Implementation Methods One to Five in that, in step 2.1, a dataset is obtained from the RUGD public dataset. Each data in the dataset includes an image and the corresponding label for the image (the label is [0,24], 25 labels).

[0123] The images and labels in each dataset are remapped to obtain remapped images and labels, which are then used for subsequent semantic segmentation results.

[0124] The size of the remapped image is ;

[0125] in, The height of the remapped image. The width of the remapped image. Indicates the number of channels;

[0126] The specific process is as follows:

[0127] Step 2.1.1: Define the new category label mapping and the color mapping corresponding to each category label; the process is as follows:

[0128] Six new category labels are defined: 0, 1, 2, 3, 4, and 5.

[0129] 0 represents the background; 1 represents a puddle; 2 represents an obstacle; 3 represents smooth terrain; 4 represents rough terrain; 5 represents rugged terrain.

[0130] For example, soil corresponds to the label "rough terrain", concrete corresponds to the label "smooth terrain", trees correspond to the label "obstacles", the sky corresponds to the label "background", sand and gravel corresponds to the label "rugged terrain", and water corresponds to the label "puddles".

[0131] Define the color mapping for each type of label, specifically as follows:

[0132] The background corresponds to black, smooth terrain to white, rough terrain to blue, rugged terrain to yellow, puddles to purple, and obstacles to green;

[0133] Step 2.1.2: Map the labels corresponding to the images in the data according to the newly defined category labels, and regenerate new category labels;

[0134] For example, the labels corresponding to the images in the data are [0, 24] (25 labels);

[0135] Before mapping: 0: Blank; 1: Soil; 2: Sand; 3: Grass; 4: Trees; 5: Pole (utility pole / pole); 6: Water; 7: Sky; 8: Vehicle; 9: Container; 10: Asphalt road; 11: Gravel; 12: Building; 13: Mulch (bark mulch); 14: Gravel bed; 15: Log; 16: Bicycle; 17: Pedestrian; 18: Fence; 19: Shrub; 20: Sign; 21: Rock; 22: Bridge; 23: Concrete; 24: Picnic table;

[0136] After mapping: Background (0): blank, sky, signboard; Smooth terrain (1): concrete, asphalt road; Rough terrain (2): gravel, grass, soil, sand, cover (bark cover); Rugged terrain (3): rocks, gravel bed; Waterhole (4): water body; Obstacles (5): trees, poles (power poles / pole), vehicles, containers, buildings, logs, bicycles, pedestrians, fences, shrubs, picnic tables, bridge;

[0137] The six new category labels are 0, 1, 2, 3, 4, and 5.

[0138] Map the 25 labels corresponding to the images in the data to 6 labels according to the newly defined category labels;

[0139] Step 2.1.3: Based on the new category labels regenerated in Step 2.1.2, obtain the remapped image according to the color mapping corresponding to each newly defined category label;

[0140] For example, 0 corresponds to black, 1 to purple, 2 to green, 3 to white, 4 to blue, and 5 to yellow.

[0141] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0142] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that, in step 2.2, the remapped image is segmented into... Image blocks;

[0143] Each Image block input block 1, block 1 output feature map Feature map Input block 2, output feature map of block 2 Feature map Input block 3, output feature map of block 3 Feature map Input block 4, output feature map of block 4 ;

[0144] The specific process is as follows:

[0145] Step 2.21: Segment the remapped image into... Image blocks;

[0146] Step 2.22: Each Image block input block 1, block 1 output feature map ;

[0147] The specific process is as follows:

[0148] Each Image blocks are input to the first single-head self-attention mechanism module, and the first single-head self-attention mechanism module outputs... ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature maps are obtained by splicing. ;

[0149] Each The image block is processed by block 1 to obtain the feature map. Feature map The resolution becomes:

[0150]

[0151] in, and The width and height of the image after remapping in step 2.1 for The number of channels the image block passes through in block 1;

[0152] For passing blocks The height of the feature map, For passing blocks The width of the feature map;

[0153] The resolution is obtained After obtaining the multi-level features, these multi-level features are passed to the decoder.

[0154] Step 2.23: Feature Map Input block 2, output feature map of block 2 ;

[0155] The specific process is as follows:

[0156] Feature map Input to the second single-head self-attention mechanism module, output of the second single-head self-attention mechanism module ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature map obtained by splicing ;

[0157] Feature map Input to the third single-head self-attention mechanism module; output of the third single-head self-attention mechanism module. ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Concatenation yields the output feature map ;

[0158] Feature map After processing by block 2, the feature map is obtained. Feature map The resolution becomes:

[0159]

[0160] in, and The width and height of the image after remapping in step 2.1 For feature map The number of channels passing through block 2;

[0161] The resolution is obtained After obtaining the multi-level features, these multi-level features are passed to the decoder.

[0162] Step 2.24: Feature Map Input block 3, output feature map of block 3 ;

[0163] The specific process is as follows:

[0164] Feature map Input to the fourth single-head self-attention mechanism module; output of the fourth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0165] Feature map Input to the fifth single-head self-attention mechanism module; output of the fifth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0166] Feature map Input to the sixth single-head self-attention mechanism module, output of the sixth single-head self-attention mechanism module ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0167] Feature map Input to the seventh single-head self-attention mechanism module, output of the seventh single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0168] Feature map Input to the eighth single-head self-attention mechanism module, output of the eighth single-head self-attention mechanism module. ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature maps are obtained by splicing. ;

[0169] Feature map The feature map is obtained after processing by block 3. Feature map The resolution becomes:

[0170]

[0171] in, and The width and height of the image after remapping in step 2.1 For feature map The number of channels passing through block 3;

[0172] The resolution is obtained After obtaining the multi-level features, these multi-level features are passed to the decoder.

[0173] Step 2.25: Feature Map Input block 4, output feature map of block 4 ;

[0174] The specific process is as follows:

[0175] Feature map Input to the ninth single-head self-attention mechanism module, output of the ninth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0176] Feature map Input to the tenth single-head self-attention mechanism module, output of the tenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0177] Feature map Input to the eleventh single-head self-attention mechanism module, output of the eleventh single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0178] Feature map Input the twelfth single-head self-attention mechanism module, output the twelfth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0179] Feature map Input to the thirteenth single-head self-attention mechanism module; output of the thirteenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0180] Feature map Input to the fourteenth single-head self-attention mechanism module; output of the fourteenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0181] Feature map Input the fifteenth single-head self-attention mechanism module, output the fifteenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ;

[0182] Feature map Input to the sixteenth single-head self-attention mechanism module; output of the sixteenth single-head self-attention mechanism module. ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature maps are obtained by splicing. ;

[0183] Feature map After processing by block 4, the feature map is obtained. Feature map The resolution becomes:

[0184]

[0185] in, and The width and height of the image after remapping in step 2.1 For feature map The number of channels passing through block 4;

[0186] The resolution is obtained After obtaining the multi-level features, these multi-level features are passed to the decoder.

[0187] Four blocks are used to obtain multi-layer features with different resolutions, where each block further reduces the spatial resolution by half in both height and width dimensions. This preserves features at different resolutions for subsequent feature reconstruction.

[0188] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0189] Specific Implementation Method Eight: This implementation method differs from any of Specific Implementation Methods One to Seven in that, in step 2.6, the auxiliary branch loss is calculated based on the six attention matrices output by the six single-head self-attention models and the labels of the corresponding pixels in the remapped image obtained in step 2.1.

[0190] The specific process is as follows:

[0191] Extract the diagonal elements of the attention matrix output by the first single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the first self-attention score map.

[0192] Extract the diagonal elements of the attention matrix output by the second single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the second self-attention score map.

[0193] Extract the diagonal elements of the attention matrix output by the third single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the third self-attention score map.

[0194] Extract the diagonal elements of the attention matrix output by the fourth single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the fourth self-attention score map.

[0195] Extract the diagonal elements of the attention matrix output by the 5th single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the 5th self-attention score map.

[0196] Extract the diagonal elements of the attention matrix output by the 6th single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the 6th self-attention score map.

[0197] Calculate the cross-entropy loss between the category result of each pixel in the first self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1 to constrain the spatial consistency of the attention distribution;

[0198] Calculate the cross-entropy loss between the category result of each pixel in the second self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1 to constrain the spatial consistency of the attention distribution;

[0199] Calculate the cross-entropy loss between the category result of each pixel in the third self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1 to constrain the spatial consistency of the attention distribution;

[0200] Calculate the cross-entropy loss between the category result of each pixel in the fourth self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1 to constrain the spatial consistency of the attention distribution;

[0201] Calculate the cross-entropy loss between the category result of each pixel in the 5th self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1 to constrain the spatial consistency of the attention distribution;

[0202] Calculate the cross-entropy loss between the category result of each pixel in the 6th self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1 to constrain the spatial consistency of the attention distribution;

[0203] The sum of the six cross-entropy losses is used as the auxiliary branch loss;

[0204] In decreasing and While the training and inference speed of the model can be significantly improved, it will also lead to a decrease in terrain recognition accuracy. In order to alleviate the performance degradation caused by the reduction in resolution while ensuring the training efficiency of the model, this invention introduces an intra-group self-attention mechanism into the network structure, which effectively improves the model's perception accuracy of key terrain features.

[0205] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0206] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One to Eight in that, in step 3...

[0207] The image data acquired by the vehicle-mounted RGB-D camera is input into the image processing model trained in step 2. The trained image processing model outputs the terrain category (6 types) corresponding to each pixel in the image, with a value range of [0,5].

[0208] Based on the remapping rules, the terrain category is converted into a binary navigability value. ;

[0209] Combined with binary navigability value Obtain the point cloud; obtain the semantic cost map based on the point cloud;

[0210] The specific process is as follows:

[0211] Step 3.1: To reduce computational complexity, based on the intelligent vehicle's driving capabilities, the six categories in Step 2.1 are remapped to "accessible areas" and "impassable areas"; the process is as follows:

[0212] Map the background, puddles, and obstacles as impassable areas, and map smooth, rough, and rugged terrain as passable areas.

[0213] If the vehicle's driving ability is weak, rugged terrain can also be mapped as an impassable area;

[0214] Step 3.2: Using the node-based programming approach of the ROS system, create an action service object. The action service object receives the depth point cloud data / realsense / depth / color / points and image data / realsense / color / image_raw output from the vehicle-mounted RGB-D camera.

[0215] Realsense refers to the model of the RGB-D camera, depth refers to depth, color refers to color, points refers to point cloud, and / realsense / depth / color / points refers to the depth and point cloud topic published by the Realsense camera in ROS.

[0216] image_raw represents image data, and / realsense / color / image_raw represents an image topic published by a RealSense camera in ROS;

[0217] The image size output by the vehicle-mounted RGB-D camera is [size missing]. ;

[0218] Meanwhile, the timestamps of the vehicle-mounted RGB-D camera and the LiDAR are aligned;

[0219] Step 3.3: Input the image data output by the vehicle-mounted RGB-D camera into the image processing model trained in Step 2. The trained image processing model outputs the category labels (6 types) corresponding to each pixel of the image. The value range of the category labels is [0,5].

[0220] 0 represents the background; 1 represents a puddle; 2 represents an obstacle; 3 represents smooth terrain; 4 represents rough terrain; 5 represents rugged terrain.

[0221] Based on the remapping rules described in step 3.1, the category labels corresponding to each pixel in the output image of the trained image processing model are converted into binary navigability values. ;

[0222] Each pixel in the image data output by the vehicle-mounted RGB-D camera has a binary navigability value. ;

[0223] The binary navigability value corresponding to category labels 0, 1, and 2 is represented as 1;

[0224] The binary navigability value corresponding to category labels 3, 4, and 5 is 0;

[0225] Step 3.4: The depth point cloud data output by the vehicle-mounted RGB-D camera received by the action service object is represented as a four-tuple. ;

[0226] in, For three-dimensional spatial position, For color information;

[0227] The depth point cloud received by the action service object is located in the optical coordinate system of the vehicle-mounted RGB-D camera;

[0228] Transform the depth point cloud in the optical coordinate system of the vehicle-mounted RGB-D camera to the "radar-horizontal" coordinate system;

[0229] The optical coordinate system of the vehicle-mounted RGB-D camera points in the same direction as the coordinate system of the lidar.

[0230] The point cloud output by the RGB-D camera is already a mapped 3D coordinate data;

[0231] Step 3.5: Based on the point cloud data in the "radar-horizontal" coordinate system received in Step 3.4 and the binary navigability value in Step 3.3. , obtain point cloud ;

[0232] Step 3.6: Construct a rasterized map, which serves as a semantic cost map. Specifically:

[0233] The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.2. The rasterized map serves as a semantic cost map. ;

[0234] Step 3.7: Map the point cloud obtained in Step 3.5 to the semantic cost map. Above, we obtain the semantic cost map. ;

[0235] Step 3.8: Use action communication to transmit the semantic cost map obtained in Step 3.7. It is sent as a topic and received by the client.

[0236] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0237] Specific Implementation Method 10: This implementation method differs from one of the specific implementation methods one to nine in that, in step 4, the uncertainty map and semantic cost map are used as constraint masks for the elevation cost map to generate the final traversability map.

[0238] The specific process is as follows:

[0239] Step 4.1: Create an action communication client, open rviz, select the 2D target point, and send the target point data to the server;

[0240] Step 4.2: Receive the elevation cost map from the server (step 1.11). Uncertainty Map in Step 1.11 and the semantic cost map in step 3.8 data;

[0241] Step 4.3: Construct a accessibility map; the specific process is as follows:

[0242] Step 4.31: Uncertainty Map of Step 1.11 As an elevation cost map The mask, by making the uncertainty exceed a critical threshold The point is marked as Uncertainty graph Middle position This is an impassable or unsafe area; the expression is as follows:

[0243]

[0244] in,

[0245] Elevation cost map Middle position The value;

[0246] Uncertainty diagram Middle position The value;

[0247] Step 4.32: Semantic Cost Map from Step 3.8 As an elevation cost map The mask is created by marking points in the semantic cost map that are impassable. The expression is as follows:

[0248]

[0249] in,

[0250] Semantic cost map Middle position The value;

[0251] Step 4.33: Elevation Cost Map The points in the map that exclude impassable areas are the accessibility map.

[0252] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0253] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for constructing intelligent vehicle accessibility maps based on multi-sensor fusion, characterized in that: The specific process of the method is as follows: Step 1: Based on lidar point cloud data, fuse a sparse Gaussian process model to obtain an elevation cost map and an uncertainty map; Step 2: Obtain the dataset. Each data point in the dataset includes an image and its corresponding label. The images and labels in the data are remapped to obtain the remapped images and labels. The remapped image is used as the input to the image processing model, and the label corresponding to the remapped image is used as the output of the input image processing model. The image processing model is trained until the total loss function converges, and the trained image processing model is obtained. Step 3: Input the image data acquired by the vehicle-mounted RGB-D camera into the image processing model trained in Step 2. The image processing model outputs the class label values ​​of each pixel in the image, which range from [0,5]. Based on the remapping rules, the terrain category labels are converted into binary navigability values. ; Combined with binary navigability value Obtain the point cloud; obtain the semantic cost map based on the point cloud; Step 4: Use the uncertainty map and semantic cost map as constraint masks for the elevation cost map to generate the final drivability map.

2. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 1, characterized in that: In step 1, based on lidar point cloud data, a sparse Gaussian process model is fused to obtain an elevation cost map and an uncertainty map. The specific process is as follows: Step 1.1: Using the node-based programming method of the ROS system, create an action service object. The action service object receives the point cloud data / points output by the vehicle-mounted LiDAR. The point cloud data is located in the LiDAR coordinate system. The lidar coordinate system uses a right-handed coordinate system, where The axis points in the direction the intelligent vehicle is moving. The axis points to the left of the vehicle. Axis perpendicular to Flat and pointing upwards; Step 1.2: Based on the current pitch and roll angles of the lidar, transform the point cloud data in the lidar coordinate system to the "radar-horizontal" coordinate system to obtain the point cloud data in the "radar-horizontal" coordinate system; The timestamps of the LiDAR and RGB-D camera are aligned; "Radar-Horizontal" Coordinate System The axis points in the direction the intelligent vehicle is moving. The axis points to the left of the vehicle. The plane is parallel to the ground. Axis perpendicular to Flat and pointing upwards; Step 1.3: Calculate the distance between all point clouds in the "radar-horizontal" coordinate system and the origin of the "radar-horizontal" coordinate system, filter out point clouds with a distance less than 5, and take all the filtered point clouds with a distance less than 5 as the local observation range; Step 1.4: Select 1000 point clouds within the local observation range obtained in Step 1.

3. These 1000 point clouds are used as the training sample set; the selected 1000 point cloud data are called induced points. Step 1.5: Each induced point is decomposed into its position on a two-dimensional plane. and the corresponding elevation values A dataset containing 1000 observation points was constructed based on 1000 point cloud data. ; in, Represents the first in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane Represents the first [radar] in the "horizontal" coordinate system. The position of a point cloud on a two-dimensional plane Represents the first [radar] in the "horizontal" coordinate system. Elevation values ​​of point clouds; ; Step 1.6: Using the dataset The regression model of the two-dimensional sparse Gaussian process is trained to obtain the trained regression model of the two-dimensional sparse Gaussian process. Step 1.7: Discretize the local observation area to obtain a rasterized map; specifically: The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.2, resulting in 2500 point cloud locations on a two-dimensional plane. ; The point cloud positions on each two-dimensional plane obtained in step 1.7 Input the regression model of the two-dimensional sparse Gaussian process trained in step 1.

6. The regression model of the two-dimensional sparse Gaussian process trained in step 1.6 outputs the elevation value, gradient value and uncertainty of the corresponding point cloud. Step 1.8: Based on the 2500 point cloud locations on the two-dimensional plane obtained in Step 1.7 The elevation values ​​of the corresponding point cloud output by the trained two-dimensional sparse Gaussian process regression model constitute a set of 2500 points. Elevation map ; Based on the 2500 point cloud locations on the two-dimensional plane obtained in step 1.7 The gradient values ​​of the corresponding point cloud output by the trained 2D sparse Gaussian process regression model constitute a set of 2500 points. Slope map ; Based on the 2500 point cloud locations on the two-dimensional plane obtained in step 1.7 The uncertainty of the corresponding point cloud output by the regression model of the trained two-dimensional sparse Gaussian process, and the uncertainty of the corresponding point cloud, constitute a set of 2500 points. Uncertainty map ; Step 1.9: Construct a flatness map and elevation change map and elevation cost map ; Step 1.10: Use the elevation cost map constructed in Step 1.9 The elevation cost values ​​in the data are normalized to obtain a normalized elevation cost map. ; The elevation cost refers to the elevation cost map. Point cloud location The third element outside; Step 1.11: Using action communication, transfer the normalized elevation cost map obtained in Step 1.

10. And the uncertainty map obtained in step 1.8 The information is sent in the form of topics and received by the client.

3. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 2, characterized in that: The dataset used in step 1.6 The regression model of the two-dimensional sparse Gaussian process is trained to obtain the trained regression model of the two-dimensional sparse Gaussian process. The specific process is as follows: Dataset The first radar in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane The dataset is used as input to the regression model of a two-dimensional sparse Gaussian process. The first radar in the "radar-horizontal" coordinate system The elevation values ​​of the point cloud are used as the output of the regression model of the two-dimensional sparse Gaussian process. The regression model of the two-dimensional sparse Gaussian process is trained to obtain the trained regression model of the two-dimensional sparse Gaussian process. The regression model of a two-dimensional sparse Gaussian process is expressed as: in SGP stands for Sparse Gaussian process regression notation; To show obedience; Represents the dataset The first radar in the "radar-horizontal" coordinate system The elevation values ​​corresponding to each point cloud on a two-dimensional plane; Represents the dataset The first radar in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane; Represents the dataset The first radar in the "radar-horizontal" coordinate system The position of a point cloud on a two-dimensional plane ; Represents the mean function; Represents the squared exponent kernel; Length scale and signal variance represents the hyperparameters of the regression model for a two-dimensional sparse Gaussian process.

4. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 3, characterized in that: In step 1.9, a flatness map is constructed. and elevation change map and elevation cost map ; The specific process is as follows: Step 1.9.1: Construct a rasterized map, which serves as a flatness map. Specifically: The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.

2. The rasterized map serves as a flatness map. ; Step 1.9.2: Construct a space of size Sliding window; In elevation map The step size is 1, and the size is... The sliding window is scrolled to obtain the point cloud within each sliding window area; Step 1.9.3: Solve for the fitting plane of the point cloud within each sliding window region using the least squares method, and obtain the value of the plane normal vector of the corresponding sliding window region based on each fitting plane; Calculate the angle between the plane normal vector of each sliding window region and the z-axis of the world coordinate system, and store the angle in the flatness map corresponding to the center point cloud of the current sliding window region. middle; Step 1.9.4: Construct a rasterized map, which will serve as an elevation change map. Specifically: The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.

2. The rasterized map serves as an elevation change map. ; Step 1.9.5: On the elevation map The step size is 1, and the size is... The sliding window is scrolled to obtain the point cloud within each sliding window area; Step 1.9.6: Find the maximum height difference between all point clouds in each sliding window and the current intelligent vehicle, and store the maximum height difference in the elevation change map corresponding to the center point cloud within the current sliding window area. middle; Step 1.9.7: Slope map based on Step 1.8 The flatness map obtained in step 1.9.3 Elevation change map obtained in step 1.9.6 Constructing an elevation cost map , is represented as: in, , and These are weighting coefficients. ; , and These represent the critical thresholds for the maximum gradient, minimum flatness, and maximum elevation change that an intelligent vehicle can withstand before reaching an unsafe state.

5. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 4, characterized in that: In step 2, a dataset is obtained, and each data point in the dataset includes an image and its corresponding label. The images and labels in the data are remapped to obtain the remapped images and labels. The remapped image is used as the input to the image processing model, and the label corresponding to the remapped image is used as the output of the image processing model. The image processing model is trained until the total loss function converges, and the trained image processing model is obtained. The specific process is as follows: Step 2.1: Obtain the dataset from the RUGD public dataset. Each data point in the dataset includes an image and its corresponding label. Remap the images and labels in each dataset to obtain the remapped images and labels; The size of the remapped image is ; in, The height of the remapped image. The width of the remapped image. Indicates the number of channels; Step 2.2: Segment the remapped image into... Image blocks; Each Image block input block 1, block 1 output feature map Feature map Input block 2, output feature map of block 2 Feature map Input block 3, output feature map of block 3 Feature map Input block 4, output feature map of block 4 ; Step 2.3: Output feature map of block 1 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 2 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 3 Resampling to shape The resampled feature map is obtained. ; Output feature map of block 4 Resampling to shape The resampled feature map is obtained. ; in, In order to be in , , or The height corresponding to any selected channel; In order to be in , , or The width corresponding to any selected channel; Resampled feature map Feature maps after resampling Feature maps after resampling Feature maps after resampling The features are concatenated along the channel dimension to obtain the concatenated features. ; Features after splicing Size: ; Step 2.4: Combine the spliced ​​features Input a multi-head self-attention model, and the multi-head self-attention model outputs new features. ; The spliced ​​features and new features Concatenate, then input a... Convolutional layers produce feature maps. ; The number of heads is 6; In a multi-head self-attention model, each single-head self-attention model outputs one attention matrix, and a 6-head self-attention model outputs 6 attention matrices. Step 2.5: Based on feature maps The cross-entropy loss is calculated based on the category result of each pixel and the category label of the corresponding pixel in the remapped image obtained in step 2.1, and is denoted as the principal loss. Step 2.6: Calculate the auxiliary branch loss based on the six attention matrices output by the six single-head self-attention models and the labels of the corresponding pixels in the remapped image obtained in Step 2.1; Step 2.7: The total loss is calculated by weighted summation of the main loss calculated in Step 2.5 and the auxiliary branch loss calculated in Step 2.6; ; ; , As weight; Step 2.8: Use the remapped image as the input to the image processing model and the corresponding label of the remapped image as the output of the image processing model. Train the image processing model until the total loss function converges to obtain the trained image processing model.

6. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 5, characterized in that: In step 2.1, the dataset is obtained from the RUGD public dataset, and each data in the dataset includes an image and the corresponding label for the image; Remap the images and labels in each dataset to obtain the remapped images and labels; The size of the remapped image is ; in, The height of the remapped image. The width of the remapped image. Indicates the number of channels; The specific process is as follows: Step 2.1.1: Define the new category label mapping and the color mapping corresponding to each category label; the process is as follows: Six new category labels are defined: 0, 1, 2, 3, 4, and 5. 0 represents the background; 1 represents a puddle; 2 represents an obstacle; 3 represents smooth terrain; 4 represents rough terrain; 5 represents rugged terrain. Define the color mapping for each type of label, specifically as follows: The background corresponds to black, smooth terrain to white, rough terrain to blue, rugged terrain to yellow, puddles to purple, and obstacles to green; Step 2.1.2: Map the labels corresponding to the images in the data according to the newly defined category labels, and regenerate new category labels; Step 2.1.3: Based on the new category labels regenerated in Step 2.1.2, obtain the remapped image according to the color mapping corresponding to each newly defined category label.

7. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 6, characterized in that: In step 2.2, the remapped image is segmented into... Image blocks; Each Image block input block 1, block 1 output feature map Feature map Input block 2, output feature map of block 2 Feature map Input block 3, output feature map of block 3 Feature map Input block 4, output feature map of block 4 ; The specific process is as follows: Step 2.21: Segment the remapped image into... Image blocks; Step 2.22: Each Image block input block 1, block 1 output feature map ; The specific process is as follows: Each Image blocks are input to the first single-head self-attention mechanism module, and the first single-head self-attention mechanism module outputs... ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature maps are obtained by splicing. ; Feature map The resolution becomes: in, and The width and height of the image after remapping in step 2.1 for The number of channels the image block passes through in block 1; For passing blocks The height of the feature map, For passing blocks The width of the feature map; Step 2.23: Feature Map Input block 2, output feature map of block 2 ; The specific process is as follows: Feature map Input to the second single-head self-attention mechanism module, output of the second single-head self-attention mechanism module ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature map obtained by splicing ; Feature map Input to the third single-head self-attention mechanism module; output of the third single-head self-attention mechanism module. ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Concatenation yields the output feature map ; Feature map The resolution becomes: in, and The width and height of the image after remapping in step 2.1 For feature map The number of channels passing through block 2; Step 2.24: Feature Map Input block 3, output feature map of block 3 ; The specific process is as follows: Feature map Input to the fourth single-head self-attention mechanism module; output of the fourth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the fifth single-head self-attention mechanism module; output of the fifth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the sixth single-head self-attention mechanism module, output of the sixth single-head self-attention mechanism module ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the seventh single-head self-attention mechanism module, output of the seventh single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the eighth single-head self-attention mechanism module, output of the eighth single-head self-attention mechanism module. ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature maps are obtained by splicing. ; Feature map The resolution becomes: in, and The width and height of the image after remapping in step 2.1 For feature map The number of channels passing through block 3; Step 2.25: Feature Map Input block 4, output feature map of block 4 ; The specific process is as follows: Feature map Input to the ninth single-head self-attention mechanism module, output of the ninth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the tenth single-head self-attention mechanism module, output of the tenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the eleventh single-head self-attention mechanism module, output of the eleventh single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input the twelfth single-head self-attention mechanism module, output the twelfth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the thirteenth single-head self-attention mechanism module; output of the thirteenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the fourteenth single-head self-attention mechanism module; output of the fourteenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input the fifteenth single-head self-attention mechanism module, output the fifteenth single-head self-attention mechanism module. ; Input to a feedforward neural network FFN, the feedforward neural network FFN and splicing output feature map ; Feature map Input to the sixteenth single-head self-attention mechanism module; output of the sixteenth single-head self-attention mechanism module. ; The input is a feedforward neural network (FFN), and the output of the feedforward neural network (FFN) is the same as the input. Feature maps are obtained by splicing. ; Feature map The resolution becomes: in, and The width and height of the image after remapping in step 2.1 For feature map The number of channels passing through block 4.

8. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 7, characterized in that: In step 2.6, the auxiliary branch loss is calculated based on the six attention matrices output by the six single-head self-attention models and the labels of the corresponding pixels in the remapped image obtained in step 2.

1. The specific process is as follows: Extract the diagonal elements of the attention matrix output by the first single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the first self-attention score map. Extract the diagonal elements of the attention matrix output by the second single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the second self-attention score map. Extract the diagonal elements of the attention matrix output by the third single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the third self-attention score map. Extract the diagonal elements of the attention matrix output by the fourth single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the fourth self-attention score map. Extract the diagonal elements of the attention matrix output by the 5th single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the 5th self-attention score map. Extract the diagonal elements of the attention matrix output by the 6th single-head self-attention model, adjust the size of the diagonal elements by bilinear interpolation, and obtain the adjusted diagonal element values. The adjusted diagonal element values ​​represent the 6th self-attention score map. Calculate the cross-entropy loss between the category result of each pixel in the first self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1; Calculate the cross-entropy loss between the category result of each pixel in the second self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1; Calculate the cross-entropy loss between the category result of each pixel in the third self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1; Calculate the cross-entropy loss between the category result of each pixel in the fourth self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1; Calculate the cross-entropy loss between the category result of each pixel in the 5th self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1; Calculate the cross-entropy loss between the category result of each pixel in the 6th self-attention score map and the label of the corresponding pixel in the remapped image obtained in step 2.1; The sum of the six cross-entropy losses is used as the auxiliary branch loss.

9. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 8, characterized in that: In step 3, the image data acquired by the vehicle-mounted RGB-D camera is input into the image processing model trained in step 2. The terrain category value range of each pixel in the image output by the trained image processing model is [0,5]. Based on the remapping rules, the terrain category is converted into a binary navigability value. ; Combined with binary navigability value Obtain the point cloud; obtain the semantic cost map based on the point cloud; The specific process is as follows: Step 3.1: Remap the 6 categories from Step 2.1 to "accessible regions" and "inaccessible regions"; the process is as follows: Map the background, puddles, and obstacles as impassable areas, and map smooth, rough, and rugged terrain as passable areas. Step 3.2: Using the node-based programming approach of the ROS system, create an action service object. The action service object receives the depth point cloud data / realsense / depth / color / points and image data / realsense / color / image_raw output from the vehicle-mounted RGB-D camera. Realsense refers to the model of the RGB-D camera, depth refers to depth, color refers to color, points refers to point cloud, and / realsense / depth / color / points refers to the depth and point cloud topic published by the Realsense camera in ROS. image_raw represents image data, and / realsense / color / image_raw represents an image topic published by a RealSense camera in ROS; The image size output by the vehicle-mounted RGB-D camera is [size missing]. ; Meanwhile, the timestamps of the vehicle-mounted RGB-D camera and the LiDAR are aligned; Step 3.3: Input the image data output by the vehicle-mounted RGB-D camera into the image processing model trained in Step 2. The trained image processing model outputs the category label corresponding to each pixel in the image. The category label value range is [0,5]. 0 represents the background; 1 represents a puddle; 2 represents an obstacle; 3 represents smooth terrain; 4 represents rough terrain; 5 represents rugged terrain. Based on the remapping rules described in step 3.1, the category labels corresponding to each pixel in the output image of the trained image processing model are converted into binary navigability values. ; Each pixel in the image data output by the vehicle-mounted RGB-D camera has a binary navigability value. ; The binary navigability value corresponding to category labels 0, 1, and 2 is represented as 1; The binary navigability value corresponding to category labels 3, 4, and 5 is 0; Step 3.4: The depth point cloud data output by the vehicle-mounted RGB-D camera received by the action service object is represented as a four-tuple. ; in, For three-dimensional spatial position, For color information; The depth point cloud received by the action service object is located in the optical coordinate system of the vehicle-mounted RGB-D camera; Transform the depth point cloud in the optical coordinate system of the vehicle-mounted RGB-D camera to the "radar-horizontal" coordinate system; The optical coordinate system of the vehicle-mounted RGB-D camera points in the same direction as the coordinate system of the lidar. Step 3.5: Based on the point cloud data in the "radar-horizontal" coordinate system received in Step 3.4 and the binary navigability value in Step 3.

3. , obtain point cloud ; Step 3.6: Construct a rasterized map, which serves as a semantic cost map. Specifically: The rasterized map has a total width of 10 and a total height of 10. Each small grid cell in the rasterized map has a width and height of 0.

2. The rasterized map serves as a semantic cost map. ; Step 3.7: Map the point cloud obtained in Step 3.5 to the semantic cost map. The semantic cost map is obtained from the above. ; Step 3.8: Use action communication to transmit the semantic cost map obtained in Step 3.

7. It is sent as a topic and received by the client.

10. The method for constructing a smart vehicle accessibility map based on multi-sensor fusion according to claim 9, characterized in that: In step 4, the uncertainty map and semantic cost map are used as constraint masks for the elevation cost map to generate the final drivability map. The specific process is as follows: Step 4.1: Create an action communication client, open rviz, select the 2D target point, and send the target point data to the server; Step 4.2: Receive the elevation cost map from the server (step 1.11). Uncertainty Map in Step 1.11 and the semantic cost map in step 3.8 data; Step 4.3: Construct a accessibility map; the specific process is as follows: Step 4.31: Uncertainty Map of Step 1.11 As an elevation cost map The mask, by making the uncertainty exceed a critical threshold The point is marked as Uncertainty graph Middle position This is an impassable or unsafe area; the expression is as follows: in, Elevation cost map Middle position The value; Uncertainty diagram Middle position The value; Step 4.32: Semantic Cost Map from Step 3.8 As an elevation cost map The mask is created by marking points in the semantic cost map that are impassable. The expression is as follows: in, Semantic cost map Middle position The value; Step 4.33: Elevation Cost Map The points in the map that exclude impassable areas are the accessibility map.

Images