Unstructured environment accessibility perception method that integrates visual semantics and radar geometry information

Abstract

This invention discloses a method for perceiving the accessibility of unstructured environments by fusing visual semantics and radar geometric information. The method includes: processing images using a lightweight network to obtain terrain semantic labels and probability distributions; preprocessing lidar point clouds and spatiotemporally registering them with the images; projecting semantic information onto a local raster map and constructing a semantic cost layer based on the physical properties of the terrain material; simultaneously, constructing a 3D elevation map based on the point cloud and pose, extracting slope and flatness features, and constructing a geometric cost layer. Finally, calculating the normalized information entropy based on the semantic probability distribution projected onto the raster, thereby generating an adaptive fusion factor, dynamically weighting and fusing semantic and geometric costs to generate a continuous accessibility cost map. This invention improves the accuracy and robustness of accessibility assessment for complex unstructured terrain through multi-source information complementarity and adaptive fusion.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing and multi-sensor fusion technology for unmanned platforms, and in particular to a method for perceiving unstructured environment accessibility based on the fusion of visual semantics and radar geometric information. Background Technology

[0002] With the widespread application of unmanned ground vehicles in disaster relief, field surveys, agricultural exploration, and military operations, environmental perception in unstructured field environments has become a research hotspot and challenge. In these complex and ever-changing scenarios, drivability perception is a crucial foundation for environmental understanding and autonomous decision-making. The accuracy of drivability assessment directly impacts the path planning, off-road navigation, obstacle avoidance, and mission execution capabilities of unmanned systems, thus becoming a core technology in intelligent mobile robot systems.

[0003] Traditional methods for perceiving the accessibility of unstructured environments typically rely on a single type of sensor, such as purely visual methods based on cameras or purely geometric inference methods based on lidar. Visual sensors can provide high-resolution image information, suitable for capturing rich environmental color, texture, and semantic features. However, in unstructured outdoor environments, their performance is easily affected by factors such as drastic changes in lighting, inclement weather, shadow occlusion, and similar terrain textures, leading to significant performance degradation. In contrast, lidar sensors offer superior resistance to ambient lighting interference and high-precision 3D spatial ranging capabilities, accurately characterizing terrain undulations and obstacle outlines. However, because the point clouds they acquire lack color and texture information, they struggle to accurately distinguish semantic categories of terrain with different physical support properties.

[0004] To address the limitations imposed by the lack of perception dimensions in single sensors, multi-sensor fusion has become a key approach to enhancing the perception of unstructured environments. In particular, the fusion of vision and radar can fully leverage the complementary advantages of the two types of sensors: utilizing the semantic recognition capabilities of visual sensors for terrain materials and categories, while relying on the geometric characterization capabilities of radar sensors for terrain elevation, slope, and obstacles, thereby achieving a more comprehensive, accurate, and robust accessibility assessment.

[0005] Existing methods for fusion-based accessibility perception mainly involve joint modeling of geometric and semantic features. On the one hand, geometric costs such as terrain slope and flatness extracted from radar elevation maps or raster maps can effectively avoid the risks of physical collisions and overturning, but they are prone to classifying "pseudo-obstacles" such as tall grass and shallow puddles as impassable areas. On the other hand, semantic costs extracted from deep learning image segmentation can assign corresponding friction and stiffness weights to different terrains, but they lack three-dimensional constraints in physical space.

[0006] While the fusion of vision and radar in accessibility perception offers significant theoretical advantages, it still faces a series of challenges in practical deployment and application in unstructured environments. First, the uneven pixel distribution and high feature similarity of terrain categories in the wild make obtaining high-precision two-dimensional visual semantic masks challenging while maintaining system real-time performance. Second, visual cameras and LiDAR differ significantly in data structure, field of view, and sampling frequency, making it difficult to maintain high-precision spatiotemporal synchronization and spatial registration of heterogeneous data under complex terrain conditions. More critically, in actual patrols, the prediction of visual semantics is often uncertain due to localized lighting or mud, and traditional fixed-weight fusion or simple superposition strategies are ill-suited to handle sensor malfunctions or confidence fluctuations, easily leading to distorted accessibility assessments. Therefore, achieving a balance between accuracy and computational cost, and enabling adaptive fusion of multiple modalities based on dynamic changes in the perceived environment, are pressing technical challenges that need to be addressed. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method for perceiving unstructured environment accessibility by integrating visual semantics and radar geometric information, so as to solve the technical problem that the perception accuracy and robustness of using single sensor data to assess unstructured terrain accessibility in complex scenarios are insufficient.

[0008] The unstructured environment accessibility perception method of the present invention, which integrates visual semantics and radar geometric information, includes the following steps:

[0009] S1. Visual semantic segmentation and probability output: The image captured by the camera is input into the lightweight visual semantic segmentation network for processing to obtain the terrain category label of each pixel, forming a two-dimensional semantic segmentation mask, and outputting the semantic probability distribution of each pixel belonging to each terrain category;

[0010] S2. Point cloud preprocessing and multi-source data alignment: Voxel filtering and outlier removal are performed on the 3D point cloud data acquired by the LiDAR. Using the rotation matrix and translation vector obtained by the joint calibration of the camera and LiDAR, the preprocessed point cloud is spatially registered with the image, and spatiotemporal synchronization is achieved by combining the timestamp, so as to obtain the spatiotemporal synchronization and spatial registration results.

[0011] S3. Semantic Cost Layer Construction: Based on the spatiotemporal synchronization and spatial registration results, the two-dimensional semantic segmentation mask and semantic probability distribution obtained in S1 are projected onto a local three-dimensional grid map centered on the mobile robot to generate a three-dimensional semantic point cloud. For each grid in the local three-dimensional grid map, a semantic cost value is assigned according to its corresponding terrain material category. The semantic cost value is pre-set based on the surface friction coefficient and support stiffness characteristics of different terrain materials. The semantic cost values ​​of all grids constitute the semantic cost layer.

[0012] S4. Geometric Cost Layer Construction: Based on the local 3D grid map, the point cloud data preprocessed in S2 and the pose of the mobile robot are fused. A one-dimensional Kalman filter is used to dynamically update the elevation information of the local 3D grid map to construct a 3D elevation map. For each grid in the elevation map, principal component analysis is used to fit a plane to its local neighborhood point set. The normal vector of the fitted plane is calculated to obtain the terrain slope. The terrain flatness is extracted based on the fitting residual. The slope and flatness are normalized into a geometric cost value. The geometric cost values ​​of all grids constitute the geometric cost layer.

[0013] S5. Adaptive Fusion and Accessibility Graph Generation: For each grid corresponding to a position in the semantic cost layer and geometric cost layer, based on the semantic probability distribution projected onto the grid from the output of S1, calculate the normalized information entropy of each grid; construct an adaptive fusion factor based on information entropy; use the adaptive fusion factor to generate dynamic fusion weights for the semantic cost and geometric cost of each grid; use the dynamic fusion weights to perform a weighted summation of the semantic cost and geometric cost of each grid to obtain the final fused accessibility cost of each grid; traverse all grids to generate a continuous accessibility cost mapping graph of the unstructured environment.

[0014] Furthermore, the lightweight visual semantic segmentation network in step S1 includes a backbone network for preliminary feature extraction, a channel-pixel joint attention mechanism module, and a spatial-frequency domain joint multi-scale feature extraction module;

[0015] The channel-pixel joint attention mechanism module processes these two sets of features as follows:

[0016] a) Spatial branching characteristics of receiving shallow outputs from the backbone network Semantic branch features of deep output from the backbone network To conduct them separately Convolutional mapping and batch normalization yield spatial features. and semantic features Semantic features Upsampling to After reaching the same size, in each spatial location Calculate the dot product And through the Sigmoid function Obtain pixel-space attention weights :

[0017] (1)

[0018] (2)

[0019] in, The number of channels in the feature map. For spatial features in The vector at that location, For semantic features in The vector at that location;

[0020] b) Input features for spatial branching Extract channel statistics to obtain average channel features and global channel features :

[0021] (3)

[0022] (4)

[0023] The two are added together and then passed through the Sigmoid function. Obtain channel attention weights :

[0024] (5)

[0025] In the formula, and The height and width of the representative feature map;

[0026] c) Based on the calculation results of equations (2) and (5), multiply the pixel spatial attention weights and the channel attention weights to obtain the final dual attention weights. And based on this, spatial characteristics and semantic features Weighting is applied to achieve dual attention fusion of features from both branches:

[0027] (6)

[0028] (7)

[0029] In the formula, This represents the fused output feature map;

[0030] The spatial-frequency domain joint multi-scale feature extraction module processes the input feature map as follows:

[0031] Receive feature maps output from the channel-pixel joint attention mechanism The feature map is mapped to the frequency domain using discrete cosine transform and then masked for filtering to extract high-frequency components used to characterize surface texture details. With low-frequency components used to characterize the overall terrain structure The extracted two features are concatenated along the channel dimension to generate enhanced features. Then through The convolutional mapping to the feature space yields the frequency domain combined features. :

[0032] (8)

[0033] (9).

[0034] Furthermore, the formula for calculating the semantic probability distribution in step S1 is as follows:

[0035] (10)

[0036] In the formula, Let g be the semantic probability distribution of the grid. This represents the total number of semantic points within the grid g. For grid Belongs to the category The number of semantic points;

[0037] (11)

[0038] In the formula, This represents the total number of semantic categories.

[0039] Furthermore, in step S2, the voxel filtering of the 3D point cloud data acquired by the lidar includes:

[0040] Calculate the side length of the bounding box of 3D point cloud data. :

[0041] (12)

[0042] In the formula, and These represent the maximum and minimum values ​​of the point cloud on the three coordinate axes, respectively.

[0043] Subsequently, an octree structure is constructed to divide the point cloud into corresponding voxels. The centroid of all data points within each voxel is calculated and used to replace other points within that voxel to achieve point cloud homogenization and downsampling. The i-th voxel... The centroid of all data points within the area is calculated as follows:

[0044] (13)

[0045] In the formula, voxels The centroid of all data points within the area The three-dimensional spatial coordinates of the center voxels The j-th data point Three-dimensional spatial coordinates, For the j-th data point The center of gravity The number of data points contained in a voxel;

[0046] The outlier removal process includes: establishing a search radius with the sampling point as the center, counting the number of neighboring points within the radius, and removing the sampling point with the center when the number is lower than the set minimum neighbor threshold, thus completing the outlier filtering.

[0047] Step S2, which involves spatially registering the preprocessed point cloud with the image, includes:

[0048] Based on the rotation matrix obtained from joint calibration With translation vector Construct a homogeneous transformation relationship from the radar coordinate system to the camera coordinate system:

[0049] (14)

[0050] In the formula, Let these be the homogeneous coordinates of the point cloud in the radar coordinate system. This represents the homogeneous representation of the point cloud in the camera's physical coordinate system.

[0051] Establish the perspective transformation relationship between the image physical coordinate system and the image pixel coordinate system based on the camera's intrinsic parameters:

[0052] (15)

[0053] In the formula, This represents the depth value of the point cloud in the camera coordinate system. This represents the two-dimensional coordinates of the radar point cloud projected onto the camera pixel coordinate system. and These are the focal lengths of the camera along the x and y axes, respectively. , ) is the coordinate of the intersection of the camera's optical axis and the imaging plane in the pixel coordinate system; combining the derivation results of equations (5) and (6), the three-dimensional point cloud in the radar coordinate system is directly transformed and projected into the camera's pixel coordinate system to complete the spatial registration of heterogeneous sensors:

[0054] (16)

[0055] In the formula, This represents the two-dimensional coordinates of the radar point cloud projected onto the camera pixel coordinate system.

[0056] Furthermore, in step S3, constructing the semantic cost layer also includes: addressing the issue caused by the sparsity of the point cloud. For a hollow raster with a value of 0, search for all neighboring rasteres with terrain category labels within a spherical neighborhood centered at the raster's center coordinates and with a preset radius. This search forms a neighborhood raster set. If the neighborhood raster set is not empty, then weights are calculated based on spatial distance. :

[0057] (17)

[0058] In the formula, Hollow grid coordinates The coordinates of the grid with terrain category labels; This is the distance attenuation scale parameter, used to control the spatial propagation range of semantic information.

[0059] Furthermore, step S4, which uses a one-dimensional Kalman filter to dynamically update the elevation information of the local three-dimensional raster map, includes:

[0060] During the movement of the mobile robot, when new point cloud data is projected onto the same grid cell already recorded in the 3D elevation map, a one-dimensional Kalman filter is used to convert the current height measurement value. Elevation estimate of the same grid Perform a merge update to obtain the updated elevation estimate. Variance of elevation estimation :

[0061] (18)

[0062] (19)

[0063] In the formula, The elevation estimate before the update. The variance of the prediction, This represents the variance of the measurement.

[0064] Furthermore, the calculation of the terrain slope in step S4 is as follows:

[0065] A local window is constructed centered on the target raster, and the elevation values ​​of each raster within the window are extracted to form a local point set. For local point sets Obtain the center point by performing mean centering. :

[0066] (20)

[0067] In the formula, This represents the total number of samples participating in the fitting process within the local point set. Let X be the X-axis coordinates, Y-axis coordinates, and elevation value of the i-th sample point; Let be the average of all sample points in the point set along the x-axis, and let be the average of all sample points in the point set along the y-axis. This is the average value of all sample points in the point set on the elevation axis;

[0068] Construct the covariance matrix of the point set :

[0069] (twenty one)

[0070] For covariance matrix Perform eigenvalue decomposition to obtain eigenvalues ​​and their corresponding eigenvectors:

[0071] (twenty two)

[0072] In the formula, , and To obtain the eigenvalues, , and The corresponding feature vector;

[0073] The eigenvalues ​​are arranged in ascending order, and the eigenvector corresponding to the smallest eigenvalue is taken as the normal vector of the fitted local plane. ; Calculate the normal vector With respect to the Z-axis direction vector of the global coordinate system The angle between them is used to obtain the slope of the target grid. :

[0074] (twenty three)

[0075] The flatness mentioned in step S4 The calculation formula is as follows:

[0076] (twenty four)

[0077] In the formula, For grid The projected coordinates on the fitting plane are used; at the same time, the square root of the minimum eigenvalue is used as the equivalent feature operator to measure the terrain smoothness.

[0078] Furthermore, the expression for the geometric cost is as follows:

[0079] (25)

[0080] In the formula, For geometric cost, The maximum allowable slope threshold, This is the maximum permissible flatness threshold.

[0081] Furthermore, the formula for calculating the normalized information entropy is as follows:

[0082] (26)

[0083] In the formula, To normalize information entropy, Information entropy is calculated using the following formula:

[0084] (27)

[0085] In the formula, The total number of semantic segmentation categories. For the first The probability of each category occurring.

[0086] Furthermore, the formula for calculating the adaptive fusion factor is as follows:

[0087] (28)

[0088] In the formula, This represents the set semantic uncertainty threshold. Parameters used to control the smoothness of weight transitions;

[0089] The dynamic fusion weights of the semantic and geometric costs of the raster are as follows:

[0090] (29)

[0091] In the formula, For the dynamic fusion weights of semantic cost, The dynamic fusion weights represent geometric costs;

[0092] The final fusion access cost The calculation is as follows:

[0093] (30)

[0094] In the formula, For grid The expected cost of semantic output is calculated using the following formula:

[0095] (31)

[0096] In the formula, For the first Pre-defined value of each category.

[0097] Beneficial effects:

[0098] 1. This method deeply integrates the rich semantic understanding capabilities of visual cameras with the precise 3D geometric perception capabilities of LiDAR. The semantic cost layer provides terrain material attributes (such as grass, cement, and soil), enabling the system to distinguish terrains with similar geometric features but different physical accessibility attributes (e.g., flat grass versus flat cement roads). The geometric cost layer provides precise slope and flatness information, enabling the system to detect terrains with similar semantic features but physical obstacles (e.g., within a grassy area, whether there are hidden raised slopes or potholes). This multi-source information complementarity fundamentally overcomes the shortcomings of relying on a single sensor in complex, unstructured environments, such as incomplete perception and susceptibility to interference, achieving a more comprehensive and accurate assessment of accessibility.

[0099] 2. Traditional fixed-weight fusion strategies, when the semantic prediction reliability of visual sensors decreases due to interference such as sudden changes in illumination, shadows, water stains, and specular reflections, continuously output erroneous dominant information, leading to evaluation distortion. This method introduces normalized information entropy as a real-time metric for visual semantic prediction confidence. When information entropy is high (high prediction uncertainty), the system automatically reduces the weight of semantic cost and increases the weight of the more reliable LiDAR geometric cost through a sigmoid function; conversely, it increases the weight of semantic cost. This gives the fusion system a closed-loop adaptive capability of "perception-diagnosis-adjustment." When some sensor information is temporarily unreliable, the system can automatically rely on more reliable modalities, thus maintaining stable and reliable accessibility output in complex and ever-changing environments such as illumination changes and partial occlusion, greatly improving the overall robustness and environmental adaptability of the system.

[0100] 3. This solution fully considers the deployment constraints on real mobile robot platforms. The lightweight visual network used has been optimized in terms of model structure, significantly reducing computational complexity while ensuring high-precision semantic segmentation, enabling the entire system to run in real time on limited embedded computing power. Simultaneously, preprocessing of the LiDAR point cloud (voxel filtering, outlier removal) effectively reduces data volume and improves data quality, while explicit spatiotemporal synchronization and registration methods solve the practical challenges of multi-sensor data fusion. These design features collectively ensure that this technical solution is not merely a laboratory algorithm, but a complete solution with high engineering practicality and deployability. Attached Figure Description

[0101] Figure 1 This is a framework diagram for unstructured environment accessibility perception based on the fusion of visual semantics and radar geometric information.

[0102] Figure 2 It is a four-wheel differential drive-by-wire unmanned vehicle experimental platform;

[0103] Figure 3 It is an unstructured field testing scenario;

[0104] Figure 4 This is a visual diagram of semantic space mapping and semantic cost layer.

[0105] Figure 5 It is a passability determination result that integrates visual semantics and radar geometric information. Detailed Implementation

[0106] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0107] The unstructured environment accessibility perception method in this embodiment, which integrates visual semantics and radar geometric information, includes the following steps:

[0108] S1. Visual semantic segmentation and probability output: The image captured by the camera is input into the lightweight visual semantic segmentation network for processing to obtain the terrain category label of each pixel, forming a two-dimensional semantic segmentation mask, and outputting the semantic probability distribution of each pixel belonging to each terrain category.

[0109] In this embodiment, the lightweight visual semantic segmentation network (named TPSMNet) includes a backbone network for preliminary feature extraction, a channel-pixel joint attention mechanism (CPA-FM), and a spatial-frequency joint multi-scale feature extraction module (SFPM). The image is first subjected to preliminary feature extraction through a lightweight backbone network to obtain a series of multi-scale feature maps.

[0110] The channel-pixel joint attention mechanism aims to address the complementary advantages of features at different levels within the feature pyramid. Specifically, this module receives two sets of inputs: one set consists of feature maps from the shallow layers of the backbone network, which have high spatial resolution and retain rich details such as surface texture and edges; these feature maps from the shallow layers are referred to as spatial branch features. The other set consists of feature maps from the deeper layers of the backbone network, which have undergone multiple downsampling and have lower spatial resolution, but contain more abstract, higher-order semantic information; these feature maps from the deeper layers are referred to as semantic branch features. The channel-pixel joint attention mechanism module processes these two sets of features as follows:

[0111] a) Receiving spatial branch features from high resolution and low-resolution semantic branch features To conduct them separately Convolutional mapping and batch normalization yield spatial features. and semantic features Semantic features Upsampling to After reaching the same size, in each spatial location Calculate the dot product And through the Sigmoid function Obtain pixel-space attention weights :

[0112] (1)

[0113] (2)

[0114] in, The number of channels in the feature map. For spatial features in The vector at that location, For semantic features in The vector at that location.

[0115] b) Apply global average pooling and global max pooling to the spatial branch input features. Extract channel statistics to obtain average channel features and global channel features :

[0116] (3)

[0117] (4)

[0118] The two are added together and then passed through the Sigmoid function. Obtain channel attention weights :

[0119] (5)

[0120] In the formula, and The height and width of the feature map are represented.

[0121] c) Based on the calculation results of equations (2) and (5), multiply the pixel spatial attention weights and the channel attention weights to obtain the final dual attention weights. And based on this, spatial characteristics and semantic features Weighting is applied to achieve dual attention fusion of features from both branches:

[0122] (6)

[0123] (7)

[0124] In the formula, This represents the fused output feature map;

[0125] The Space-Frequency Joint Multi-Scale Feature Extraction Module (SFPM) processes the input feature map as follows:

[0126] Receive feature maps output from the channel-pixel joint attention mechanism The feature map is mapped to the frequency domain using discrete cosine transform and then masked for filtering to extract high-frequency components used to characterize surface texture details. With low-frequency components used to characterize the overall terrain structure The extracted two features are concatenated along the channel dimension to generate enhanced features. Then through The convolutional mapping to the feature space yields the frequency domain combined features. :

[0127] (8)

[0128] (9)

[0129] The classification and output layers (specifically, the Softmax function layer) of the lightweight visual semantic segmentation network combine frequency domain features. Perform the following processing:

[0130] Channel mapping: A 1×1 convolutional layer maps the number of channels in the feature map to the total number of preset terrain semantic categories C, generating C category scores for each pixel.

[0131] Probabilistic normalization: Apply the Softmax function to the C-dimensional category score vector at each pixel location to normalize it into a probability distribution. This C-dimensional probability vector is the semantic probability distribution of each pixel belonging to each terrain category.

[0132] Labeling: Perform the Argmax operation on the probability distribution vector of each pixel (take the category index corresponding to the maximum value) to obtain the terrain category label of that pixel.

[0133] Forming a mask: Arranging the category labels of all pixels according to their spatial positions in the original image constitutes a two-dimensional semantic segmentation mask.

[0134] S2. Point cloud preprocessing and multi-source data alignment: Voxel filtering and outlier removal are performed on the 3D point cloud data acquired by the LiDAR. Using the rotation matrix and translation vector obtained by the joint calibration of the camera and LiDAR, the preprocessed point cloud is spatially registered with the image, and spatiotemporal synchronization is achieved by combining the timestamp, so as to obtain the spatiotemporal synchronization and spatial registration results.

[0135] The voxel filtering of the 3D point cloud data acquired by the lidar includes:

[0136] Calculate the side length of the bounding box of 3D point cloud data. :

[0137] (10)

[0138] In the formula, and These represent the maximum and minimum values ​​of the point cloud on the three coordinate axes, respectively.

[0139] Subsequently, an octree structure is constructed to divide the point cloud into corresponding voxels. The centroid of all data points within each voxel is calculated and used to replace other points within that voxel to achieve point cloud homogenization and downsampling. The i-th voxel... The centroid of all data points within the area is calculated as follows:

[0140] (11)

[0141] In the formula, voxels The centroid of all data points within the area The three-dimensional spatial coordinates of the center voxels The j-th data point Three-dimensional spatial coordinates, For the j-th data point The center of gravity This represents the number of data points contained in a voxel.

[0142] The outlier removal process includes: establishing a search radius centered on the sampling point, counting the number of neighboring points within the radius, and removing the sampling point centered on the sampling point when the number is lower than the set minimum neighbor threshold, thus completing the outlier filtering.

[0143] The step of spatially registering the preprocessed point cloud with the image includes:

[0144] Based on the rotation matrix obtained from joint calibration With translation vector Construct a homogeneous transformation relationship from the radar coordinate system to the camera coordinate system:

[0145] (12)

[0146] In the formula, Let these be the homogeneous coordinates of the point cloud in the radar coordinate system. This represents the homogeneous representation of the point cloud in the camera's physical coordinate system.

[0147] Establish the perspective transformation relationship between the image physical coordinate system and the image pixel coordinate system based on the camera's intrinsic parameters:

[0148] (13)

[0149] In the formula, This represents the depth value of the point cloud in the camera coordinate system. This represents the two-dimensional coordinates of the radar point cloud projected onto the camera pixel coordinate system. and These are the focal lengths of the camera along the x and y axes, respectively. , ) is the coordinate of the intersection of the camera's optical axis and the imaging plane in the pixel coordinate system; combining the derivation results of equations (5) and (6), the three-dimensional point cloud in the radar coordinate system is directly transformed and projected into the camera's pixel coordinate system to complete the spatial registration of heterogeneous sensors:

[0150] (14)

[0151] In the formula, This represents the two-dimensional coordinates of the radar point cloud projected onto the camera pixel coordinate system.

[0152] S3. Semantic Cost Layer Construction: Based on the spatiotemporal synchronization and spatial registration results, the two-dimensional semantic segmentation mask and semantic probability distribution obtained in S1 are projected onto a local three-dimensional grid map centered on the mobile robot to generate a three-dimensional semantic point cloud. For each grid in the local three-dimensional grid map, a semantic cost value is assigned according to its corresponding terrain material category. The semantic cost value is pre-set based on the surface friction coefficient and support stiffness characteristics of different terrain materials. The semantic cost values ​​of all grids constitute the semantic cost layer.

[0153] The construction of the semantic cost layer also includes: addressing the issue caused by point cloud sparsity. For a hollow raster with a value of 0, search for all neighboring rasteres with terrain category labels within a spherical neighborhood centered at the raster's center coordinates and with a preset radius. This search forms a neighborhood raster set. If the neighborhood raster set is not empty, then weights are calculated based on spatial distance. :

[0154] (15)

[0155] In the formula, Hollow grid coordinates The coordinates of the grid with terrain category labels; This is the distance attenuation scale parameter, used to control the spatial propagation range of semantic information.

[0156] S4. Geometric Cost Layer Construction: Based on the local 3D grid map, the point cloud data preprocessed in S2 and the pose of the mobile robot are fused. A one-dimensional Kalman filter is used to dynamically update the elevation information of the local 3D grid map to construct a 3D elevation map. For each grid in the elevation map, principal component analysis is used to fit a plane to its local neighborhood point set. The normal vector of the fitted plane is calculated to obtain the terrain slope, and the terrain flatness is extracted based on the fitting residual. The slope and flatness are normalized into a geometric cost value. The geometric cost values ​​of all grids constitute the geometric cost layer.

[0157] The method of dynamically updating the elevation information of a local three-dimensional raster map using one-dimensional Kalman filtering includes:

[0158] During the movement of the mobile robot, when new point cloud data is projected onto the same grid cell already recorded in the 3D elevation map, a one-dimensional Kalman filter is used to convert the current height measurement value. Elevation estimate of the same grid Perform a merge update to obtain the updated elevation estimate. Variance of elevation estimation :

[0159] (16)

[0160] (17)

[0161] In the formula, The elevation estimate before the update. The variance of the prediction, This represents the variance of the measurement.

[0162] The slope of the terrain is calculated as follows:

[0163] A local window is constructed centered on the target raster, and the elevation values ​​of each raster within the window are extracted to form a local point set. For local point sets Obtain the center point by performing mean centering. :

[0164] (18)

[0165] In the formula, This represents the total number of samples participating in the fitting process within the local point set. Let X be the X-axis coordinates, Y-axis coordinates, and elevation value of the i-th sample point; Let be the average of all sample points in the point set along the x-axis, and let be the average of all sample points in the point set along the y-axis. This is the average value of all sample points in the point set on the elevation axis;

[0166] Construct the covariance matrix of the point set :

[0167] (19)

[0168] For covariance matrix Perform eigenvalue decomposition to obtain eigenvalues ​​and their corresponding eigenvectors:

[0169] (20)

[0170] In the formula, , and To obtain the eigenvalues, , and The corresponding eigenvectors are used; the eigenvalues ​​are arranged in ascending order, and the eigenvector corresponding to the smallest eigenvalue is taken as the normal vector of the fitted local plane. ; Calculate the normal vector With respect to the Z-axis direction vector of the global coordinate system The angle between them is used to obtain the slope of the target grid. :

[0171] (twenty one)

[0172] The flatness The calculation formula is as follows:

[0173] (twenty two)

[0174] In the formula, For grid The projected coordinates on the fitting plane are used; at the same time, the square root of the minimum eigenvalue is used as the equivalent feature operator to measure the terrain smoothness.

[0175] The physical slope and flatness are combined through a linear mapping mechanism to obtain a normalized comprehensive geometric passability score. A geometric cost layer is constructed. The expression for the geometric cost is as follows:

[0176] (twenty three)

[0177] In the formula, For geometric cost, The maximum allowable slope threshold, This is the maximum permissible flatness threshold.

[0178] S5. Adaptive Fusion and Accessibility Graph Generation: For each grid corresponding to a position in the semantic cost layer and geometric cost layer, based on the semantic probability distribution projected onto the grid from the output of S1, calculate the normalized information entropy of each grid; construct an adaptive fusion factor based on information entropy; use the adaptive fusion factor to generate dynamic fusion weights for the semantic cost and geometric cost of each grid; use the dynamic fusion weights to perform a weighted summation of the semantic cost and geometric cost of each grid to obtain the final fused accessibility cost of each grid; traverse all grids to generate a continuous accessibility cost mapping graph of the unstructured environment.

[0179] The formula for calculating the normalized information entropy is as follows:

[0180] (twenty four)

[0181] In the formula, To normalize information entropy,

[0182] Information entropy is calculated using the following formula:

[0183] (25)

[0184] In the formula, The total number of semantic segmentation categories. For the pixel in the g-th grid to belong to the th Normalized probability values ​​for each terrain category.

[0185] The formula for calculating the semantic probability distribution in step S1 is as follows:

[0186] (26)

[0187] In the formula, Let g be the semantic probability distribution of the grid. This represents the total number of semantic points within the grid g. For grid Belongs to the category The number of semantic points;

[0188] (27)

[0189] In the formula, This represents the total number of semantic categories.

[0190] The semantic probability distribution described in step S1 is a pixel-level category probability matrix. After projecting it onto a raster map, the semantic probability distribution of raster g is obtained by statistically analyzing the category attribution of each pixel within the raster (as shown in formulas 26 and 27). This distribution is represented as a column vector with C dimensions, representing the probability set of all possible terrain categories to which raster g belongs. To calculate the information entropy (formula 27), specific probability values ​​need to be extracted from the semantic probability distribution. Specifically, That is, the semantic probability distribution at the th The value taken in the nth dimension, that is, it is the raster g belonging to the nth dimension. Normalized probability values ​​for each terrain category.

[0191] The formula for calculating the adaptive fusion factor is as follows:

[0192] (28)

[0193] In the formula, This represents the set semantic uncertainty threshold. Parameters used to control the smoothness of weight transitions;

[0194] The dynamic fusion weights of the semantic and geometric costs of the raster are as follows:

[0195] (29)

[0196] In the formula, For the dynamic fusion weights of semantic cost, The dynamic fusion weights represent geometric costs;

[0197] The final fusion access cost The calculation is as follows:

[0198] (30)

[0199] In the formula, For grid The expected cost of semantic output is calculated using the following formula:

[0200] (31)

[0201] In the formula, For the first Pre-defined value of each category.

[0202] The following experiment compares the unstructured environment accessibility perception method that integrates visual semantics and radar geometry information with single visual semantics methods and single radar geometry methods:

[0203] Accessibility perception assessment experiments in different scenarios. A four-wheel differential drive-by-wire unmanned vehicle was selected as the experimental platform (e.g., Figure 2 (As shown). This intelligent platform is equipped with a high-precision integrated navigation and positioning system, solid-state LiDAR, visible light camera, and industrial control computer. It can accurately measure its own pose changes and simultaneously record the raw sensor data of images and radar point clouds. Based on this unmanned vehicle experimental platform, data was collected in two representative unstructured outdoor environments. Scenario A includes a flat cement road and grass. In this scenario, the terrain is geometrically at the same plane, and the radar geometric features are highly consistent. In Scenario B, a large area of ​​grass contains hidden raised slopes. In this scenario, the visual semantic attributes are highly consistent, but there are actual geometric obstacles to pass through. To obtain high-quality reference data, manually labeled local grid accessibility scores were used as ground truth reference data.

[0204] Before performing drivability fusion and discrimination, the accuracy of key intermediate steps was first verified. In the elevation map accuracy evaluation, the elevation map constructed based on LiDAR point cloud and pose update showed that the errors between the measured dimensions and the actual physical dimensions were all less than 0.06m in experiments with convex obstacles and stair terrain. The system has high-precision geometric reconstruction capabilities in planar structure recognition and multi-level height change representation. In visual semantic extraction, the lightweight visual semantic segmentation network (TPSMNet) effectively overcomes the problem of similar terrain textures in unstructured scenes while ensuring real-time inference, and obtains high-precision two-dimensional semantic masks.

[0205] For the acquired outdoor scene data, a single visual semantic method, a single radar geometric method, and the method of this invention were used to estimate the accessibility cost. Mean absolute error (MAE) and mean squared error (MSE) were used as evaluation metrics, and the evaluation results of different methods in different scenarios are shown in Table 1.

[0206]

[0207] The quantitative experimental results and visualization mapping performance in Table 1 show that the method of this invention outperforms methods using a single data source in terms of accessibility assessment accuracy. In scenario A, relying solely on LiDAR assigns almost the same cost to concrete roads and grass at the same geometric height. However, this method introduces semantic segmentation results from the camera perspective for adaptive weighted compensation, successfully identifying the physical material of the surface in flat areas and assigning lower access costs to concrete roads. In scenario B, pure visual semantics identifies the entire area as grass, ignoring the raised slopes hidden within the grass. However, this method accurately captures anomalies in surface slope and flatness through a geometric cost layer, successfully marking the slope locations as high-cost areas in the fused accessibility mapping map. Figure 3Two outdoor scenarios were shown for the accessibility perception experiment.

[0208] like Figure 4 As shown, where Figure 4 (a) shows the spatial mapping effect of the three-dimensional semantic point cloud generated after projecting two-dimensional semantic labels onto a local raster map, with different colors representing different terrain semantic categories; Figure 4 (b) is based on the above semantic classification results combined with the friction coefficient of terrain material and support stiffness characteristics.

[0209] The visualization results of the feasibility determination of the method of the present invention are as follows: Figure 5 As shown. Figure 5 (a) shows the judgment result of scenario A. It can be seen that in the environment where flat cement road and grass coexist, this method effectively distinguishes two types of ground with the same geometric height but different materials through the semantic compensation mechanism. The cement road area is given a lower passage cost, while the grass area is given a relatively higher passage cost. Figure 5 (b) shows the judgment result for scenario B. It can be observed that the method accurately identifies the location of the protruding earthen slope obstacle hidden in a large area of ​​grassland, and marks the area as high cost in the fused accessibility cost map, which verifies the effective perception capability of the geometric cost layer for obstacles in three-dimensional space.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for perceiving unstructured environmental accessibility by integrating visual semantics and radar geometric information, characterized in that: Includes the following steps: S1. Visual semantic segmentation and probability output: The image captured by the camera is input into the lightweight visual semantic segmentation network for processing to obtain the terrain category label of each pixel, forming a two-dimensional semantic segmentation mask, and outputting the semantic probability distribution of each pixel belonging to each terrain category; S2. Point cloud preprocessing and multi-source data alignment: Voxel filtering and outlier removal are performed on the 3D point cloud data acquired by the LiDAR. Using the rotation matrix and translation vector obtained by the joint calibration of the camera and LiDAR, the preprocessed point cloud is spatially registered with the image, and spatiotemporal synchronization is achieved by combining the timestamp, so as to obtain the spatiotemporal synchronization and spatial registration results. S3. Semantic Cost Layer Construction: Based on the spatiotemporal synchronization and spatial registration results, the two-dimensional semantic segmentation mask and semantic probability distribution obtained in S1 are projected onto a local three-dimensional grid map centered on the mobile robot to generate a three-dimensional semantic point cloud. For each grid in the local three-dimensional grid map, a semantic cost value is assigned according to its corresponding terrain material category. The semantic cost value is pre-set based on the surface friction coefficient and support stiffness characteristics of different terrain materials. The semantic cost values ​​of all grids constitute the semantic cost layer. S4. Geometric Cost Layer Construction: Based on the local 3D grid map, the point cloud data preprocessed in S2 and the pose of the mobile robot are fused. A one-dimensional Kalman filter is used to dynamically update the elevation information of the local 3D grid map to construct a 3D elevation map. For each grid in the elevation map, principal component analysis is used to fit a plane to its local neighborhood point set. The normal vector of the fitted plane is calculated to obtain the terrain slope. The terrain flatness is extracted based on the fitting residual. The slope and flatness are normalized into a geometric cost value. The geometric cost values ​​of all grids constitute the geometric cost layer. S5. Adaptive Fusion and Accessibility Graph Generation: For each grid corresponding to a position in the semantic cost layer and geometric cost layer, based on the semantic probability distribution projected onto the grid from the output of S1, calculate the normalized information entropy of each grid; construct an adaptive fusion factor based on information entropy; use the adaptive fusion factor to generate dynamic fusion weights for the semantic cost and geometric cost of each grid; use the dynamic fusion weights to perform a weighted summation of the semantic cost and geometric cost of each grid to obtain the final fused accessibility cost of each grid; traverse all grids to generate a continuous accessibility cost mapping graph of the unstructured environment.

2. The unstructured environment accessibility perception method according to claim 1, which integrates visual semantics and radar geometric information, is characterized in that: The lightweight visual semantic segmentation network described in step S1 includes a backbone network for preliminary feature extraction, a channel-pixel joint attention mechanism module, and a spatial-frequency domain joint multi-scale feature extraction module. The channel-pixel joint attention mechanism module processes these two sets of features as follows: a) Spatial branching characteristics of receiving shallow outputs from the backbone network Semantic branch features of deep output from the backbone network To conduct them separately Convolutional mapping and batch normalization yield spatial features. and semantic features Semantic features Upsampling to After reaching the same size, in each spatial location Calculate the dot product And through the Sigmoid function Obtain pixel-space attention weights : （1） （2） in, The number of channels in the feature map. For spatial features in The vector at that location, For semantic features in The vector at that location; b) Input features for spatial branching Extract channel statistics to obtain average channel features and global channel features : （3） （4） The two are added together and then passed through the Sigmoid function. Obtain channel attention weights : （5） In the formula, and The height and width of the representative feature map; c) Based on the calculation results of equations (2) and (5), multiply the pixel spatial attention weights and the channel attention weights to obtain the final dual attention weights. And based on this, spatial characteristics and semantic features Weighting is applied to achieve dual attention fusion of features from both branches: （6） （7） In the formula, This represents the fused output feature map; The spatial-frequency domain joint multi-scale feature extraction module processes the input feature map as follows: Receive feature maps output from the channel-pixel joint attention mechanism The feature map is mapped to the frequency domain using discrete cosine transform and then masked for filtering to extract high-frequency components used to characterize surface texture details. With low-frequency components used to characterize the overall terrain structure The extracted two features are concatenated along the channel dimension to generate enhanced features. Then through The convolutional mapping to the feature space yields the frequency domain combined features. : （8） （9）。 3. The unstructured environment accessibility perception method according to claim 2, which integrates visual semantics and radar geometric information, is characterized in that: The formula for calculating the semantic probability distribution in step S1 is as follows: （10） In the formula, Let g be the semantic probability distribution of the grid. This represents the total number of semantic points within the grid g. For grid Belongs to the category The number of semantic points; （11） In the formula, This represents the total number of semantic categories.

4. The unstructured environment accessibility perception method according to claim 1, which integrates visual semantics and radar geometric information, is characterized in that: In step S2, the voxel filtering of the 3D point cloud data acquired by the lidar includes: Calculate the side length of the bounding box of 3D point cloud data. : （12） In the formula, and These represent the maximum and minimum values ​​of the point cloud on the three coordinate axes, respectively. Subsequently, an octree structure is constructed to divide the point cloud into corresponding voxels. The centroid of all data points within each voxel is calculated and used to replace other points within that voxel to achieve point cloud homogenization and downsampling. The i-th voxel... The centroid of all data points within the area is calculated as follows: （13） In the formula, voxels The centroid of all data points within the area The three-dimensional spatial coordinates of the center voxels The j-th data point Three-dimensional spatial coordinates, For the j-th data point The center of gravity The number of data points contained in a voxel; The outlier removal process includes: establishing a search radius with the sampling point as the center, counting the number of neighboring points within the radius, and removing the sampling point with the center when the number is lower than the set minimum neighbor threshold, thus completing the outlier filtering. Step S2, which involves spatially registering the preprocessed point cloud with the image, includes: Based on the rotation matrix obtained from joint calibration With translation vector Construct a homogeneous transformation relationship from the radar coordinate system to the camera coordinate system: （14） In the formula, Let these be the homogeneous coordinates of the point cloud in the radar coordinate system. This represents the homogeneous representation of the point cloud in the camera's physical coordinate system. Establish the perspective transformation relationship between the image physical coordinate system and the image pixel coordinate system based on the camera's intrinsic parameters: （15） In the formula, This represents the depth value of the point cloud in the camera coordinate system. This represents the two-dimensional coordinates of the radar point cloud projected onto the camera pixel coordinate system. and These are the focal lengths of the camera along the x and y axes, respectively. , ) is the coordinate of the intersection of the camera's optical axis and the imaging plane in the pixel coordinate system; combining the derivation results of equations (5) and (6), the three-dimensional point cloud in the radar coordinate system is directly transformed and projected into the camera's pixel coordinate system to complete the spatial registration of heterogeneous sensors: （16） In the formula, This represents the two-dimensional coordinates of the radar point cloud projected onto the camera pixel coordinate system.

5. The unstructured environment accessibility perception method according to claim 1, which integrates visual semantics and radar geometric information, is characterized in that: In step S3, constructing the semantic cost layer further includes: addressing the issue caused by the sparsity of the point cloud. For a hollow raster with a value of 0, search for all neighboring rasteres with terrain category labels within a spherical neighborhood centered at the raster's center coordinates and with a preset radius. This search forms a neighborhood raster set. If the neighborhood raster set is not empty, then weights are calculated based on spatial distance. : （17） In the formula, Hollow grid coordinates The coordinates of the grid with terrain category labels; This is the distance attenuation scale parameter, used to control the spatial propagation range of semantic information.

6. The unstructured environment accessibility perception method according to claim 1, which integrates visual semantics and radar geometric information, is characterized in that: Step S4 involves dynamically updating the elevation information of the local 3D raster map using a one-dimensional Kalman filter, including: During the movement of the mobile robot, when new point cloud data is projected onto the same grid cell already recorded in the 3D elevation map, a one-dimensional Kalman filter is used to convert the current height measurement value. Elevation estimate of the same grid Perform a merge update to obtain the updated elevation estimate. Variance of elevation estimation : （18） （19） In the formula, The elevation estimate before the update. The variance of the prediction, This represents the variance of the measurement.

7. The unstructured environment accessibility perception method according to claim 5, which integrates visual semantics and radar geometric information, is characterized in that: The calculation of the terrain slope in step S4 is as follows: A local window is constructed centered on the target raster, and the elevation values ​​of each raster within the window are extracted to form a local point set. For local point sets Obtain the center point by performing mean centering. : （20） In the formula, This represents the total number of samples participating in the fitting process within the local point set. Let X be the X-axis coordinates, Y-axis coordinates, and elevation value of the i-th sample point; Let be the average of all sample points in the point set along the x-axis, and let be the average of all sample points in the point set along the y-axis. This is the average value of all sample points in the point set on the elevation axis; Construct the covariance matrix of the point set : （21） For covariance matrix Perform eigenvalue decomposition to obtain eigenvalues ​​and their corresponding eigenvectors: （22） In the formula, , and To obtain the eigenvalues, , and The corresponding eigenvectors are used; the eigenvalues ​​are arranged in ascending order, and the eigenvector corresponding to the smallest eigenvalue is taken as the normal vector of the fitted local plane. ; Calculate the normal vector With respect to the Z-axis direction vector of the global coordinate system The angle between them is used to obtain the slope of the target grid. : （23） The flatness mentioned in step S4 The calculation formula is as follows: （24） In the formula, For grid The projected coordinates on the fitting plane are used; at the same time, the square root of the minimum eigenvalue is used as the equivalent feature operator to measure the terrain smoothness.

8. The unstructured environment accessibility perception method according to claim 7, which integrates visual semantics and radar geometric information, is characterized in that: The expression for the geometric cost is as follows: （25） In the formula, For geometric cost, The maximum allowable slope threshold, This is the maximum permissible flatness threshold.

9. The unstructured environment accessibility perception method according to claim 9, which integrates visual semantics and radar geometric information, is characterized in that: The formula for calculating the normalized information entropy is as follows: （26） In the formula, To normalize information entropy, Information entropy is calculated using the following formula: （27） In the formula, The total number of semantic segmentation categories. For the first The probability of each category occurring.

10. The unstructured environment accessibility perception method according to claim 9, which integrates visual semantics and radar geometric information, is characterized in that: The formula for calculating the adaptive fusion factor is as follows: （28） In the formula, This represents the set semantic uncertainty threshold. Parameters used to control the smoothness of weight transitions; The dynamic fusion weights of the semantic and geometric costs of the raster are as follows: （29） In the formula, For the dynamic fusion weights of semantic cost, The dynamic fusion weights represent geometric costs; The final fusion access cost The calculation is as follows: （30） In the formula, For grid The expected cost of semantic output is calculated using the following formula: （31） In the formula, For the first Pre-defined value of each category.

Images