A method for constructing a global semantic elevation relief map in a complex environment

Abstract

The application discloses a kind of global semantic elevation geomorphologic map construction methods under complex environment, it uses semantic segmentation network to carry out more detailed division to complex landform in the mobile environment of outdoor robot, and through image key frame screening, point cloud semantic mapping and multi-sensing fusion etc., create the elevation geomorphologic map containing rich environmental information, to be used to improve the navigation efficiency of outdoor robot.Meanwhile, the application can detailedly characterize the height information of landform ups and downs change and surrounding environmental object, make up the deficiency of two-dimensional grid map construction outdoor environment model, provide more detailed environmental information for the navigation of outdoor robot, improve the accuracy and rationality of robot navigation;In addition, semantic elevation map is a kind of grid map, can reduce map storage capacity, greatly improve the operation efficiency of robot processor, also can reduce the performance requirement of field robot chip.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot perception and mapping, and particularly relates to a global semantic elevation topographic map construction method in a complex environment. BACKGROUND

[0002] With the continuous development of robot technology, outdoor robots are gradually applied to disaster rescue transportation, mine transportation and other unstructured outdoor environments, which contain complex and diverse topographic environments, such as soil roads, grasslands, brick and stone ground and the like. The safety, travel time and energy consumption of the robot are greatly different when passing through different topographic environments. Therefore, the robot needs to simulate human thinking, perceive the surrounding topographic environment, and plan the most reasonable travel route according to different task requirements. For example, when the robot works in a disaster area to perform a rescue task, the robot needs to pass through high-cost topography in a straight line to shorten the travel time and achieve rapid rescue; when the robot performs mine transportation, the robot needs to select easy-to-travel topography to reduce its own loss, so as to reduce the maintenance cost of the robot. For the above-mentioned unstructured outdoor topographic environment, the construction of a topographic map containing rich topographic information and travel cost can help the robot to maximize the navigation efficiency on the premise of ensuring its safety.

[0003] At present, the robot perception technology is relatively mature, and for the outdoor diversified environment, many existing perception algorithms use three-dimensional point cloud maps to construct an environment model, but the three-dimensional point cloud map cannot be planned in real time. As for the two-dimensional occupancy grid map representation method, it lacks topographic height information and undulation information, and cannot fully meet the perception needs of the robot. The outdoor robot has certain off-road capability, and can pass through the road surface with certain slope undulation and height difference. Therefore, the construction of an outdoor environment model capable of detailed representation of topographic undulation and height change is helpful for the outdoor robot to make more reasonable planning and further release the potential of the robot in outdoor work. SUMMARY

[0004] The purpose of the present application is to provide a global semantic elevation topographic map construction method in a complex environment.

[0005] The present application provides a global semantic elevation topographic map construction method in a complex environment, which comprises the following steps:

[0006] Step one, joint calibration of the external parameters of the multi-line laser radar and the depth camera to obtain the camera intrinsic parameter matrix R and the depth camera radar external parameter matrix RT;

[0007] Step two, obtaining the real-time pose information of the robot, and obtaining the laser point cloud according to the laser radar;

[0008] Step three, obtaining and training a real-time topographic segmentation network model;

[0009] Step four, obtaining the landform segmentation semantic map of the local environment according to the trained real-time landform segmentation network model;

[0010] Step five, time synchronizing the laser point cloud and the landform segmentation semantic map, and projecting and mapping the point cloud to the landform segmentation semantic map by using the intrinsic parameter matrix R and the extrinsic parameter matrix RT to obtain the semantic point cloud corresponding to each frame of landform image;

[0011] Step six, constructing a color layer grid map by using the semantic information of the semantic point cloud, constructing an elevation layer grid map by using the z coordinate of the semantic point cloud, fusing the color layer grid map and the elevation layer grid map to form a local semantic elevation landform map, and constructing a global semantic elevation landform map according to the real-time pose of the robot and the coordinate transformation relationship.

[0012] As preferred, in step three, the real-time landform segmentation network model comprises a real-time mobile network coding module, a multi-feature fusion module and a network decoding module; the real-time mobile network coding module comprises a convolution network layer, an inverse residual network module and a self-attention learning mechanism module connected in sequence;

[0013] The self-attention learning mechanism module comprises a first self-attention learning mechanism layer and a second self-attention learning mechanism layer connected in sequence; the first self-attention learning mechanism layer comprises an inverse residual network layer and two self-attention layers connected in sequence; the second self-attention learning mechanism layer comprises an inverse residual network layer and four self-attention layers connected in sequence; all the self-attention layers have the same structure and comprise a local feature extraction module and a global feature extraction module connected in sequence; the local feature extraction module comprises a depth separable convolution and a normal convolution; the global feature extraction module comprises a linear multi-head self-attention mechanism and a convolution network; the output of the self-attention layer is the feature fused from the input and output of the global feature extraction module;

[0014] The multi-feature fusion module is used for fusing the features output by the inverse residual network module, the first self-attention learning mechanism layer and the second self-attention learning mechanism layer; the network decoding module is used for obtaining the landform segmentation semantic map of the local environment according to the image features output by the multi-feature fusion module.

[0015] As preferred, in step three, the inverse residual network module comprises three inverse residual network layers connected in sequence; the three inverse residual network layers have the same structure and each comprises a first convolution layer, a second convolution layer, an axial attention mechanism and a third convolution layer connected in sequence; the output feature of the third convolution layer is fused with the feature of the corresponding inverse residual network layer to form the output of the inverse residual network layer;

[0016] The multi-feature fusion module comprises a context information fusion module and a multi-scale information fusion module; the context information fusion module comprises a first branch, a second branch and a channel shuffle module; the first branch comprises a convolution block, a maximum pooling layer and an average pooling layer, and is used for feature extraction on the output of the inverse residual network module; the convolution block comprises a convolution layer, a batch normalization layer and an activation function connected in sequence; the maximum pooling layer and the average pooling layer perform pooling operations on the output of the convolution layer respectively; the second branch comprises a convolution block and an up-sampling layer connected in sequence, and is used for feature extraction on the output of the first self-attention learning mechanism layer; the outputs of the maximum pooling layer and the average pooling layer in the first branch are added, spliced with the output of the second branch, and input to the channel shuffle module for channel shuffle operation, so as to obtain more comprehensive image features; the multi-scale information fusion module adopts a spatial pyramid pooling layer to perform a pooling operation on the image features output by the self-attention learning mechanism module; the output of the context information fusion module after down-sampling operation and the output of the multi-scale information fusion module are added pixel by pixel to obtain the output of the multi-feature fusion module.

[0017] The network decoding module comprises two convolution up-sampling layers and one ordinary up-sampling layer connected in sequence; the two convolution up-sampling layers in series receive the image features output by the multi-feature fusion module, gradually restore the channel number and size of the image through convolution and up-sampling operations; and the output is a topographic segmentation semantic map of a local environment.

[0018] As preferred, the specific process of step five is as follows:

[0019] 5-1. Create a point cloud message queue and a semantic segmentation map message queue, limit the time difference to a certain range by intercepting the time stamps of the two message queues, and thereby synchronize the reception of the two topics;

[0020] 5-2. Convert the points in the laser point cloud to homogeneous coordinates and store them in the 4D variable ;

[0021] 5-3. Map the variable to the topographic semantic segmentation image through the intrinsic matrix R and the extrinsic matrix RT and store the result in the variable ;

[0022] 5-4. Convert the variable back to Euclidean coordinates, and store the image coordinates to the current image frame topographic pixel ;

[0023] 5-5. Transmit the color of the current image frame topographic pixel to the semantic point cloud ;

[0024] 5-6. Current semantic point cloud is filtered to obtain a filtered semantic point cloud . .

[0025] As preferred, in step four, the method for obtaining the topographic segmentation semantic map is as follows:

[0026] The image frame at the initial time is taken as the latest key image frame; a latest image key frame screening condition is constructed to screen each image frame obtained at each time; the latest key image frame closest to the current time is obtained, and the latest key image frame is input into the real-time topographic segmentation network model to obtain the topographic segmentation semantic map.

[0027] As preferred, the latest image key frame screening condition comprises an image feature matching condition , an odometry condition and a time constraint condition ;

[0028] The expression of the image feature matching condition is as follows:

[0029]

[0030] wherein, is the matching score between the current image frame and the latest key image frame; is the score threshold value;

[0031] The expression of the odometry condition is as follows:

[0032]

[0033] wherein, is the position difference value of the robot between the current image frame and the latest key image frame; is the difference value of the yaw angle of the robot between the current image frame and the latest key image frame; is the maximum distance threshold value; is the maximum angle threshold value;

[0034] The expression of the time constraint condition is as follows:

[0035]

[0036] wherein, is the time difference between the current image frame and the latest key image frame; is the time threshold value.

[0037] As preferred, the score threshold value Set to 0.2; maximum distance threshold and the maximum angle threshold Set to 2 meters and 5 degrees respectively; time threshold Set to 3 seconds.

[0038] Preferably, the method for acquiring the real-time posture information of the robot is: integrating the sensor data of the robot's lidar, GPS and inertial measurement unit to construct a laser odometer to acquire the real-time posture information of the robot.

[0039] As a preference, the specific process of step six is ​​as follows:

[0040] 6-1. Convert the point cloud coordinates into two-dimensional grid cells, convert the corresponding color information into floating-point values ​​as the values ​​of the color layer cells, and record the color information of the grid;

[0041] When the grid contains multiple point clouds, the Kalman filter is used to update the grid state. The specific operations are as follows:

[0042]

[0043] in, is the Kalman gain coefficient at the current time t; is the variance of the prediction error; is the noise variance; is the updated color information; is the estimated value of the predicted state; is the color information before updating; is the updated error covariance;

[0044] 6-2. Record the point cloud in each unit grid in the elevation grid map Coordinates, cluster the z coordinates according to the k-median algorithm, and use the clustered point cloud z coordinates to build an elevation layer grid map; when the same unit grid contains multiple point cloud z coordinates, use the Kalman filter to update the height. The specific operations are as follows:

[0045]

[0046] in, is the updated height information; is the estimated value of the predicted state; This is the height information before updating.

[0047] As a preference, the specific process of step three is as follows:

[0048] a. Use a robot to carry a depth camera to collect a variety of topographic images in different environments to construct a data set, and label the data set with the topographic type;

[0049] b. Construct a real-time topographic segmentation network model;

[0050] c. Use the data set to train the topographic segmentation network model to obtain the trained real-time topographic segmentation network model.

[0051] The present application has the beneficial effects that:

[0052] 1. The present application fuses three-dimensional laser point cloud and image information of a depth camera, increases semantic information of three-dimensional point cloud, constructs a semantic elevation map capable of representing changes in each topographic relief and height information, can represent changes in topographic relief and height information of surrounding environmental objects in detail, makes up for the deficiency of two-dimensional grid map in constructing an outdoor environment model, provides more detailed environmental information for outdoor robot navigation, improves the accuracy and rationality of robot navigation; at the same time, the semantic elevation map is a grid map, which can reduce the map storage amount, greatly improve the operation efficiency of the robot processor, and also reduce the performance requirements of the field robot chip.

[0053] 2. The present application introduces a self-attention learning mechanism module, and fuses the multiple outputs of the self-attention learning mechanism module through a multi-feature fusion module, improves the topographic segmentation speed and accuracy of the network, and is suitable for robots carrying edge devices; at the same time, the present application uses a semantic segmentation network to more carefully divide the complex topography in the moving environment of the outdoor robot.

[0054] 3. The present application uses high-frequency IMU information to remove motion distortion of the acquired laser point cloud, ensures the stability of the laser point cloud, and constructs a laser odometry; at the same time, the image data obtained is selected for key frames, and the calculation load brought by a large number of repeated image frames is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The present application is a global semantic elevation topographic map construction flowchart.

[0056] Figure 2 The present application is a real-time topographic semantic segmentation model schematic diagram.

[0057] Figure 3 The present application is a multi-feature fusion module schematic diagram. DETAILED DESCRIPTION

[0058] The present application will be further described below in conjunction with the drawings.

[0059] As Figure 1As shown, a method for constructing a global semantic elevation landform map in a complex environment includes the following steps:

[0060] Step 1: Use the laser vision joint calibration tool Calibration_toolkit to perform external parameter joint calibration on the multi-line laser radar and depth camera carried by the robot to obtain the camera intrinsic parameter matrix R and the depth camera radar extrinsic parameter matrix RT, so as to facilitate the subsequent fusion of 3D laser point cloud and depth camera image information.

[0061] Step 2: Use the LIO-SAM algorithm to fuse the robot's lidar, GPS, and IMU (inertial measurement unit) sensor data to build a laser odometry, obtain the robot's real-time position information, and provide the necessary coordinate transformation relationship for the construction of a semantic elevation map; obtain the laser point cloud through a multi-line lidar, and perform dedistortion, filtering, and other processing operations on the laser point cloud.

[0062] Step 3: Obtain real-time terrain segmentation network model

[0063] 3-1. Use a robot equipped with a depth camera to collect images of various landforms in different environments. After manual data annotation, a labeled landform dataset is constructed. The dataset contains a variety of landform types, including cement roads, bumpy roads, grass, dirt roads, and woodlands.

[0064] 3-2. If Figure 2 As shown, a real-time terrain segmentation network model is constructed; the real-time terrain segmentation network model includes a real-time mobile network coding module, a multi-feature fusion module, and a network decoding module.

[0065] 3-2-1. The real-time mobile network coding module includes a sequentially connected convolutional network layer, an inverse residual network module, and a self-attention learning mechanism module; the size of the convolution kernel in the convolutional network layer is 3×3, which is used to adjust the channels of the input image; the inverse residual network module includes three sequentially connected inverse residual network layers (MV2Block), which are used to downsample and extract the input image features to obtain shallow features containing image information such as image color and texture; the three inverse residual network layers have the same structure, including a sequentially connected 1x1 convolution layer, a 3×3 convolution layer, an axial attention mechanism, and a 1×1 convolution layer; the features input to the inverse residual network layer first pass through a 1x1 convolution layer to increase the number of channels, then pass through a 3×3 convolution layer to extract image features, and then use the axial attention mechanism to enhance the learning ability of the features, and finally pass through a 1×1 convolution layer to restore the number of channels, and the output features of the last 1×1 convolution layer are fused with the input features of the corresponding inverse residual network layer as the output of the inverse residual network layer;

[0066] In this embodiment, the step length of the convolution kernel in the first inverse residual network layer in the inverse residual network module is 1, and the step length of the convolution kernel in the last two inverse residual network layers is 2.

[0067] The self-attention learning mechanism module includes a first self-attention learning mechanism layer and a second self-attention learning mechanism layer connected in sequence; the first self-attention learning mechanism layer includes an inverse residual network layer and two self-attention layers (Res-MobileViT) connected in sequence; the second self-attention learning mechanism layer includes an inverse residual network layer and four self-attention layers connected in sequence; all the self-attention layers have the same structure and include a local feature extraction module and a global feature extraction module connected in sequence; the local feature extraction module includes a depth separable convolution with a convolution kernel of 3 and a normal convolution with a convolution kernel of 1; the global feature extraction module includes a linear multi-head self-attention mechanism and a convolution network with a convolution kernel of 1; the linear multi-head self-attention mechanism is used to expand the receptive field of the network model; the convolution network is used to adjust the channel and enhance the learning ability of the feature; the input and output of the global feature extraction module are fused as the output of the self-attention layer.

[0068] 3-2-2. As shown in Figure 3 The multi-feature fusion module includes a context information fusion module and a multi-scale information fusion module; the context information fusion module includes a first branch, a second branch, and a channel shuffle module connected with the first branch and the second branch respectively; the first branch includes a convolution block (DWconv), a max-pooling layer, and an average-pooling layer; the convolution block includes a 3x3 convolution layer, a batch normalization layer, and a Silu activation function connected in sequence, and is used for feature extraction on the output of the inverse residual network module; the max-pooling layer and the average-pooling layer perform pooling operations on the output of the convolution block respectively; the second branch includes a convolution block and an up-sampling layer connected in sequence, and is used for feature extraction on the output of the first self-attention learning mechanism layer; the outputs of the max-pooling layer and the average-pooling layer in the first branch are added, spliced with the output of the second branch, and input into the channel shuffle module for channel shuffle operation to obtain more comprehensive image features; the multi-scale information fusion module uses a spatial pyramid pooling layer to perform a pooling operation on the image features output by the self-attention learning mechanism module to fuse information of different scales; the output of the context information fusion module after down-sampling operation is fused with the output of the multi-scale information fusion module through pixel-level addition to obtain the output of the multi-feature fusion module.

[0069] 3-2-3. The network decoding module includes two convolutional upsampling layers and a normal upsampling layer connected in sequence; the two convolutional upsampling layers connected in series receive the image features output by the multi-feature fusion module, adjust the image channels through 3x3 convolution and upsampling operations to gradually restore the number of channels and size of the image, and integrate information of different dimensions to enhance the network's ability to understand different receptive fields of the image; the normal upsampling layer receives the image features output by the two convolutional upsampling layers, adjusts the number of channels to the specified number of landform categories, and restores the image size.

[0070] 3-3. Use complex landform datasets to train the landform segmentation network model to obtain stable model weight information and obtain a real-time landform segmentation network model after training.

[0071] Step 4: Obtaining the semantic map of the local environment

[0072] 4-1. Use the depth camera on the robot to collect the original terrain image; select the image frame at the initial moment as the latest key image frame, extract the ORB feature points from the key image frame, and transform the feature points from the image coordinate system to the camera coordinate system to form spatial map points. The transformation relationship is as follows:

[0073] (1)

[0074] (2)

[0075] (3)

[0076] in, is the three-dimensional spatial coordinate corresponding to the spatial map point; For the The depth value of feature points; For the The coordinates of feature points in the image coordinate system; 、 are the optical center positions of the internal parameter matrix R respectively; 、 are the focal lengths in the camera intrinsic parameter matrix R respectively.

[0077] 4-2. Extract ORB feature points from the image frame captured by the depth camera at the current moment, and use the bag-of-words matching method to calculate the matching score between the current image frame and the latest key image frame. The expression is:

[0078] (4)

[0079] in, are two bag-of-words vectors, consisting of the feature descriptors of the latest key image frame and the current image frame; the frequency of the i-th word in the two bag-of-words vectors; the set of words common to the two bag-of-words vectors.

[0080] If the matching score is less than the score threshold , it is considered that the environment has changed greatly, and the current image frame is taken as the latest key image frame. This screening condition is denoted as image feature matching condition , and the case where the matching condition is met is as follows:

[0081] (5)

[0082] In this embodiment, the score threshold is 0.2.

[0083] 4-3. The expressions of the robot position difference and the yaw angle difference between the front and rear frames are as follows:

[0084] (6)

[0085] (7)

[0086] wherein, is the real-time pose of the robot; is the position information of the robot corresponding to the latest key image frame; is the real-time yaw angle of the robot; is the yaw angle of the robot corresponding to the latest key image frame.

[0087] When the robot position difference exceeds the maximum distance threshold , or the yaw angle difference exceeds the maximum angle threshold , it is considered that the robot field of view has changed greatly, and the current image frame is taken as the latest key image frame. This screening condition is denoted as odometer condition :

[0088] (8)

[0089] In this embodiment, the maximum distance threshold and the maximum angle threshold are set to 2 meters and 5 degrees, respectively.

[0090] ​4-4. To prevent the robot from staying in the same position for a long time and not adding new members, a time filtering condition is added: when the time difference between the time corresponding to the latest key image frame and the current time exceeds the time threshold , the current image frame is taken as the latest key image frame, and this condition is recorded as the time constraint condition , and the time constraint condition is specifically expressed as:

[0091] (9)

[0092] , wherein is the time difference between the current image frame and the latest key image frame .

[0093] In this embodiment, the time threshold is set to 3 seconds.

[0094] 4-5. Build a complete latest image key frame filtering condition, and the complete image key frame filtering condition C is defined as follows:

[0095] (10)

[0096] The filtered key image frame is input into the trained real-time terrain segmentation network model to predict and divide different terrain regions, and a terrain semantic segmentation image is obtained.

[0097] Step five, semantic mapping

[0098] 5-1. Time synchronization is performed on the laser point cloud and the terrain semantic segmentation image, a point cloud message queue and a semantic segmentation image message queue are created, and the time difference is limited within a certain range by intercepting the time stamps of the two message queues, so as to synchronize the reception of the two topics.

[0099] 5-2. The points in the laser point cloud are converted into homogeneous coordinates and stored in the 4D variable ;

[0100] 5-3. The variable is mapped to the terrain semantic segmentation image through the intrinsic matrix R and the extrinsic matrix RT, and the result is stored in the variable , and the conversion relationship is defined as follows:

[0101] (11)

[0102] 5-4. The variable is converted back to Euclidean coordinates, and the image coordinates are stored in the current image frame terrain pixel ;

[0103] 5-5. The topographic pixel points of the current image frame are filtered Color transmission to semantic point cloud

[0104] (12)

[0105] p t color .g=img.(pt.y,py.x)[1] (13)

[0106] p t color .b=img.(pt.y,py.x)[0] (14)

[0107] In the above formula, , , respectively, the numerical values of the RGB three channels of the semantic point cloud; is the pixel value corresponding to the topographic pixel point of the current image frame; and are the positions of the topographic pixel points of the current image frame.

[0108] 5-6. The current semantic point cloud is filtered to filter out excessively high point cloud data and prevent interference with topographic information. The specific filtering operation is as follows:

[0109] (15)

[0110] wherein, is the filtered semantic point cloud; is the semantic point cloud coordinate value on the Z-axis coordinate; is the height threshold.

[0111] Step six, a 2.5-dimensional elevation raster map is constructed using a multi-layer two-dimensional grid map format. The semantic information of the semantic point cloud is used to construct a color layer raster map, the point cloud coordinates are converted into two-dimensional grid cells, the corresponding color information is converted into floating point values as the values of the color layer cells, and the color information of the raster is recorded. The correspondence between color information and floating point numbers is as follows:

[0112] (16)

[0113] wherein, is the floating point number converted from the semantic information; , , respectively, the numerical values of the RGB three channels of the semantic point cloud; 16 represents data left shift 16 bits.

[0114] When the grid contains multiple point clouds, Kalman filtering is used to update the grid state, and the specific operation is as follows:

[0115] (17)

[0116] (18)

[0117] (19)

[0118] wherein, is the Kalman gain coefficient at the current time t; is the variance of the prediction error; is the noise variance; is the updated color information; is the predicted state estimate value; is the color information before updating; is the updated error covariance.

[0119] Step seven, record the point cloud in each unit grid in the elevation grid map coordinates, according to the k-medians algorithm, k cluster center points are randomly selected, the distance between each point and each center point is calculated, the point is assigned to the cluster to which the nearest center point belongs, the median of each cluster is calculated to update the center point of the cluster, the above process is repeated until the cluster center no longer changes significantly or reaches the preset maximum iteration number, so that the objective function S of the k-medians algorithm is minimized, and the expression of the objective function S is:

[0120] (20)

[0121] wherein, is the point in the cluster; K is the number of cluster center points; is the number of points in the kth cluster; is the median of the data in the kth cluster.

[0122] The z coordinates of the clustered point cloud are used to construct the elevation layer grid map, and when the same unit grid contains multiple point cloud z coordinates, Kalman filtering is used to update the height, and the specific operation is as follows:

[0123] (21)

[0124] (22)

[0125] (23)

[0126] wherein, is a Kalman gain coefficient for the current time t; is a variance of a prediction error; is a noise variance; is updated height information; is a predicted state estimate value; is height information before updating; is an updated error covariance.

[0127] Step eight, fuse the color layer raster map with the elevation layer raster map to form a local semantic elevation terrain map, and according to the real-time pose change of the robot, construct an incremental global semantic elevation terrain map capable of representing the undulation and height change of a complex terrain environment.

Claims

1. A method for constructing a global semantic elevation landform map in a complex environment, characterized by: The method comprises the following steps: Step one, joint calibration of the external parameters of the multi-line laser radar and the depth camera, to obtain the camera intrinsic parameter matrix and the depth camera radar external parameter matrix; Step two, obtaining real-time pose information of the robot, and obtaining laser point cloud according to the laser radar; Step three, obtaining and training a real-time terrain segmentation network model; In step three, the real-time terrain segmentation network model comprises a real-time mobile network encoding module, a multi-feature fusion module and a network decoding module; the real-time mobile network encoding module comprises a convolution network layer, an inverse residual network module and a self-attention learning mechanism module connected in sequence; The self-attention learning mechanism module comprises a first self-attention learning mechanism layer and a second self-attention learning mechanism layer connected in sequence; the first self-attention learning mechanism layer comprises an inverse residual network layer and two self-attention layers connected in sequence; the second self-attention learning mechanism layer comprises an inverse residual network layer and four self-attention layers connected in sequence; all the self-attention layers have the same structure and comprise a local feature extraction module and a global feature extraction module connected in sequence; the local feature extraction module comprises a depth separable convolution and a normal convolution; the global feature extraction module comprises a linear multi-head self-attention mechanism and a convolution network; the output of the self-attention layer is the feature fused by the input and output of the global feature extraction module; The multi-feature fusion module is used for fusing the features output by the inverse residual network module, the first self-attention learning mechanism layer and the second self-attention learning mechanism layer; the network decoding module is used for obtaining a terrain segmentation semantic map of a local environment according to the image features output by the multi-feature fusion module; In step three, the inverse residual network module comprises three inverse residual network layers connected in sequence; the three inverse residual network layers have the same structure and each comprises a first convolution layer, a second convolution layer, an axial attention mechanism and a third convolution layer connected in sequence; the output feature of the third convolution layer and the feature fused by the input corresponding to the inverse residual network layer are taken as the output of the inverse residual network layer; The multi-feature fusion module comprises a context information fusion module and a multi-scale information fusion module; the context information fusion module comprises a first branch, a second branch and a channel shuffle module; the first branch processes the output of the inverse residual network module through a convolution block, and performs a pooling operation on the output of the convolution layer through a maximum pooling layer and an average pooling layer respectively, adds the outputs of the maximum pooling layer and the average pooling layer to obtain a first output feature map; the second branch extracts features from the output of the first self-attention learning mechanism layer through a convolution block and an up-sampling layer in sequence to obtain a second output feature map; the convolution block comprises a convolution layer, a batch normalization layer and an activation function connected in sequence; the channel shuffle module specifically obtains a preliminary fusion feature map through a channel shuffle operation after splicing the first output feature map and the second output feature map; The multi-scale information fusion module adopts a spatial pyramid pooling layer to perform a pooling operation on image features output by the self-attention learning mechanism module; and the features obtained by adding and fusing the features after the preliminary fused feature map is down-sampled to the features output by the multi-scale information fusion module are taken as the output of the multi-feature fusion module; The network decoding module sequentially processes the image features output by the multi-feature fusion module through two convolutional up-sampling layers and one ordinary up-sampling layer to obtain a landform segmentation semantic map of the local environment; Step four: obtaining a landform segmentation semantic map of the local environment according to the trained real-time landform segmentation network model; In step four, the method for obtaining the landform segmentation semantic map is as follows: Taking the image frame at the initial moment as the latest key image frame; Constructing a latest image key frame screening condition to screen each image frame obtained at each moment; obtaining the latest key image frame closest to the current moment, inputting the latest key image frame into the real-time landform segmentation network model, and obtaining the landform segmentation semantic map; The latest image key frame screening condition includes an image feature matching condition an odometer condition and a time constraint condition ​ Step five: time-synchronizing the laser point cloud and the landform segmentation semantic map, and projecting and mapping the point cloud to the landform segmentation semantic map by using the intrinsic parameter matrix and the extrinsic parameter matrix to obtain the semantic point cloud corresponding to each frame of landform image; Step six: constructing a color layer grid map by using the semantic information of the semantic point cloud; constructing an elevation layer grid map by using the z coordinate of the semantic point cloud; fusing the color layer grid map and the elevation layer grid map to form a local semantic elevation landform map, and constructing a global semantic elevation landform map according to the real-time pose of the robot and the coordinate transformation relationship. 2.The method of claim 1, wherein the method further comprises: determining a semantic height of each point in the point cloud based on the semantic label of each point and the height of each point; and constructing a global semantic height landscape map based on the semantic height of each point in the point cloud. The specific process of step five is as follows: Step 5-1. Synchronously receiving the point cloud message queue and the semantic segmentation map message queue; Step 5-2. Convert points in the laser point cloud to homogeneous coordinates and store them in a 4D variable ; Step 5-3. Map the variables by the intrinsic and extrinsic matrix onto the topographic semantic segmentation image and store the result in the variable ​ Step 5 - 4. The variable Convert back to Euclidean coordinates, store image coordinates to the current image frame topography pixel in the middle; Step 5 - 5. The current image frame topography pixel points Color transfer to semantic point cloud ; Step 5-6. Filtering the current semantic point cloud by height performing height filtering to filter out point cloud data greater than a height threshold to obtain a filtered semantic point cloud .

3. The method of claim 1, wherein the method comprises: The image feature matching condition The expression is: ； wherein, is a matching score between the current image frame and the latest key image frame; is a score threshold value; The odometer condition The expression is: ； wherein, is a difference in position of the robot between the current image frame and the latest key image frame; is a difference in yaw angle of the robot between the current image frame and the latest key image frame; is a maximum distance threshold value; is a maximum angle threshold value; The time constraint condition The expression is: ； wherein, is a time difference between the current image frame and the latest key image frame; is a time threshold.

4. The method of claim 3, wherein the method further comprises: The score threshold Set to 0.2; maximum distance threshold and the maximum angle threshold Set to 2 meters and 5 degrees respectively; time threshold Set to 3 seconds.

5. The method of claim 1, wherein the method is used for constructing a global semantic elevation landform map in a complex environment. The method for obtaining the real-time pose information of the robot is to fuse the sensor data of the laser radar, GPS and inertial measurement unit of the robot, construct a laser odometry, and obtain the real-time pose information of the robot.

6. The method of claim 1, wherein the method is used for constructing a global semantic elevation landform map in a complex environment. The specific process of step six is as follows: Step 6-1. Converting the point cloud coordinates into two-dimensional grid units, converting the corresponding color information into floating-point values as the values of the color layer cells, and recording the color information of the grid; when the grid contains multiple point clouds, Kalman filtering is used to update the state of the grid; Step 6-2. Record the point cloud in each unit grid in the elevation grid map The z coordinates are clustered according to the k-medians algorithm, the z coordinates of the clustered point cloud are used to construct an elevation layer grid map, and when a same unit grid contains multiple point cloud z coordinates, the height is updated using Kalman filtering.

7. The method of claim 1, wherein the method is used for constructing a global semantic elevation landform map in a complex environment. The specific process of step three is as follows: a. Using the robot to carry a depth camera to collect multiple landform images in different environments to construct a data set, and labeling the data set with a landform type; b. Constructing a real-time landform segmentation network model; c. Training the landform segmentation network model using the data set to obtain the trained real-time landform segmentation network model.

Images