A method, system, apparatus, and medium for constructing robot navigation maps.

Abstract

This application relates to a method and system for constructing a robot navigation map. The method includes: acquiring laser point clouds, RGB images, and RGBD dense point clouds; processing the laser point clouds using odometry to obtain a transformation matrix; segmenting the RGB images using a neural network to obtain a semantic segmentation image; projecting the laser point clouds onto the semantic segmentation image based on camera intrinsic and extrinsic parameters to acquire semantic information from the laser point clouds; aligning the point clouds with semantic information with a 3D sub-map based on the transformation matrix; performing dimensionality reduction processing on the 3D sub-map with semantic information in the Z-direction, fusing the occupancy probability, semantic information, and dense point cloud information in the Z-direction grid into the occupancy probability, semantic information, and elevation information of a 2.5D grid to obtain a 2.5D sub-map; adjusting and updating the pose of the 2.5D grid sub-map when correcting accumulated odometry errors, and stitching the updated sub-map to obtain a final 2.5D multi-level map. This improves the accuracy and efficiency of map construction and reduces resource consumption.

Description

Technical Field

[0001] This application relates to the field of map construction technology, and in particular to a method, system, apparatus and medium for constructing a robot navigation map. Background Technology

[0002] Existing methods for constructing robot navigation maps include: real-time acquisition of environmental information surrounding the mobile robot during its movement, including environmental point cloud data; then, extraction of obstacle point cloud data and road surface point cloud data from each frame of environmental point cloud data; generation of a map point cloud layer and a map semantic layer based on the obstacle point cloud data, wherein the map point cloud layer is a layer used to represent obstacle information, and the map semantic layer is a layer used to represent the semantic information of obstacles; next, generation of a map topology layer used to represent road topology information based on the road surface point cloud data; and finally, generation of a high-precision 3D map based on the map point cloud layer, map semantic layer, and map topology layer.

[0003] However, in existing technologies, 3D point cloud layers generally require a large amount of storage space, placing high demands on data storage, loading, and processing. Furthermore, the unstructured nature of point cloud data makes indexing even more difficult. In addition, object detection and semantic segmentation methods based on point cloud data suffer from low accuracy and reliability due to the sparsity and lack of texture features, particularly for distant objects or objects with similar geometric features. Moreover, for non-standard paved surfaces, especially rugged roads and staircases, existing technologies lack elevation information and processing for accumulated errors, leading to inaccurate maps that frequently require updates, resulting in significant time and effort.

[0004] Therefore, no effective solution has yet been proposed to address the problems of low accuracy, low efficiency, and high computer resource consumption in the construction of robot navigation maps in existing technologies. Summary of the Invention

[0005] This application provides a method, system, apparatus, and medium for constructing robot navigation maps, to at least solve the problems of low accuracy, low efficiency, and high computer resource consumption in the construction of robot navigation maps in related technologies.

[0006] In a first aspect, embodiments of this application provide a method for constructing a robot navigation map, the method comprising:

[0007] Acquire lidar point cloud data, RGB images, and RGBD dense point cloud data. Process the lidar point cloud data using the odometry method to obtain the transformation matrix of the aligned point cloud frames. Segment the RGB images using a deep neural network model to obtain semantic segmentation images.

[0008] Based on the camera's intrinsic and extrinsic parameters, the laser point cloud is projected onto the semantic segmentation image, enabling the laser point cloud to acquire semantic information from the semantic segmentation image. Then, based on the transformation matrix, the point cloud with semantic information is aligned with the 3D grid map to obtain a 3D grid map with semantic information.

[0009] The 3D raster sub-map with semantic information is subjected to dimensionality reduction processing in the Z direction. The occupancy probability information, semantic information and RGBD dense point cloud information in the raster in the Z direction are fused into the occupancy probability information, semantic information and elevation information of the 2.5D raster, and a 2.5D raster sub-map is obtained.

[0010] When correcting accumulated errors in the odometer, the pose of the 2.5D grid sub-map is adjusted and updated, and the updated sub-maps are stitched together to obtain the final 2.5D multi-level map.

[0011] In some embodiments, segmenting the RGB image using a deep neural network model to obtain a semantic segmentation image includes:

[0012] The RGB image is semantically segmented using a DeepLabv3 neural network model, into passable regions, impassable regions, and other regions.

[0013] In some embodiments, after acquiring RGBD dense point cloud data, the method includes:

[0014] The specific calculation formula for converting RGBD depth values ​​into 3D point clouds is as follows:

[0015]

[0016] Where (u, v) is the pixel position, d is the depth, and f x and f y c is the scaling intrinsic parameter in the camera intrinsic parameters. x and c y This is for shifting internal references.

[0017] In some embodiments, the method includes, before projecting the laser point cloud onto the semantic segmentation image based on camera intrinsic and extrinsic parameters:

[0018] The RGBD dense point cloud data and the LiDAR point cloud data are respectively converted to the robot's base_link coordinate system. The specific conversion formula is as follows:

[0019]

[0020] in, and These represent dense point cloud data in the RGBD camera coordinate system and point cloud data in the LiDAR coordinate system, respectively. This represents the transformation matrix from the camera coordinate system to the world coordinate system. This represents the transformation matrix from the world coordinate system to the base_link coordinate system. This represents the transformation matrix from the lidar coordinate system to the base_link coordinate system.

[0021] In some embodiments, the laser point cloud is projected onto the semantic segmentation image based on the camera's intrinsic and extrinsic parameters, so that the laser point cloud acquires semantic information from the semantic segmentation image, including:

[0022] Based on the camera's intrinsic parameters and the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is projected onto the semantic segmentation image to obtain the semantic label information of the pixel positions corresponding to the point cloud. The specific projection formula is as follows:

[0023]

[0024] Among them, f x and f y c is the scaling intrinsic parameter in the camera intrinsic parameters. x and c y To shift the internal reference, This represents the transformation matrix from the base_link coordinate system to the camera coordinate system, (u i v i (x) represents the pixel position, (x) i y i , z i ) represents the coordinates of a point in the laser point cloud in the camera coordinate system.

[0025] In some embodiments, aligning the point cloud with semantic information to a 3D raster map according to a transformation matrix to obtain a 3D raster map with semantic information includes:

[0026] Based on the transformation matrix provided by the odometer, the semantic point cloud is aligned with the 3D grid submap, and points with semantic information are inserted into the 3D grid map to update the occupancy probability value of the 3D grid and register semantic information for the 3D grid.

[0027] In some embodiments, fusing occupancy probability information, semantic information, and RGBD dense point cloud information in the Z-direction raster into occupancy probability information, semantic information, and elevation information of the 2.5D raster includes:

[0028] Remove grid cells from the 3D grid that are semantically passable areas and contain moving objects, and then calculate the occupancy probability value of the 2.5D grid by applying a Gaussian function for height weighting. The specific calculation formula is as follows:

[0029]

[0030]

[0031]

[0032] Where μ is the expectation of the Gaussian function, w(h) is the Gaussian function, and w i Let p be the weight of grid i, and p be the occupancy probability value of the 2.5D grid.

[0033] Semantic information from the 3D raster is accumulated in the Z direction to obtain semantic information from the 2.5D raster.

[0034] The elevation information of the 2.5D raster is obtained by weighting the heights of the 3D raster with an occupancy probability exceeding a threshold by using the weights calculated from the Gaussian distribution of the height of the dense point cloud fused from multiple frames of RGBD within the raster strip range.

[0035] Secondly, embodiments of this application provide a robot navigation map construction system, the system comprising:

[0036] The acquisition module is used to acquire lidar point cloud data, RGB images, and RGBD dense point cloud data. It processes the lidar point cloud data using an odometry method to obtain a transformation matrix for aligned point cloud frames, and segments the RGB images using a deep neural network model to obtain semantic segmentation images.

[0037] The projection alignment module is used to project the laser point cloud onto the semantic segmentation image according to the camera's intrinsic and extrinsic parameters, so that the laser point cloud obtains the semantic information on the semantic segmentation image, and aligns the point cloud with the semantic information with the 3D grid map according to the transformation matrix to obtain the 3D grid map with semantic information.

[0038] The dimension reduction and fusion module is used to perform dimension reduction processing on the 3D grid sub-map with semantic information in the Z direction, and to fuse the occupancy probability information, semantic information and RGBD dense point cloud information in the grid in the Z direction into the occupancy probability information, semantic information and elevation information of the 2.5D grid, so as to obtain the 2.5D grid sub-map.

[0039] The update stitching module is used to adjust and update the pose of the 2.5D grid sub-map when the odometer corrects the accumulated error, and stitch the updated sub-map to obtain the final 2.5D multi-level map.

[0040] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0041] Fourthly, embodiments of this application provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0042] Compared to related technologies, the robot navigation map construction method provided in this application acquires LiDAR point cloud data, RGB images, and RGBD dense point cloud data. It processes the LiDAR point cloud data using odometry to obtain a transformation matrix for aligning point cloud frames. A deep neural network model is used to segment the RGB image to obtain a semantic segmentation image. Based on the camera's intrinsic and extrinsic parameters, the LiDAR point cloud is projected onto the semantic segmentation image, enabling the LiDAR point cloud to acquire semantic information from the semantic segmentation image. The point cloud with semantic information is then aligned with a 3D grid sub-map based on the transformation matrix, resulting in a 3D grid sub-map with semantic information. The 3D grid sub-map with semantic information undergoes dimensionality reduction processing in the Z-direction, fusing the occupancy probability information, semantic information, and RGBD dense point cloud information in the grid along the Z-direction into occupancy probability information, semantic information, and elevation information of a 2.5D grid, resulting in a 2.5D grid sub-map. When correcting accumulated errors using odometry, the pose of the 2.5D grid sub-map is adjusted and updated, and the updated sub-maps are stitched together to obtain the final 2.5D multi-level map.

[0043] Compared to existing 3D grid maps, this application automatically constructs a 2.5D grid map with less data volume and describes the 3D environment in which the robot works using a 2.5D multi-level grid map. Semantic information from images is incorporated into the point cloud to obtain a semantic map with positional coordinates. Furthermore, point cloud data from RGBD cameras and LiDAR are fused to obtain more accurate elevation information. This results in a map with significant advantages: structured data, lower data volume, and richer data information. In addition, this application corrects accumulated errors more efficiently by correcting existing maps without requiring reconstruction, effectively saving time. This solves the problems of low accuracy, low efficiency, and high computer resource consumption in the construction of robot navigation maps in related technologies. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1 This is a flowchart of a method for constructing a robot navigation map according to an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of projecting a point cloud onto an image plane according to an embodiment of this application;

[0047] Figure 3 This is a schematic diagram illustrating the number of times a grid is hit or penetrated according to an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of weight calculation according to an embodiment of this application;

[0049] Figure 5 This is a structural block diagram of a robot navigation map construction system according to an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0053] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] This embodiment provides a method for constructing a robot navigation map. Figure 1 This is a flowchart of a robot navigation map construction method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0055] Step S101: Acquire lidar point cloud data, RGB image and RGBD dense point cloud data. Process the lidar point cloud data using the odometry method to obtain the transformation matrix of the aligned point cloud frame. Segment the RGB image using a deep neural network model to obtain a semantic segmentation image.

[0056] Preferably, in this embodiment, LiDAR point cloud data, RGB images, and RGBD dense point cloud data of the surrounding environment are acquired. The LiDAR point cloud data and IMU data are used as inputs, with GNSS as an optional input, and input into the LIO-SAM odometry algorithm. This algorithm outputs odometry information at the same frequency as the IMU, obtaining a transformation matrix for aligned point cloud frames. This facilitates the optimization of accumulated errors and updates of key poses during subsequent mapping. Different input data can be selected based on the robot's hardware conditions, such as IMU, GNSS, encoders, and cameras. The output odometry frequency must be at least greater than or equal to the keyframe frequency of the point cloud inserted into the 3D grid map, and must carry time information or the corresponding point cloud frame sequence number.

[0057] It should be noted that the sensor components in this embodiment are a Velodyne VLP16 LiDAR and a RealSense D435i RGBD camera; the software deployment environment is an Ubuntu 18.04 operating system and a ROS Melodic robot operating system; the chassis is a bionic quadruped robot. The Velodyne VLP16 line LiDAR has a horizontal angular resolution of 0.1°–0.4°, a horizontal field of view of 360°, a vertical angular resolution of 2°, a vertical field of view of -15°–+15°, and an effective detection range of up to 100m; the RealSense D435i RGBD camera has a depth resolution of 1280×720, an RGB resolution of 1920×1080, and a depth detection range of 0.2m–10m. LiDAR has a long detection range, high accuracy, and a large field of view. RGBD depth maps generate dense point clouds, but their accuracy is insufficient and their field of view is small. In this embodiment, LiDAR point clouds are used as the main source and RGBD point clouds are used as a supplement. The point cloud data collected by the two are fused together, which can effectively improve the density and reliability of elevation information.

[0058] Simultaneously, a semantic segmentation image is obtained by segmenting the RGB image using a deep neural network model. Preferably, in this embodiment, the RGB image is semantically segmented twice using a DeepLabv3 neural network model. The first segmentation divides the image into passable areas, impassable areas, and others. The second segmentation further subdivides the passable areas into four categories: paved roads, dirt roads, steps, and grass. The impassable areas are subdivided into four categories: pedestrians, vegetation, walls, and vehicles. The segmentation result is a pixel-level labeled image.

[0059] In one embodiment, after acquiring RGBD dense point cloud data, it is necessary to convert the RGBD depth values ​​into 3D point clouds. The specific calculation formula for the three-dimensional points is as follows (1):

[0060]

[0061] Where (u, v) is the pixel position, d is the depth, and f x and f y c is the scaling intrinsic parameter in the camera intrinsic parameters. x and c y This is for shifting internal references.

[0062] Step S102: Based on the camera's intrinsic and extrinsic parameters, the laser point cloud is projected onto the semantic segmentation image, so that the laser point cloud obtains semantic information from the semantic segmentation image. Then, based on the transformation matrix, the point cloud with semantic information is aligned with the 3D grid map to obtain a 3D grid map with semantic information.

[0063] Preferably, before projecting the laser point cloud onto the semantic segmentation image based on the camera's intrinsic and extrinsic parameters, the RGBD dense point cloud data and the LiDAR point cloud data are converted to the robot's base_link coordinate system, respectively. The specific conversion formulas are shown in equations (2) and (3) below:

[0064]

[0065]

[0066] in, and These represent dense point cloud data in the RGBD camera coordinate system and point cloud data in the LiDAR coordinate system, respectively. This represents the transformation matrix from the camera coordinate system to the world coordinate system. This represents the transformation matrix from the world coordinate system to the base_link coordinate system. This represents the transformation matrix from the lidar coordinate system to the base_link coordinate system.

[0067] Figure 2 This is a schematic diagram of projecting a point cloud onto an image plane according to an embodiment of this application, such as... Figure 2 As shown in the left figure, based on the camera's intrinsic parameters and the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is projected onto the semantic segmentation image to obtain the semantic label information of the pixel positions corresponding to the point cloud. Assuming the points in the point cloud... Its corresponding pixel position is (u i v i The specific projection formula is as follows (4):

[0068]

[0069] Among them, f x and f y c is the scaling intrinsic parameter in the camera intrinsic parameters. x and c y To shift the internal reference, This represents the transformation matrix from the base_link coordinate system to the camera coordinate system.

[0070] Next, the point cloud with semantic information is aligned with the 3D grid submap according to the transformation matrix, resulting in a 3D grid submap with semantic information. Specifically, the point cloud with semantic information is aligned with the 3D grid map according to the transformation matrix provided by the odometry, and points with semantic information are inserted into the 3D grid map to update the occupancy probability value of the 3D grid and register semantic information for the 3D grid. The occupancy probability value is proportional to the number of points hitting the grid and inversely proportional to the number of times the grid is traversed. The step size of its probability update can be changed according to the number of times the grid is hit by different point cloud frames; the semantic information of the 3D grid comes directly from the semantic information carried by the points hitting the grid.

[0071] Specifically, such as Figure 2 As shown in the right image, the 3D raster submap is a three-dimensional voxel grid with a fixed resolution. Each grid cell contains an occupancy probability value p, a hit count n, and a penetration count m. The occupancy probability value ranges from 0 to 1, with a default of 0.5 indicating unknown. A hit count of 0 indicates neither hit nor penetration. For ease of calculation, the odds value is introduced:

[0072] A scan line is determined by a point, and the grid that intersects with the scan line is found and updated using the Bresenham algorithm. The specific update calculation formula is shown in equation (5) below:

[0073] odds k =k h，m odds k-1 (5)

[0074] Since a grid cell can be hit or penetrated, the odds value of a hit grid cell is multiplied by the hit coefficient k. h Therefore, the corresponding occupancy probability value will increase; the odds value of the penetrated grid is multiplied by the miss coefficient k. m Therefore, the corresponding occupancy probability value will decrease.

[0075] It should be noted that this embodiment uses a lookup table to update the odds values, i.e., pre-calculates the possible odds values. A variable step size is used when updating the occupancy probability value because if a grid is hit in several consecutive frames, it is more likely to be a static object; increasing the step size allows it to update to a higher occupancy probability more quickly. If a grid is hit or penetrated in some point cloud frames, it may be a dynamic object, and the step size should be decreased to update the occupancy probability more slowly. Similarly, a grid penetrated by multiple point cloud frames is more likely to be unoccupied, and the step size should be increased to update to a lower occupancy probability more quickly. The specific process is as follows: Figure 3 As shown, Figure 3This is a schematic diagram illustrating the number of times a grid is hit or penetrated according to an embodiment of this application. It is assumed that initially, both the number of hits (n) and the number of penetrations (m) are 0. If a grid is hit in the next instance, the number of hits (n) is incremented by 1, and the grid is looked up to the right using n as the step size. The number of penetrations (m) is decremented by 1, and this decrement stops when m is 0. Conversely, if a grid is penetrated, the number of hits (n) is decremented by 1, and the number of penetrations (m) is incremented by 1, and the grid is looked up to the left using m as the step size, and this decrement stops when n is 0. Based on the odds value obtained from the lookup table, the grid occupancy probability value (p) can be calculated. The specific calculation formula is shown in equation (6) below:

[0076]

[0077] The semantic information of the 3D grid comes directly from the semantic information carried by the points that hit the grid. Since the number of semantic labels is predetermined, the semantic information is stored in a 16-bit integer. Each of the lower eight bits represents a label category. A grid can have multiple semantic labels, and the labels come from the semantic information carried by the points that hit the grid.

[0078] Step S103: Perform dimensionality reduction processing on the 3D raster sub-map with semantic information in the Z direction, and fuse the occupancy probability information, semantic information and RGBD dense point cloud information in the raster in the Z direction into the occupancy probability information, semantic information and elevation information of the 2.5D raster to obtain the 2.5D raster sub-map.

[0079] Preferably, in this embodiment, the 3D raster sub-map with semantic information undergoes dimensionality reduction processing in the Z-direction, fusing the occupancy probability information, semantic information, and RGBD dense point cloud information in the raster in the Z-direction into occupancy probability information, semantic information, and elevation information of the 2.5D raster. Specifically:

[0080] Regarding occupancy probability information, thanks to the semantic information carried in the 3D grid, when compressing the 3D grid into a 2.5D grid, grids in the 3D grid that semantically represent passable areas and moving objects, such as people and vehicles, are first removed. These grids are directly treated as idle grids and do not participate in the calculation of occupancy probability. Then, the occupancy probability of the grid in the Z direction is weighted and summed using the integral value of the Gaussian function over the grid's height range, thus becoming the occupancy probability value of the 2.5D grid at position (x, Y). It should be noted that the Gaussian function is used for weighting because the robot's passability mainly considers whether there are obstacles within the robot's own height range. Therefore, the expectation μ of the Gaussian function is set to the height of the robot's center of gravity, and the variance should be chosen to have a large cumulative weight within the robot's height range. For grid i, its weight w... i The specific calculation formulas are shown in equations (7) and (8) below:

[0081]

[0082]

[0083] Where μ is the expectation of the Gaussian function, and w(h) is the Gaussian function; Figure 4 This is a schematic diagram of weight calculation according to an embodiment of this application, such as... Figure 4 As shown in the left figure, the cumulative weight of the weight calculated by formula (8) in the interval from a to b is 1. Then, the occupancy probability value of the 2.5D grid is calculated by weighted summation of multiple grids in the Z direction. The specific calculation formula is shown in formula (9) below:

[0084]

[0085] For semantic information, the 2.5D raster stores semantic information in the same way as the 3D raster. Therefore, it is only necessary to accumulate the semantic information of multiple rasters in the Z direction into a single 2.5D raster, or accumulate only the semantic information within a height range, to obtain the semantic information of the 2.5D raster. Specifically, as long as there is a certain semantic label of a certain raster in the Z direction, the corresponding position of the 16-bit integer in the 2.5D raster will be 1, thus obtaining the semantic information of the 2.5D raster.

[0086] For elevation information, the elevation value of the 2.5D raster is obtained by weighted summation of the heights of 3D rasters whose occupancy probability exceeds a threshold, calculated using a Gaussian height distribution fused from multiple frames of dense RGBD point clouds within the raster strip area. Here, the raster strip is the region obtained by extending the 2.5D raster in the Z direction, such as... Figure 4 As shown in the right figure, the dense points (x) obtained by the RGBD camera within the grid bars are statistically analyzed. i y i , z i Height value z j The mean and variance are calculated as shown in equations (10) and (11):

[0087]

[0088]

[0089] Assuming the point distribution follows a Gaussian distribution, the height distribution of points within a grid strip can be expressed as f(x) = N(u, σ). 2 Considering that the point cloud data acquired by the RGBD camera is not very stable and may have jumps, the mean and variance of the dense RGBD point cloud data from the previous few frames can be combined to make the estimated distribution more accurate. Therefore, the point cloud data from the previous few frames needs to be unified to the coordinate system of the current frame first, and the specific calculation is shown in the following formula (12):

[0090]

[0091] Where i represents the current frame, and the RGBD dense point cloud P of the j-th frame preceding i. i-j through Transform to the current frame coordinate system to obtain P' i-j ,and At this point, the raster bars can contain data from multiple frames of point clouds. The specific calculation of the elevation value of the 2.5D raster is shown in equation (13) below:

[0092]

[0093] Among them, h i w is the height of the center of the i-th 3D grid. i The weight of the i-th 3D grid cell is obtained by integrating the Gaussian distribution f(x) over the interval from i to i+1. For example... Figure 4 As shown in the right figure, 3D graticules with an occupancy probability value below a threshold will not be included in the elevation value calculation. If the occupancy probability of 3D graticules in the Z direction does not exceed the threshold, but the variance of the RGBD point cloud computing within the graticule strip is less than the threshold, and the number of points within the graticule strip exceeds the threshold, then the mean of the RGBD point cloud computing within the graticule strip can be used as the elevation value of this 2.5D graticule to fill the elevation gap caused by insufficient density of the LiDAR. In this way, the elevation information of the 2.5D graticule can be obtained, thereby determining the 2.5D graticule sub-map.

[0094] In step S104, when the odometer corrects the accumulated error, the pose of the 2.5D grid sub-map is adjusted and updated, and the updated sub-map is stitched together to obtain the final 2.5D multi-level map.

[0095] In this implementation, when the odometry corrects the accumulated error and updates the pose, the pose of the keyframes is updated. When building the grid map, only these keyframes are used. Therefore, adjusting the keyframes also updates the pose of the entire map. By updating the keyframes, the computational load of map building can be greatly reduced, and the odometry-updated pose can be used more conveniently.

[0096] After updating and obtaining new sub-maps, the updated sub-maps are stitched together to obtain the final 2.5D multi-level map.

[0097] Specifically, the purpose of stitching submaps is to reuse previously constructed maps. When odometry optimizes poses, it adjusts all unfrozen poses to minimize the error between the overall calculated and observed values. This results in a large number of poses being adjusted, which means that the previously constructed map is no longer accurate. However, by using submaps, it can be ensured that within the smaller area of ​​the submap, the pose adjustment has little impact on the submap. Therefore, only the pose of the submap needs to be adjusted to construct a more accurate map.

[0098] Furthermore, there is some overlap between sub-maps. That is, the previous sub-map is constructed from the point cloud of frames i to i+m, and the next sub-map is constructed from the point cloud of frames i+mn to i+2m-n. In other words, the sub-map contains m frames of point cloud, and there is n frames of overlap between the sub-maps.

[0099] Therefore, stitching sub-maps together can reduce the computational load of map stitching and ensure the real-time nature of map generation.

[0100] Furthermore, during submap stitching, the new pose is applied to the submap's index to obtain the index of the 2.5D submap's grid within the 2.5D map. If the grid was previously empty, the value from the submap grid is directly copied to the new grid. If the 2.5D map's grid already has a value, the average occupancy probability of the two grids is used as the occupancy probability of the new grid. Semantic information is directly accumulated, and the elevation value is also the average of the two grids. This process yields the final 2.5D multi-level grid map.

[0101] Through steps S101 to S104 above, this embodiment uses LiDAR and RGBD camera data as input, and inserts point cloud data with semantic information into a 3D grid sub-map using odometry information. Then, the 3D grid sub-map is reduced to a 2.5D grid map, where each grid cell in the 2.5D grid map integrates the occupancy probability information, semantic information, and elevation information of multiple grid cells at different heights in the 3D grid map. This solves the problems of low accuracy, low efficiency, and high computer resource consumption in the construction of robot navigation maps in related technologies.

[0102] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0103] This embodiment also provides a robot navigation map construction system, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as described above. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0104] Figure 5 This is a structural block diagram of a robot navigation map construction system according to an embodiment of this application, such as... Figure 2 As shown, the system includes an acquisition module 51, a projection alignment module 52, a dimensionality reduction and fusion module 53, and an update and stitching module 54.

[0105] The acquisition module 51 is used to acquire LiDAR point cloud data, RGB images, and RGBD dense point cloud data. It processes the LiDAR point cloud data using odometry to obtain a transformation matrix for aligning point cloud frames. A deep neural network model is used to segment the RGB images to obtain a semantically segmented image. The projection alignment module 52 projects the LiDAR point cloud onto the semantically segmented image based on the camera's intrinsic and extrinsic parameters, enabling the LiDAR point cloud to acquire semantic information from the semantically segmented image. It then aligns the point cloud with semantic information with the 3D raster submap according to the transformation matrix, obtaining a semantically segmented image. The system includes a 3D raster sub-map with semantic information; a dimensionality reduction and fusion module 53, which performs dimensionality reduction processing on the 3D raster sub-map with semantic information in the Z direction, and fuses the occupancy probability information, semantic information and RGBD dense point cloud information in the raster in the Z direction into the occupancy probability information, semantic information and elevation information of the 2.5D raster, to obtain a 2.5D raster sub-map; and an update and stitching module 54, which adjusts and updates the pose of the 2.5D raster sub-map when the odometer corrects the cumulative error, and stitches the updated sub-map to obtain the final 2.5D multi-level map.

[0106] This embodiment uses LiDAR and RGBD camera data as input, and inserts point cloud data with semantic information into a 3D grid map using odometry information. Then, the 3D grid map is reduced to a 2.5D grid map, where each grid cell integrates occupancy probability information, semantic information, and elevation information from multiple grid cells at different heights in the 3D grid map. This solves the problems of low accuracy, low efficiency, and high computer resource consumption in the construction of robot navigation maps in related technologies.

[0107] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0108] Furthermore, it should be noted that the aforementioned modules can be either functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the aforementioned modules can reside in the same processor; or the aforementioned modules can be located in different processors in any combination.

[0109] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0110] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0111] Furthermore, in conjunction with the robot navigation map construction method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the robot navigation map construction methods in the above embodiments.

[0112] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a method for constructing a robot navigation map. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0113] In one embodiment, Figure 6 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 6 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 6As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a method for constructing a robot navigation map, and the database stores data.

[0114] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0116] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for constructing a robot navigation map, characterized in that, The method includes: Acquire lidar point cloud data, RGB images, IMU data, and RGBD dense point cloud data. Process the lidar point cloud data and IMU data using an odometry method to output odometry information with the same frequency as the IMU, obtain a transformation matrix for aligned point cloud frames, and segment the RGB image using a deep neural network model to obtain a semantic segmentation image. The output odometry frequency must be at least greater than or equal to the point cloud keyframe frequency inserted into the 3D raster map, and carry time information or the corresponding point cloud frame sequence information. Based on the camera's intrinsic and extrinsic parameters, the laser point cloud is projected onto the semantic segmentation image, enabling the laser point cloud to acquire semantic information from the semantic segmentation image. Then, based on the transformation matrix, the point cloud with semantic information is aligned with the 3D grid map to obtain a 3D grid map with semantic information. The 3D raster sub-map with semantic information is subjected to dimensionality reduction processing in the Z-direction. The occupancy probability information, semantic information, and RGBD dense point cloud information in the Z-direction raster are fused into occupancy probability information, semantic information, and elevation information of the 2.5D raster, resulting in a 2.5D raster sub-map. The fusion of the occupancy probability information, semantic information, and RGBD dense point cloud information in the Z-direction raster into 2.5D raster occupancy probability information, semantic information, and elevation information includes: Remove grid cells from the 3D grid that are semantically passable areas and contain moving objects, and then calculate the occupancy probability value of the 2.5D grid by applying a Gaussian function for height weighting. The specific calculation formula is as follows: Where μ is the expectation of the Gaussian function, and w(h) is the Gaussian function. The weight of grid i, This represents the occupancy probability value of a 2.5D grid. Semantic information from the 3D raster is accumulated in the Z direction to obtain semantic information from the 2.5D raster. The elevation information of the 2.5D grid is obtained by weighting the heights of 3D grids whose occupancy probability exceeds a threshold by calculating the weights of the Gaussian distribution of heights of dense point clouds fused from multiple frames of RGBD within the grid strip range. When correcting accumulated errors in the odometer, the keyframe poses of the 2.5D grid sub-map are adjusted and updated, and the updated sub-maps are stitched together to obtain the final 2.5D multi-level map.

2. The method according to claim 1, characterized in that, After acquiring RGBD dense point cloud data, the method includes: The specific calculation formula for converting RGBD depth values ​​into 3D point clouds is as follows: in, d is the pixel position, and d is the depth. and The scaling intrinsic parameter in the camera intrinsic parameters. and This is for shifting internal references.

3. The method according to claim 1, characterized in that, Before projecting the laser point cloud onto the semantic segmentation image based on the camera's intrinsic and extrinsic parameters, the method includes: The RGBD dense point cloud data and the LiDAR point cloud data are respectively converted to the robot's base_link coordinate system. The specific conversion formulas are as follows: ， in, and These represent dense point cloud data in the RGBD camera coordinate system and point cloud data in the LiDAR coordinate system, respectively. This represents the transformation matrix from the camera coordinate system to the world coordinate system. This represents the transformation matrix from the world coordinate system to the base_link coordinate system. This represents the transformation matrix from the lidar coordinate system to the base_link coordinate system.

4. The method according to claim 1, characterized in that, Based on the camera's intrinsic and extrinsic parameters, the laser point cloud is projected onto the semantic segmentation image, enabling the laser point cloud to acquire semantic information from the semantic segmentation image, including: Based on the camera's intrinsic parameters and the extrinsic parameters between the camera and the LiDAR, the LiDAR point cloud is projected onto the semantic segmentation image to obtain the semantic label information of the pixel positions corresponding to the point cloud. The specific projection formula is as follows: in, and The scaling intrinsic parameter in the camera intrinsic parameters. and To shift the internal reference, This represents the transformation matrix from the base_link coordinate system to the camera coordinate system. For pixel position, The coordinates of a point in the laser point cloud are given in the camera coordinate system.

5. The method according to claim 1, characterized in that, Aligning the point cloud with semantic information with the 3D raster map using the transformation matrix, the resulting 3D raster map with semantic information includes: Based on the transformation matrix provided by the odometer, the semantic point cloud is aligned with the 3D grid submap, and points with semantic information are inserted into the 3D grid map to update the occupancy probability value of the 3D grid and register semantic information for the 3D grid.

6. The method according to claim 1, characterized in that, The semantic segmentation image obtained by segmenting the RGB image using a deep neural network model includes: The RGB image is semantically segmented using a DeepLabv3 neural network model, into passable regions, impassable regions, and other regions.

7. A system for constructing a robot navigation map, characterized in that, The system includes: The acquisition module is used to acquire LiDAR point cloud data, RGB images, IMU data, and RGBD dense point cloud data. It processes the LiDAR point cloud data and IMU data using an odometry method, outputs odometry information with the same frequency as the IMU, obtains the transformation matrix of the aligned point cloud frames, and segments the RGB image using a deep neural network model to obtain a semantic segmentation image. The output odometry frequency must be at least greater than or equal to the point cloud keyframe frequency inserted into the 3D raster map, and carry time information or the corresponding point cloud frame sequence information. The projection alignment module is used to project the laser point cloud onto the semantic segmentation image according to the camera's intrinsic and extrinsic parameters, so that the laser point cloud can obtain semantic information on the semantic segmentation image, and align the point cloud with the semantic information with the 3D grid map according to the transformation matrix to obtain a 3D grid map with semantic information. The dimensionality reduction and fusion module is used to perform dimensionality reduction processing on the 3D raster sub-map with semantic information in the Z direction, fusing the occupancy probability information, semantic information, and RGBD dense point cloud information in the Z-direction raster into occupancy probability information, semantic information, and elevation information of the 2.5D raster, thus obtaining a 2.5D raster sub-map; wherein, fusing the occupancy probability information, semantic information, and RGBD dense point cloud information in the Z-direction raster into occupancy probability information, semantic information, and elevation information of the 2.5D raster includes: Remove grid cells from the 3D grid that are semantically passable areas and contain moving objects, and then calculate the occupancy probability value of the 2.5D grid by applying a Gaussian function for height weighting. The specific calculation formula is as follows: Where μ is the expectation of the Gaussian function, and w(h) is the Gaussian function. The weight of grid i, This represents the occupancy probability value of a 2.5D grid. Semantic information from the 3D raster is accumulated in the Z direction to obtain semantic information from the 2.5D raster. The elevation information of the 2.5D grid is obtained by weighting the heights of 3D grids whose occupancy probability exceeds a threshold by calculating the weights of the Gaussian distribution of heights of dense point clouds fused from multiple frames of RGBD within the grid strip range. The stitching module is updated to adjust the keyframe pose of the 2.5D raster sub-map when the odometer corrects the accumulated error, and stitches the updated sub-map to obtain the final 2.5D multi-level map.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1 to 6 when it is run.

Images