Semantic navigation map construction method and device based on depth camera and storage medium

Abstract

The invention provides a semantic navigation map construction method and device based on a depth camera and a storage medium. The method comprises the following steps: acquiring a depth image and image data of an environment by using a depth camera, and generating point cloud data; processing the point cloud data by using a three-dimensional object detection model so as to identify different types of objects in the environment and generate point cloud data with semantic tags; converting the point cloud data with the semantic tag into single-line laser data with the semantic tag, and inserting the single-line laser data with the semantic tag into a local semantic sub-map; when the local semantic sub-maps meet preset conditions, the local semantic sub-maps are inserted into a sub-map queue, and the local semantic sub-maps are spliced into a global map; in the map generation process, the loop detection technology is used for recognizing the previous scene passed by the robot to correct the pose of the robot. According to the method, the map construction cost and computing resources are reduced, and the real-time performance of map construction and the adaptability of a dynamic environment are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and storage medium for constructing a semantic navigation map based on a depth camera. Background Art

[0002] With the development of computer technology, indoor service robots have become a research focus in order to liberate labor and improve the quality of life. These robots need to achieve autonomous navigation and task execution in complex environments, and usually rely on accurate maps for positioning and path planning. At present, service robots on the market generally use lidar and cameras as the main sensors. Lidar is widely used in map construction and can provide high-precision environmental information. However, the high price of lidar and its high dependence on computing resources have led to high overall costs for indoor service robots, limiting their popularity and application.

[0003] In the existing technology, although LiDAR can provide high-precision environmental modeling, its high cost and consumption of computing resources are issues that cannot be ignored. In addition, LiDAR also has shortcomings in dynamic environments, and it is difficult to update and process dynamic obstacles in real time. Therefore, the existing technology has obvious shortcomings in reducing system costs, improving real-time performance, and adapting to dynamic environments.

[0004] In order to improve the accessibility and popularity of indoor service robots, it is necessary to explore more economical sensors and algorithms to reduce system costs while ensuring performance, and to effectively process obstacle information in dynamic environments to ensure real-time updates and high accuracy of maps. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, device and storage medium for constructing a semantic navigation map based on a depth camera to solve the problems of high map construction cost, large computing resource consumption, poor real-time performance and adaptability to dynamic environments in the prior art.

[0006] According to a first aspect of an embodiment of the present application, a method for constructing a semantic navigation map based on a depth camera is provided, comprising: using a depth camera to collect depth images and image data of an environment, and generating point cloud data based on the depth images and image data; using a three-dimensional object detection model to process the point cloud data to identify different types of objects in the environment and generate point cloud data with semantic labels; converting the point cloud data with semantic labels into single-line laser data with semantic labels, and inserting the single-line laser data with semantic labels into a local semantic sub-map; when the local semantic sub-map meets a predetermined condition, inserting the local semantic sub-map into a sub-map queue, and splicing the local semantic sub-maps into a global map; in the process of map generation, using loop detection technology to correct the robot's posture by identifying previous scenes passed by the robot, and applying the corrected posture to the local semantic sub-map and the global map to obtain a final semantic navigation map.

[0007] According to a second aspect of an embodiment of the present application, a semantic navigation map construction device based on a depth camera is provided, comprising: an acquisition module, configured to acquire depth images and image data of an environment using a depth camera, and generate point cloud data based on the depth images and image data; an identification module, configured to process the point cloud data using a three-dimensional object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels; a conversion module, configured to convert the point cloud data with semantic labels into single-line laser data with semantic labels, and insert the single-line laser data with semantic labels into a local semantic sub-map; a splicing module, configured to insert the local semantic sub-map into a sub-map queue when a predetermined condition is met in the local semantic sub-map, and splice the local semantic sub-map into a global map; a correction module, configured to correct the robot's posture by identifying previous scenes passed by the robot using loop detection technology during map generation, and apply the corrected posture to the local semantic sub-map and the global map to obtain a final semantic navigation map.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0009] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0011] By using a depth camera to collect the depth image and image data of the environment, and generating point cloud data based on the depth image and image data; using a three-dimensional object detection model to process the point cloud data to identify different types of objects in the environment and generate point cloud data with semantic labels; converting the point cloud data with semantic labels into single-line laser data with semantic labels, and inserting the single-line laser data with semantic labels into a local semantic submap; when the local semantic submap meets the predetermined conditions, inserting the local semantic submap into the submap queue, and splicing the local semantic submap into a global map; in the process of map generation, using loop detection technology to correct the robot's posture by identifying the previous scenes the robot has passed through, and applying the corrected posture to the local semantic submap and the global map to obtain the final semantic navigation map. This application reduces the cost of map construction and computing resources, and improves the real-time nature and adaptability of map construction to dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 It is a schematic diagram of the overall implementation process of the semantic navigation map construction method based on the depth camera provided in the embodiment of the present application;

[0014] Figure 2 It is a flowchart of a method for constructing a semantic navigation map based on a depth camera provided in an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of converting a depth map into single-line laser parameters provided by an embodiment of the present application;

[0016] Figure 4 It is a schematic diagram of the range of converting a depth map into a single-line laser provided in an embodiment of the present application;

[0017] Figure 5 It is a schematic diagram of converting depth data within a sampling area of ​​a frame of depth image provided by an embodiment of the present application into laser data;

[0018] Figure 6 It is a schematic diagram of the route trajectory of the loop key frame feature point matching result in the circular corridor positioning experiment scenario provided in the embodiment of the present application;

[0019] Figure 7It is a schematic diagram of a loop mapping effect provided in an embodiment of the present application and a map before and after loop adjustment;

[0020] Figure 8 It is a structural schematic diagram of a semantic navigation map construction device based on a depth camera provided in an embodiment of the present application;

[0021] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0023] With the development of computer technology, indoor service robots have become one of the key research areas in order to reduce human labor intensity and improve the quality of life. These robots perform various tasks in indoor environments such as homes and offices, such as cleaning, delivering goods, and accompanying. In order to achieve autonomous navigation and task execution in complex indoor environments, indoor service robots need to rely on accurate maps for positioning and path planning. An accurate map can help robots identify obstacles and navigation paths in the environment, so as to effectively complete tasks.

[0024] Indoor service robots currently on the market generally use LiDAR and cameras as their main sensors. LiDAR is widely used in map construction and positioning navigation due to its high precision and reliability. LiDAR can accurately measure the distance and shape in the environment by emitting lasers and receiving reflected signals, generating high-precision environmental maps.

[0025] However, current map-building solutions that use lidar and cameras as sensors still have the following disadvantages: Lidar equipment is expensive and usually accounts for a large part of the overall cost of the robot. The amount of data generated by lidar is large and requires powerful computing resources to process and analyze, which further increases the cost and complexity of the system.

[0026] Due to the high cost of LiDAR and its reliance on computing resources in existing technologies, the overall cost of indoor service robots is high, which limits their popularity and widespread application. In order to improve the accessibility and popularity of indoor service robots, it is necessary to explore more economical sensors and algorithms to reduce system costs while ensuring performance.

[0027] In view of the problems existing in the prior art, this application provides a method for constructing a semantic grid navigation map using a depth camera. This application aims to study a method for constructing a semantic navigation map for an indoor robot. Since the cost of a camera is much lower than that of a laser sensor, using a camera as the main positioning and mapping sensor can help significantly reduce system costs.

[0028] The following first summarizes the overall implementation process of the semantic navigation map construction method involved in the technical solution of the present application in conjunction with the accompanying drawings. Figure 1 Schematic diagram of the overall implementation process of the semantic navigation map construction method based on the depth camera provided in the embodiment of the present application. Figure 1 As shown, the overall implementation process of the semantic navigation map construction method based on the depth camera may include:

[0029] This application proposes a method for building a semantic navigation map through a depth camera. It first introduces how to convert a depth map into single-line laser data, and then introduces a probabilistic grid map, which has the function of removing dynamic obstacles. Then the concept of sub-maps is introduced, including global maps, and the use of loop closure to correct posture is proposed to build a globally consistent navigation map. Then the indoor 3D object recognition network FCAF3D is introduced, the point cloud is input to obtain a 3D detection frame, the single-line laser in the detection frame is converted into a single-line laser with semantic information, and each grid point is assigned the matching single-line laser semantic data.

[0030] The contents of the technical solution of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Figure 2 It is a flow chart of a method for constructing a semantic navigation map based on a depth camera provided in an embodiment of the present application. Figure 2 The depth camera-based semantic navigation map construction method can be executed by a chip installed on the robot. Figure 2 As shown, the method for constructing a semantic navigation map based on a depth camera may specifically include:

[0032] S201, using a depth camera to collect a depth image and image data of an environment, and generating point cloud data according to the depth image and the image data;

[0033] S202, processing the point cloud data using a three-dimensional object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels;

[0034] S203, converting the point cloud data with semantic labels into single-line laser data with semantic labels, and inserting the single-line laser data with semantic labels into the local semantic sub-map;

[0035] S204, when the local semantic sub-map meets the predetermined condition, the local semantic sub-map is inserted into the sub-map queue, and the local semantic sub-map is spliced ​​into the global map;

[0036] S205, during the map generation process, the robot's posture is corrected by identifying previous scenes that the robot has passed through using loop detection technology, and the corrected posture is applied to the local semantic sub-map and the global map to obtain the final semantic navigation map.

[0037] In some embodiments, processing the point cloud data using the 3D object detection model includes:

[0038] Input the point cloud data into a predetermined 3D object detection model, and use a fully convolutional feature pyramid network in the 3D object detection model to perform object detection based on multi-level features;

[0039] The point cloud data is converted into a regular voxel grid by voxelization, and features are extracted from the voxel grid using convolution operations;

[0040] An anchor-free approach is used to detect objects by predicting their center points and bounding box parameters, and outputting three-dimensional object boxes corresponding to different types of objects.

[0041] Specifically, the point cloud data collected by the depth camera is first input into a predetermined 3D object detection model. The 3D object detection model adopts the FCAF (Fully Convolutional Anchor-free 3D Object Detection) method, which performs multi-level feature extraction through a fully convolutional feature pyramid network (FPN), significantly improving the effect of object detection.

[0042] Further, the point cloud data is voxelized and converted into a regular voxel grid. The voxelization process converts the continuous 3D point cloud data into a discrete voxel grid, so that the subsequent convolution operation can perform feature extraction more efficiently. After voxelization, multi-level features are extracted from the voxel grid using 3D convolution operations. These features are processed in a multi-level structure of a feature pyramid network (FPN) to capture feature information at different scales and levels.

[0043] Furthermore, unlike traditional 3D object detection methods that rely on predefined anchor points, the FCAF3D method uses an anchor-free detection method. Detection is performed by predicting the center point and bounding box parameters of the object, avoiding the computational complexity and parameter tuning problems caused by predefining a large number of anchor points. Specifically, after feature extraction, the FCAF3D method directly predicts the center point coordinates and bounding box parameters of the object through a convolutional neural network. These prediction results are post-processed to output the 3D object boxes corresponding to different types of objects.

[0044] In one example, the collected 3D point cloud data is input into the FCAF3D 3D object detection model. The point cloud data is voxelized and converted into a regular voxel grid. Features are extracted from the voxel grid through a 3D convolution operation, and multi-level feature processing is performed using a fully convolutional feature pyramid network (FPN). An anchor-free detection method is used to predict the center point and bounding box parameters of the object through a convolutional neural network. The predicted center point and bounding box parameters are post-processed to output the 3D object boxes corresponding to different types of objects.

[0045] Through the method of the above-mentioned embodiment, the present application can efficiently and accurately identify different types of objects in the environment, generate three-dimensional object frames with semantic labels, and provide high-precision environmental perception capabilities for indoor service robots.

[0046] It should be noted that this application uses the capabilities of RGBD cameras to restore high-precision 3D point cloud data through depth images, thereby achieving accurate recognition of different types of objects in the environment. RGBD cameras not only provide color image information, but also provide distance information at each pixel to generate accurate 3D point clouds. This point cloud data can accurately locate and describe the shape, size and position of objects, providing important scene understanding capabilities for autonomous driving, robotics and augmented reality applications.

[0047] Different types of objects identified are labeled accordingly, and these labels can accurately reflect the category information of the objects. By converting labeled 3D point cloud data into labeled single-line laser data, the data processing process is further simplified while retaining the key semantic information of the objects. Single-line laser data has good spatiotemporal resolution and is suitable for environmental mapping and scene analysis.

[0048] In the process of mapping, single-line laser data with semantic information is introduced to assign corresponding semantic labels to each grid. This method can not only improve the accuracy and efficiency of mapping, but also provide important data support for subsequent intelligent decisions such as path planning and obstacle avoidance. At the same time, dynamic objects such as pedestrians can be deducted from the point cloud to remove the influence of dynamic objects in the mapping process.

[0049] A reasonable 3D object detection algorithm is crucial. Point cloud-based 3D object detection aims to simultaneously locate and identify 3D objects in a given 3D point set. As a core technology for 3D scene understanding, it is widely used in autonomous driving, robotics, and AR. While 2D methods are suitable for dense fixed-size arrays, 3D methods are challenged by irregular and unstructured 3D data of arbitrary volumes. Therefore, 2D data processing techniques cannot be directly applied to 3D object detection, so 3D object detection methods adopt creative methods for 3D data processing. Convolutional 3D object detection methods have scalability issues: large-scale scenes either require an unrealistic amount of computing resources or take too much time to process. Other methods choose voxel data representation and adopt sparse convolution; however, these methods solve the scalability problem at the expense of detection accuracy. These object detection methods can provide accurate estimates and good scaling. In addition to being scalable and accurate, ideal 3D object detection methods should also handle objects of arbitrary shapes and sizes without additional modifications and manually tuned hyperparameters.

[0050] In some embodiments, the three-dimensional object detection model includes a backbone network, a neck network, and a head network. In the three-dimensional object detection model, a fully convolutional feature pyramid network is used to perform object detection based on multi-level features, including:

[0051] The high-dimensional sparse residual network structure is used in the backbone network to perform sparse modification on the residual network, and sparse 3D convolution operations are used to process irregular and unstructured 3D point cloud data.

[0052] In the neck network, based on a simplified universal sparse decoder, sparse transposed 3D convolution and 3D convolution are used to process each feature level, and a pruning layer is used to filter the input data through a probabilistic mask;

[0053] The head network consists of three parallel sparse convolutional layers, where the sparse convolutional layers share weights between feature levels and each position generates classification probability, bounding box regression parameters and centerness at output.

[0054] Specifically, the collected 3D point cloud data is input into the 3D object detection model. The input data includes 3D space coordinates and related depth information. The backbone network adopts a high-dimensional sparse residual network structure, namely HDResNet, which is a sparse modification of the traditional ResNet.

[0055] Furthermore, all 2D convolution operations are replaced by sparse 3D convolution operations, which can process irregular and unstructured 3D point cloud data. This network structure is mainly designed for processing 3D data and can effectively utilize spatial information and local features.

[0056] Furthermore, the neck network uses a simplified GSDN (Generalized Sparse Decoder Network) decoder. Each feature level is processed by sparse transposed 3D convolution and 3D convolution. The kernel size of the transposed sparse 3D convolution operation is 2, which increases the number of non-zero values ​​while maintaining network efficiency. In order to control memory usage, GSDN uses pruning layers to filter input data through probabilistic masks.

[0057] Furthermore, the head design of FCAF3D is simple and lightweight, consisting of three parallel sparse convolutional layers that share weights between feature levels. Each position (x, y, z) generates a classification probability p, a bounding box regression parameter δ, and a center c at output. This design is similar to the head structure of FCOS (Fully Convolutional One-Stage Object Detection), but adapts to the characteristics of 3D data.

[0058] In one example, the processing of a 3D object detection model is as follows:

[0059] Point cloud data input: Input the collected 3D point cloud data into the FCAF3D 3D object detection model.

[0060] Backbone network: Use the HDResNet structure to process the input 3D point cloud data and extract preliminary features.

[0061] Neck network: A simplified GSDN decoder is used to perform sparse transposed 3D convolution and 3D convolution on each feature level, and a pruning layer is used to filter out unimportant data.

[0062] Head network: The extracted features are processed through three parallel sparse convolutional layers to generate classification probabilities, bounding box regression parameters, and centerness at each position.

[0063] Anchor-free detection method: Detection is performed by predicting the center point and bounding box parameters of the object, avoiding the computational complexity and parameter tuning problems of the traditional anchor method.

[0064] Result output: The model outputs three-dimensional object boxes corresponding to different types of objects.

[0065] Through the method of the present embodiment described above, using the FCAF3D method, three-dimensional point cloud data can be processed efficiently and accurately, different types of objects in the environment can be identified, and three-dimensional object frames with semantic labels can be generated, providing indoor service robots with high-precision environmental perception capabilities.

[0066] In some embodiments, converting point cloud data with semantic labels into single-line laser data with semantic labels includes:

[0067] Select the sampling height range of the depth image and determine which row of the image to start with as the laser data;

[0068] Calculate the angle between each pixel and the central ray based on the camera's internal parameters, and calculate the angle range for converting the depth image into single-line laser data;

[0069] Determine the angle increment between two adjacent laser beams, convert the depth value corresponding to each pixel into a laser detection distance value, and map the depth value into a one-dimensional array based on the calculated angle and angle increment to generate single-line laser data;

[0070] According to the image pixel coordinates and camera intrinsic parameters, the corresponding coordinates of each pixel point in the camera coordinate system are calculated, and the distance of the laser detection point is calculated;

[0071] By traversing the sampling area of ​​the depth image, all valid depth data are converted into single-line laser data;

[0072] In the conversion process, the semantic label of the identified object is assigned to the corresponding single-line laser data to maintain the semantic information of the data.

[0073] Specifically, first, the sampling height range (scan_height) of the depth image is selected, and the row of the image from which to start as the starting row of the laser data (offset) is determined.

[0074] Furthermore, the angle between each pixel and the central ray is calculated based on the camera internal parameters (including the focal length and optical axis offset of the camera). The angle on the left side of the middle of the image is positive, and the angle on the right side is negative.

[0075] Furthermore, the angle range of the depth image converted into single-line laser data is calculated through the camera internal parameters, including the angle between the leftmost laser beam and the center laser and the angle between the rightmost laser beam and the center laser. Based on these angles, the angle increment between two adjacent laser beams is determined.

[0076] Furthermore, the depth value corresponding to each pixel is converted into a laser detection distance value. According to the calculated angle and angle increment value, these depth values ​​are mapped into a one-dimensional array to generate single-line laser data.

[0077] Furthermore, according to the image pixel coordinates and the camera intrinsic parameters, the corresponding coordinates of each pixel point in the camera coordinate system are calculated, and the distance of the laser detection point is calculated.

[0078] Furthermore, by traversing the sampling area of ​​the depth image, all valid depth data are converted into single-line laser data.

[0079] Furthermore, during the conversion process, the semantic labels of the identified objects are assigned to the corresponding single-line laser data to preserve the semantic information of the data. These semantic labels can help provide more environmental information in subsequent map construction and navigation.

[0080] In an example, the following is a detailed process to implement the above steps:

[0081] Select the range of image rows in the depth image that can be used as laser data, and determine the starting row offset. For example, starting from the offset+1 row, the depth value of a certain range of "flattened" on the image is used as the laser spot distance.

[0082] According to the image pixel coordinates and the camera internal parameters, the angle between each pixel and the central ray is calculated. The angle on the left side of the center of the image is positive, and the angle on the right side is negative. These angles are used to determine the laser beam corresponding to each pixel.

[0083] Calculate the angle range of the depth image converted to single-line laser data, that is, the angle between the leftmost laser beam and the center laser and the angle between the rightmost laser beam and the center laser. Then, calculate the angle increment between two adjacent laser beams, and use these angle ranges and increments to map the depth value to the laser data.

[0084] The depth value of each pixel is converted into a laser detection distance value, and these depth values ​​are mapped into a one-dimensional array according to the calculated angle and angle increment value to generate single-line laser data.

[0085] Using the camera intrinsic parameters, the corresponding coordinates of each pixel in the camera coordinate system are calculated. For example, if the depth value and the camera intrinsic parameters are known, the coordinates of the point in the camera coordinate system can be calculated, and the distance of the laser detection point can be further calculated.

[0086] Traverse the sampling area of ​​the depth image, convert all valid depth data into single-line laser data, and generate a frame of simulated lidar scanning data within the sampling area of ​​the depth image.

[0087] During the conversion process, the semantic labels of the identified objects are assigned to the corresponding single-line laser data to ensure that each laser data point contains the corresponding semantic information.

[0088] The following is a detailed description of the overall implementation process of converting a depth map into a single-line laser point cloud in this application, with reference to a specific example, which may include the following contents:

[0089] A depth map is a grayscale image in which each pixel corresponds to a depth value representing the distance from the camera to a point in the scene. Depth maps can be generated by a depth sensor or a monocular camera combined with a depth estimation algorithm to represent the three-dimensional structure of the environment.

[0090] The parameters used in this embodiment have their meanings shown in Table 1, where row, col, f x 、f y 、c y 、c x It is the camera internal parameter, provided by the camera manufacturer.

[0091] In Table 1:

[0092]

[0093] Figure 3 FIG. 1 is a schematic diagram of converting a depth map into single-line laser parameters provided by an embodiment of the present application. Figure 3 As shown, the angle range of the depth map converted into a single-line laser can be calculated through the camera internal parameters:

[0094]

[0095] The angle between two adjacent laser beams:

[0096]

[0097] Table 1 Depth map to laser parameters

[0098]

[0099]

[0100] The depth map is converted into single-line laser data by selecting a certain range of "flattened" depth values ​​on the image as the laser spot beam distance. Figure 4 Schematic diagram of the range of converting a depth map into a single-line laser provided by an embodiment of the present application. Figure 4 As shown in the figure, scan height is the number of image rows that can be used as laser data, and offset indicates that the laser data starts from the offset+1th row.

[0101] The angle θ between each pixel and the central ray is calculated based on the image pixel coordinates and the camera internal parameters, as follows: Figure 4 As shown, the angles to the left of the center of the image are positive and to the right are negative:

[0102]

[0103] Figure 5FIG. 1 is a schematic diagram of converting depth data in a sampling area of ​​a depth image frame provided by an embodiment of the present application into laser data. Figure 5 As shown, the optical center O is the origin of the laser coordinate system, which coincides with the origin of the camera coordinate system. The red thick line is a set of laser points. p is a point on the depth image. Given the depth and the camera internal parameters, the coordinates P (X, Y) corresponding to the point in the camera coordinate system can be calculated according to formula (2-6). Then, the distance between the two points OP can be calculated according to formula (2-7):

[0104]

[0105] OP corresponds to a laser beam emitted by the laser radar, O is the starting point, and P is the ending point. Its serial number index can be calculated according to formula (2-5). By traversing, the laser radar scanning data simulated by the sampling area of ​​a frame of depth image can be obtained.

[0106] Through the above embodiments, the present invention can efficiently convert point cloud data with semantic labels into single-line laser data with semantic labels, providing accurate environmental information and semantic recognition capabilities for map construction and navigation of indoor service robots.

[0107] In some embodiments, during the map generation process, the robot's posture is corrected by identifying previous scenes that the robot has passed through using loop detection technology, and the corrected posture is applied to the local semantic sub-map and the global map, including:

[0108] Introducing loop detection technology in the map generation process to identify whether the robot has passed through a scene that it has visited before, and measuring similar images to identify loops;

[0109] When the robot reenters a previously visited scene, constraints are imposed on the robot’s current position to correct the accumulated error;

[0110] The pose corresponding to the first frame of the point cloud inserted into the submap is used as the initial pose of the submap, and the poses of all historical key frames between the loop key frames are corrected, where the historical key frames include the old loop key frames and the new loop key frames;

[0111] Update and correct the position of the submap, and insert the submap into the global map according to the corrected submap position.

[0112] Specifically, in the map generation process, loop detection technology is introduced to identify whether the robot has passed through a scene that it has visited before. Loop detection compares the image currently observed by the robot with the image in the previous scene and measures their similarity to identify loops.

[0113] When the robot reenters a previously visited scene, loop closure detection imposes constraints on the robot’s current position. These constraints are used to correct the posture deviation caused by long-term accumulated errors and ensure the robot’s positioning accuracy.

[0114] The pose corresponding to the first frame of point cloud inserted into the submap is used as the initial pose of the submap. Each submap has its corresponding initial pose when it is generated, and this pose is determined by the first frame of point cloud data.

[0115] The loop detection technology corrects the poses of all historical keyframes between loop keyframes. These historical keyframes include old loop keyframes and new loop keyframes. By correcting the poses of historical keyframes, the offset caused by the accumulated error in the map can be corrected.

[0116] According to the results of loop detection, the sub-map position is updated and corrected. The corrected sub-map position ensures the accurate position of each sub-map in the global map.

[0117] According to the corrected submap position, the submap is inserted into the global map. The global map is composed of multiple submaps. By correcting the submap position, the accuracy and consistency of the global map can be ensured.

[0118] To solve the problem of long-term error accumulation, loop detection technology is introduced. When the robot re-enters a scene that it has visited before, loop detection eliminates the accumulated error by identifying whether the robot has passed through the previous scene and measuring similar images. Loop detection imposes constraints on its current position, thereby correcting the accumulated error to the correct position.

[0119] Loop detection can not only correct errors in long-term positioning, but also achieve repositioning. By associating with historical data, the robot can accurately locate in the explored scenes. As the system runs, many sub-maps will be generated, and each sub-map corresponds to a pose. In the program, the pose corresponding to the first frame of the point cloud inserted into the sub-map is defined as the pose of the sub-map. Since loop detection will correct all historical key frame poses between the old loop key frame and the new loop key frame, the pose of the sub-map within this range will also be corrected. The global map is formed by splicing sub-maps. Therefore, by changing the position of the sub-map inserted into the global map according to the new sub-map pose, the effect of re-correcting the global map due to loop detection can be achieved.

[0120] Through the method of the above embodiment, the present application can use loop detection technology to correct the robot's posture during the map generation process, ensure the consistency and accuracy of the local semantic sub-map and the global map, thereby improving the accuracy of robot navigation and positioning.

[0121] It should be noted that a submap is a map of a local area, which contains a fixed number of point cloud frames. When the number of inserted point cloud frames reaches a predetermined value, the submap is considered complete and no new point cloud frames are received. New submap creation: When a new point cloud frame enters the system, if the current submap is full, the system will create a new blank submap and insert subsequent point cloud frames into it, and repeat this process. The global map is formed by splicing multiple submaps. As point cloud data continues to be input, the number of submaps generated by the system increases, and the global map will expand accordingly.

[0122] When new point cloud frames are inserted, if the boundary point clouds of these frames exceed the scope of the current submap, the system will adjust the size of the submap to include these new data, thereby achieving incremental updates. For positioning and mapping based on visual odometry, short-term positioning is performed using IMU integral constraints and visual observations within a sliding window. Because these methods rely on short-term constraints, errors will gradually accumulate over time, resulting in a decrease in the accuracy of long-term positioning and mapping. To solve the problem of long-term error accumulation, loop detection technology is introduced. Loop detection eliminates accumulated errors by identifying whether the robot has passed through previous scenes and measuring similar images. When the robot re-enters a previously visited scene, loop detection imposes constraints on its current position, thereby correcting the accumulated errors to the correct position.

[0123] Loop detection can not only correct errors in long-term positioning, but also achieve re-positioning. By associating with historical data, the robot can accurately position itself in scenes that have been explored.

[0124] As the system runs, many submaps will be generated, and each submap corresponds to a posture. In the program, the posture corresponding to the first frame of the point cloud inserted into the submap is set as the posture of the submap. Since the loop will correct all the historical keyframe postures between the old loop keyframe and the new loop keyframe, the posture of the submap within this range will also be corrected. The global map is formed by splicing submaps, so the position of the submap inserted into the global map is changed according to the new submap posture, so the effect of re-correcting the global map due to the loop effect can be achieved.

[0125] To verify the loop effect, the main body of the experimental scene is a ring structure consisting of four long corridors. Figure 6 Schematic diagram of the path trajectory of the loop key frame feature point matching result in the circular corridor positioning experiment scenario provided by the embodiment of the present application. Figure 6As shown in the figure, the red line is the positioning trajectory output by the visual inertial odometer (VIO), the blue line is the positioning trajectory after adjustment by the loop module, and the blue dot is the starting point of the robot's movement, moving in a counterclockwise direction. When the robot goes around a circle and returns to the vicinity of the starting position, multiple loops occur. The loop module optimizes the pose graph of all historical key frames and outputs the corrected positioning trajectory. According to statistics, the robot has a total of 13 loops after returning to the vicinity of the starting point. After actual measurement, the VIO positioning result error is 2.71m, which is a large error, while the positioning error after the loop is 0.09m.

[0126] in addition, Figure 7 This is a schematic diagram of the loop mapping effect provided by the embodiment of the present application, comparing the map before loop adjustment and the map after loop adjustment. Figure 7 As shown in the figure, the sub-map-based mapping effect in the circular corridor scene is shown. Figure 7 From the diagram on the left, we can see that the map that should have overlapped has no loop optimization. The accumulated positioning errors after a long period of operation result in the corridors that should have overlapped not overlapping when returning to the vicinity of the starting point. Figure 7 The diagram on the right is the adjusted map, where the green line is the positioning trajectory output by VIO, and the red line is the trajectory formed by connecting the key frame poses output after loop adjustment. The pose of the sub-map is further adjusted using the loop optimization positioning effect, and the global map will also be adjusted accordingly. From the figure, it can be seen that the map after loop adjustment has been greatly improved compared to the map before adjustment, and is more in line with the actual scene.

[0127] It should be noted that the construction of the map depends on the positioning information. The error in the positioning result will cause the map to not match the real environment. Using a map that does not conform to the actual situation for navigation will cause the obstacles to be misplaced on the navigation map, which may further cause the robot to collide with obstacles.

[0128] In some embodiments, the method further comprises:

[0129] In the map generation process, the environment is divided into several small grids for mapping, where each grid represents a local area and stores the probability information of whether there are obstacles in the local area;

[0130] Set the size and resolution of the grid so that each grid corresponds to a fixed area in the real environment, and assign a probability value to each grid, which is used to represent the probability of the existence of an obstacle in the grid;

[0131] When the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. As time goes by and the observation is continuously updated, the probability value gradually decreases and returns to the obstacle-free state.

[0132] Among them, according to different probability values, the state of the grid is divided into obstacle, obstacle-free and unexplored.

[0133] Specifically, during the map generation process, the entire environment is divided into several small grids for mapping. Each grid represents a local area and stores the probability information of whether there are obstacles in the local area.

[0134] Furthermore, the size and resolution of the grid are set so that each grid corresponds to a fixed area in the real environment. Assuming the resolution is set to 0.05 meters, each grid corresponds to a 5cm×5cm area in the real environment. This high-resolution grid setting allows for detailed environment modeling, allowing the robot to accurately perceive its surroundings.

[0135] Furthermore, a probability value is assigned to each grid, indicating the probability of an obstacle existing in the grid. According to the different probability values, the grid status is divided into three types:

[0136] There is an obstacle: the probability value is close to 1.

[0137] No obstacles: The probability value is close to 0.

[0138] Unexplored: The probability value is in the middle area.

[0139] Furthermore, when the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. As time goes by and the observation is continuously updated, these probability values ​​will gradually decrease and return to the obstacle-free state. This dynamic probability update mechanism makes the probability grid map resistant to dynamic obstacles and can effectively handle dynamic changes in the environment.

[0140] In one example, a probabilistic grid map (PGM) is a technology used to represent a robot's environment by dividing the entire environment into several small grids. Each grid represents a local area and stores the probability information of whether there is an obstacle in the area. Assuming the resolution is set to 0.05 meters, each grid corresponds to a 5cm×5cm area in the real environment. This resolution allows for detailed environment modeling, allowing the robot to accurately perceive its surroundings.

[0141] Each grid stores a probability value, which indicates the probability of an obstacle existing in the grid. Depending on the probability value, the grid can be divided into three states:

[0142] There is an obstacle: The probability value is close to 1, indicating that there is an obstacle in the area.

[0143] No obstacles: The probability value is close to 0, indicating that there are no obstacles in the area.

[0144] Unexplored: The probability value is in the middle range, indicating that the area has not been explored yet.

[0145] When the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. However, as time goes by and the observation is continuously updated, these probability values ​​will gradually decrease and return to the obstacle-free state. Therefore, PGM has the ability to resist interference from dynamic obstacles, remove their influence, and avoid mistaking dynamic changes for static obstacles.

[0146] Through the method of the present embodiment described above, the present application can effectively use the probabilistic grid map for environmental modeling and dynamic obstacle processing during the map generation process, thereby ensuring the robot's environmental perception capability and map accuracy.

[0147] Probability update and dynamic obstacle handling: Dynamic obstacle impact: When the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase, but as time goes by and the observation is continuously updated, these probability values ​​will gradually decrease and return to the obstacle-free state. Therefore, PGM has the ability to resist interference from dynamic obstacles, which can remove their impact and avoid mistaking dynamic changes for static obstacles.

[0148] OpenLORIS dataset experiment: Based on the performance of the probability grid map in the OpenLORIS dataset. White grids indicate no obstacles, black grids indicate obstacles. The red curve indicates the obstacle observation of the sensor at the current moment, and the pink dotted box indicates the observation of dynamic obstacles.

[0149] No obstacle: When the sensor does not detect a person, the grid is white (no obstacle).

[0150] When a dynamic obstacle appears: When a dynamic obstacle (such as a person) enters the sensor's observation range, the sensor's observation values ​​of these grids turn red. However, after the person leaves the sensor's observation range, the state of these grids does not change to occupied state and remains white. This shows that PGM can maintain the static characteristics of the environment and has a strong resistance to interference from dynamic obstacles.

[0151] Static environment mapping: In a static environment, multiple observations of obstacles by sensors will gradually update the relevant grid status to black (occupied status). This cumulative probability update mechanism effectively improves the stability and accuracy of the map.

[0152] The probabilistic grid map effectively handles the impact of dynamic obstacles on the static map through a flexible probabilistic update mechanism. This map shows strong anti-interference ability and adaptability, providing reliable support for the robot's positioning and mapping in a dynamic environment. The experimental results verify the effectiveness and reliability of the probabilistic grid map.

[0153] In some embodiments, the state of the grid is divided into obstacle-existing, obstacle-free, and unexplored according to different probability values, including:

[0154] When the sensor does not detect an obstacle, the state of the grid is updated to no obstacle; when a dynamic obstacle enters the observation range of the sensor, the observation value of the grid is updated to the obstacle observation value of the sensor at the current moment; when the dynamic obstacle leaves the observation range of the sensor, the state of the grid is re-marked as no obstacle; in a static environment, when the sensor observes an obstacle multiple times, the state of the relevant grid is gradually updated to have an obstacle.

[0155] Specifically, in the initial stage of map generation, the states of all grids are set to unexplored states, that is, the probability values ​​of the grids are in the middle area.

[0156] Furthermore, the state update when there is no obstacle: when the sensor does not detect an obstacle, the state of the grid is updated to no obstacle, indicating that there is no obstacle in the grid area. For example, when the robot moves in an empty room and does not detect any obstacle, the grid state is updated to no obstacle.

[0157] Furthermore, the handling of dynamic obstacles: When a dynamic obstacle (such as a person) enters the observation range of the sensor, the sensor's observation value of these grids will be updated to the obstacle observation value of the sensor at the current moment. The state of the grid is temporarily updated to the dynamic obstacle state, that is, the probability value increases. Specifically, when a person enters the observation range of the robot sensor, the state of these grids will turn red, indicating that there is an obstacle currently.

[0158] Furthermore, the state is restored after the dynamic obstacle leaves: when the dynamic obstacle leaves the observation range of the sensor, the state of the grid will not immediately change to the occupied state, but will be re-marked as the obstacle-free state, indicating that there are no obstacles in the area. In this way, PGM can maintain the static characteristics of the environment and has a strong resistance to the interference of dynamic obstacles.

[0159] Furthermore, state update in static environment: In a static environment, when the sensor observes obstacles multiple times, the state of the relevant grids is gradually updated to the state with obstacles. Specifically, when the robot observes obstacles in fixed positions (such as tables and walls) multiple times, the state of these grids will gradually turn black, indicating that there are obstacles in the area and the state is relatively stable.

[0160] For example, in a specific example, when there is no obstacle: when the sensor does not observe a person, the state of the grid is white (no obstacle).

[0161] When a dynamic obstacle appears: When a dynamic obstacle (such as a person) enters the sensor's observation range, the sensor's observation values ​​of these grids turn red. However, after the person leaves the sensor's observation range, the state of these grids does not change to occupied state and remains white. This shows that PGM can maintain the static characteristics of the environment and has a strong resistance to interference from dynamic obstacles.

[0162] Static environment mapping: In a static environment, multiple observations of obstacles by sensors will gradually update the relevant grid status to black (occupied status). This cumulative probability update mechanism effectively improves the stability and accuracy of the map.

[0163] Through the method of the present embodiment described above, the present application can dynamically update the state of the grid according to the observation information of the sensor during the map generation process, and effectively handle the impact of dynamic obstacles to ensure the accuracy and stability of the map.

[0164] According to the technical solution provided in the embodiments of the present application, the technical solution of the present application has at least the following advantages:

[0165] In the process of map construction, in order to effectively remove moving objects, the Probabilistic Grid Map (PGM) method is adopted. By real-time observation and updating of each grid, the influence of detected obstacles can be dynamically removed. This method effectively handles the interference of dynamic obstacles on static maps through a flexible probabilistic update mechanism, making the constructed map more reliable and practical. With this method, the system can maintain the accuracy of the map in a dynamic environment and provide a more stable navigation foundation.

[0166] In the process of constructing the semantic map, certain frequently moving objects, such as seats, can be selectively removed. By semantically identifying and filtering these objects, their interference with navigation path planning can be reduced. Specifically, by adjusting the map, the unnecessary impact of these common movable objects on the navigation path can be avoided, thereby generating a more reasonable and optimized navigation trajectory. This method not only simplifies the planning of the navigation path, but also reduces the number of path adjustments of the robot in complex environments, thereby improving the efficiency of navigation.

[0167] The adjusted semantic map not only improves the rationality of navigation, but also provides a navigation solution with more environmental awareness. In practical applications, the robot can navigate more efficiently in complex indoor environments and adapt to dynamically changing environmental characteristics. Through real-time updates and dynamic adjustments, the system can flexibly respond to various changes in the environment, improving the overall performance and application value of robot navigation. This enables the robot to avoid dynamic obstacles more accurately when performing tasks, improving the safety and reliability of navigation.

[0168] In summary, the technical solution of the present application significantly improves the reliability of map construction, the rationality of navigation paths, and the ability of environmental perception by adopting probabilistic grid maps and semantic recognition technology, providing an efficient and reliable navigation solution for indoor service robots.

[0169] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0170] Figure 8 Schematic diagram of the structure of a semantic navigation map construction device based on a depth camera provided in an embodiment of the present application. Figure 8 As shown, the semantic navigation map construction device based on the depth camera includes:

[0171] The acquisition module 801 is configured to acquire a depth image and image data of an environment using a depth camera, and generate point cloud data according to the depth image and image data;

[0172] The recognition module 802 is configured to process the point cloud data using a three-dimensional object detection model to recognize different types of objects in the environment and generate point cloud data with semantic labels;

[0173] A conversion module 803 is configured to convert the point cloud data with semantic labels into single-line laser data with semantic labels, and insert the single-line laser data with semantic labels into the local semantic sub-map;

[0174] The splicing module 804 is configured to insert the local semantic submap into the submap queue when the local semantic submap meets the predetermined condition, and splice the local semantic submap into the global map;

[0175] The correction module 805 is configured to correct the robot's posture by identifying previous scenes passed by the robot using loop detection technology during the map generation process, and apply the corrected posture to the local semantic sub-map and the global map to obtain the final semantic navigation map.

[0176] In some embodiments, Figure 8 The recognition module 802 inputs the point cloud data into a predetermined three-dimensional object detection model, and uses a fully convolutional feature pyramid network in the three-dimensional object detection model to perform object detection based on multi-level features; converts the point cloud data into a regular voxel grid through voxelization, and uses a convolution operation to extract features from the voxel grid; adopts an anchor-free method to detect by predicting the center point and bounding box parameters of the object, and outputs the three-dimensional object box corresponding to different types of objects.

[0177] In some embodiments, the 3D object detection model includes a backbone network, a neck network, and a head network. Figure 8 The recognition module 802 utilizes a high-dimensional sparse residual network structure in the backbone network, performs sparse modification on the residual network, and utilizes sparse three-dimensional convolution operations to process irregular, unstructured three-dimensional point cloud data; in the neck network, based on a simplified universal sparse decoder, sparse transposed three-dimensional convolution and three-dimensional convolution are utilized to process each feature level, and a pruning layer is utilized to filter the input data through a probability mask; the head network is composed of three parallel sparse convolution layers, wherein the sparse convolution layers share weights between feature levels, and each position generates classification probability, bounding box regression parameters, and centrality at output.

[0178] In some embodiments, Figure 8 The conversion module 803 selects the sampling height range of the depth image and determines which row of the image to start as the laser data; calculates the angle between each pixel and the central ray according to the camera intrinsic parameters, and calculates the angle range of the depth image converted into the single-line laser data; determines the angle increment value between two adjacent laser beams, converts the depth value corresponding to each pixel into a laser detection distance value, and maps the depth value into a one-dimensional array according to the calculated angle and angle increment value to generate single-line laser data; calculates the corresponding coordinates of each pixel in the camera coordinate system according to the image pixel coordinates and the camera intrinsic parameters, and calculates the distance of the laser detection point; by traversing the sampling area of ​​the depth image, all valid depth data are converted into single-line laser data; wherein, during the conversion process, the semantic label of the identified object is assigned to the corresponding single-line laser data to maintain the semantic information of the data.

[0179] In some embodiments, Figure 8 The correction module 805 introduces loop detection technology in the map generation process to identify whether the robot has passed through a previously visited scene and measures similar images to identify loops; when the robot re-enters a previously visited scene, constraints are imposed on the current position of the robot to correct the accumulated error; the pose corresponding to the first frame of the point cloud inserted into the sub-map is used as the initial pose of the sub-map, and the poses of all historical key frames between the loop key frames are corrected, wherein the historical key frames include old loop key frames and new loop key frames; the pose of the sub-map is updated and corrected, and the sub-map is inserted into the global map according to the corrected sub-map pose.

[0180] In some embodiments, Figure 8During the map generation process, the mapping module 806 divides the environment into several small grids for mapping, wherein each grid represents a local area and stores the probability information of whether there are obstacles in the local area; the size and resolution of the grid are set so that each grid corresponds to a fixed area in the real environment, and a probability value is assigned to each grid, which is used to represent the probability of the existence of obstacles in the grid; when the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase, and as time goes by and the observation is continuously updated, the probability value gradually decreases and returns to the obstacle-free state. According to the different probability values, the state of the grid is divided into obstacle-free, obstacle-free and unexplored.

[0181] In some embodiments, Figure 8 The mapping module 806 updates the state of the grid to no obstacle when the sensor does not detect an obstacle; when a dynamic obstacle enters the observation range of the sensor, the observation value of the grid is updated to the obstacle observation value of the sensor at the current moment; when the dynamic obstacle leaves the observation range of the sensor, the state of the grid is re-marked as no obstacle; in a static environment, when the sensor observes an obstacle multiple times, the state of the relevant grid is gradually updated to have an obstacle.

[0182] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0183] Fig. 9 Schematic diagram of the structure of the electronic device 9 provided in the embodiment of the present application. Fig. 9 As shown, the electronic device 9 of this embodiment includes: a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 901 executes the computer program 903, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0184] Exemplarily, the computer program 903 may be divided into one or more modules / units, which are stored in the memory 902 and executed by the processor 901 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 903 in the electronic device 9.

[0185] The electronic device 9 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 9 may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 9 and does not constitute a limitation of the electronic device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0186] The processor 901 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0187] The memory 902 may be an internal storage unit of the electronic device 9, for example, a hard disk or memory of the electronic device 9. The memory 902 may also be an external storage device of the electronic device 9, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 9. Further, the memory 902 may also include both an internal storage unit of the electronic device 9 and an external storage device. The memory 902 is used to store computer programs and other programs and data required by the electronic device. The memory 902 may also be used to temporarily store data that has been output or is to be output.

[0188] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0189] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0191] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0194] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0195] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for constructing a semantic navigation map based on a depth camera, characterized in that: include: Using a depth camera to collect a depth image and image data of the environment, and generating point cloud data based on the depth image and image data; Processing the point cloud data using a three-dimensional object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels; Converting the point cloud data with semantic labels into single-line laser data with semantic labels, and inserting the single-line laser data with semantic labels into a local semantic submap; When the local semantic sub-map meets a predetermined condition, the local semantic sub-map is inserted into a sub-map queue, and the local semantic sub-map is spliced ​​into a global map; During the map generation process, loop closure detection technology is used to correct the robot's posture by identifying the previous scenes the robot has passed through, and the corrected posture is applied to the local semantic sub-map and the global map to obtain the final semantic navigation map.

2. The method according to claim 1, characterized in that The processing of the point cloud data using the three-dimensional object detection model includes: Inputting the point cloud data into a predetermined three-dimensional object detection model, and performing object detection based on multi-level features using a fully convolutional feature pyramid network in the three-dimensional object detection model; Converting point cloud data into a regular voxel grid by voxelization, and extracting features from the voxel grid by using a convolution operation; An anchor-free approach is used to detect objects by predicting their center points and bounding box parameters, and outputting three-dimensional object boxes corresponding to different types of objects.

3. The method according to claim 2, characterized in that The three-dimensional object detection model includes a backbone network, a neck network and a head network. In the three-dimensional object detection model, a fully convolutional feature pyramid network is used to perform object detection based on multi-level features, including: A high-dimensional sparse residual network structure is used in the backbone network to perform sparse modification on the residual network, and a sparse three-dimensional convolution operation is used to process irregular and unstructured three-dimensional point cloud data; In the neck network, based on a simplified universal sparse decoder, sparse transposed 3D convolution and 3D convolution are used to process each feature level, and a pruning layer is used to filter the input data through a probabilistic mask; The head network consists of three parallel sparse convolutional layers that share weights between feature levels and generate classification probabilities, bounding box regression parameters, and centerness at each position at output.

4. The method according to claim 1, characterized in that The step of converting the point cloud data with semantic labels into single-line laser data with semantic labels comprises: Select the sampling height range of the depth image and determine which row of the image to start with as the laser data; Calculate the angle between each pixel and the central ray based on the camera's internal parameters, and calculate the angle range for converting the depth image into single-line laser data; Determine the angle increment between two adjacent laser beams, convert the depth value corresponding to each pixel into a laser detection distance value, and map the depth value into a one-dimensional array based on the calculated angle and angle increment value to generate single-line laser data; According to the image pixel coordinates and camera intrinsic parameters, the corresponding coordinates of each pixel point in the camera coordinate system are calculated, and the distance of the laser detection point is calculated; By traversing the sampling area of ​​the depth image, all valid depth data are converted into single-line laser data; In the conversion process, the semantic label of the identified object is assigned to the corresponding single-line laser data to maintain the semantic information of the data.

5. The method according to claim 1, characterized in that In the map generation process, the robot's posture is corrected by identifying the previous scenes that the robot has passed through using loop detection technology, and the corrected posture is applied to the local semantic sub-map and the global map, including: Introducing loop detection technology in the map generation process to identify whether the robot has passed through a scene that it has visited before, and measuring similar images to identify loops; When the robot re-enters a previously visited scene, constraints are imposed on the current position of the robot to correct the accumulated error; The pose corresponding to the first frame point cloud inserted into the submap is used as the initial pose of the submap, and the poses of all historical key frames between the loop key frames are corrected, wherein the historical key frames include the old loop key frame and the new loop key frame; Update and correct the position of the submap, and insert the submap into the global map according to the corrected submap position.

6. The method according to claim 1, characterized in that The method further comprises: In the map generation process, the environment is divided into several small grids for mapping, where each grid represents a local area and stores the probability information of whether there are obstacles in the local area; Set the size and resolution of the grid so that each grid corresponds to a fixed area in the real environment, and assign a probability value to each grid, which is used to represent the probability of an obstacle existing in the grid; When the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. As time goes by and the observation is continuously updated, the probability value gradually decreases and returns to the obstacle-free state. According to different probability values, the state of the grid is divided into obstacle-existing, obstacle-free and unexplored.

7. The method according to claim 6, characterized in that The grid state is divided into obstacle-existing, obstacle-free and unexplored according to the different probability values, including: When the sensor does not detect an obstacle, the state of the grid is updated to no obstacle; when a dynamic obstacle enters the observation range of the sensor, the observation value of the grid is updated to the obstacle observation value of the sensor at the current moment; when the dynamic obstacle leaves the observation range of the sensor, the state of the grid is re-marked as no obstacle; in a static environment, when the sensor observes an obstacle multiple times, the state of the relevant grid is gradually updated to have an obstacle.

8. A semantic navigation map construction device based on a depth camera, characterized in that: include: An acquisition module is configured to acquire a depth image and image data of an environment using a depth camera, and generate point cloud data according to the depth image and image data; a recognition module configured to process the point cloud data using a three-dimensional object detection model to recognize different types of objects in the environment and generate point cloud data with semantic labels; A conversion module, configured to convert the point cloud data with semantic labels into single-line laser data with semantic labels, and insert the single-line laser data with semantic labels into a local semantic submap; a splicing module, configured to insert the local semantic submap into a submap queue when the local semantic submap meets a predetermined condition, and splice the local semantic submap into a global map; The correction module is configured to correct the robot's posture by identifying previous scenes passed by the robot using loop detection technology during the map generation process, and apply the corrected posture to the local semantic sub-map and the global map to obtain a final semantic navigation map.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images