Method and device for constructing semantic navigation map based on depth camera, and storage medium

Abstract

The application provides a kind of semantic navigation map construction method, device and storage medium based on depth camera.The method comprises: using depth camera to collect the depth image and image data of environment, and generate point cloud data;Point cloud data is processed using three-dimensional object detection model, to identify different types of objects in the environment and generate point cloud data with semantic label;With semantic label point cloud data is converted into single line laser data with semantic label, and single line laser data with semantic label is inserted into local semantic submap;When local semantic submap reaches predetermined condition, local semantic submap is inserted into submap queue, and local semantic submap is spliced into global map;In the process of map generation, the previous scene passed by robot is identified using loop detection technology, and the pose of robot is corrected.The application reduces the cost and computing resources of map construction, improves the real-time performance of map construction and the adaptability of dynamic environment.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus and storage medium for constructing semantic navigation maps based on a depth camera. Background Technology

[0002] With the development of computer technology, indoor service robots have become a research focus in order to liberate labor and improve quality of life. These robots need to achieve autonomous navigation and task execution in complex environments, typically relying on accurate maps for positioning and path planning. Currently, service robots on the market commonly use LiDAR and cameras as their primary sensors. LiDAR is widely used in map building, providing high-precision environmental information. However, the high price of LiDAR and its heavy reliance on computing resources keep the overall cost of indoor service robots high, limiting their widespread adoption and application.

[0003] While existing technologies like LiDAR can provide high-precision environmental modeling, their high cost and computational resource consumption are significant drawbacks. Furthermore, LiDAR performance in dynamic environments is inadequate, struggling to update and handle dynamic obstacles in real time. Therefore, current technologies have significant limitations in reducing system costs, improving real-time performance, and adapting to dynamic environments.

[0004] To improve the accessibility and adoption of indoor service robots, it is necessary to explore more economical sensors and algorithms to reduce system costs while ensuring performance, and to effectively handle obstacle information in dynamic environments, ensuring real-time map updates and high accuracy. Summary of the Invention

[0005] In view of this, embodiments of this application provide a semantic navigation map construction method, apparatus and storage medium based on a depth camera to solve the problems of high map construction cost, large consumption of computing resources and poor real-time performance and adaptability to dynamic environments in the prior art.

[0006] A first aspect of this application provides a method for constructing a semantic navigation map based on a depth camera, comprising: acquiring depth images and image data of the environment using a depth camera, and generating point cloud data based on the depth images and image data; processing the point cloud data using a 3D object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels; converting the point cloud data with semantic labels into single-line laser data with semantic labels, and inserting the single-line laser data with semantic labels into a local semantic sub-map; when the local semantic sub-map meets predetermined conditions, inserting the local semantic sub-map into a sub-map queue, and stitching the local semantic sub-maps into a global map; during the map generation process, using loop closure detection technology to correct the robot's pose by identifying the previous scene traversed by the robot, and applying the corrected pose to the local semantic sub-map and the global map to obtain the final semantic navigation map.

[0007] A second aspect of this application provides a semantic navigation map construction apparatus based on a depth camera, comprising: an acquisition module configured to acquire depth images and image data of the environment using a depth camera, and generate point cloud data based on the depth images and image data; an identification module configured to process the point cloud data using a 3D object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels; a conversion module configured to convert the point cloud data with semantic labels into single-line laser data with semantic labels, and insert the single-line laser data with semantic labels into a local semantic sub-map; a stitching module configured to insert the local semantic sub-map into a sub-map queue and stitch the local semantic sub-maps into a global map when the local semantic sub-map meets predetermined conditions; and a correction module configured to correct the robot's pose by identifying the previous scene traversed by the robot during the map generation process using loop closure detection technology, and apply the corrected pose to the local semantic sub-map and the global map to obtain the final semantic navigation map.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0011] This paper utilizes a depth camera to acquire depth images and image data of the environment, and generates point cloud data based on these images and data. A 3D object detection model is used to process the point cloud data to identify different types of objects in the environment and generate point cloud data with semantic labels. The semantically labeled point cloud data is then converted into semantically labeled single-line laser data, which is inserted into a local semantic sub-map. When the local semantic sub-map meets predetermined conditions, it is inserted into a sub-map queue and stitched together to form a global map. During map generation, loop closure detection technology is used to correct the robot's pose by recognizing previous scenes traversed by the robot, and the corrected pose is applied to both the local semantic sub-map and the global map to obtain the final semantic navigation map. This application reduces map construction costs and computational resources, and improves the real-time performance and adaptability to dynamic environments of map construction. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the overall implementation process of the semantic navigation map construction method based on a depth camera provided in the embodiments of this application;

[0014] Figure 2 This is a flowchart illustrating the semantic navigation map construction method based on a depth camera provided in an embodiment of this application;

[0015] Figure 3 This is a schematic diagram illustrating the conversion of a depth map into single-line laser parameters, provided in an embodiment of this application.

[0016] Figure 4 This is a schematic diagram illustrating the range of a depth map converted into a single-line laser, provided in an embodiment of this application.

[0017] Figure 5 This is a schematic diagram illustrating the conversion of depth data within a sampling area of ​​a depth image frame, as provided in an embodiment of this application, into laser data.

[0018] Figure 6 This is a schematic diagram of the route trajectory of the loop keyframe feature point matching result in the circular corridor positioning experiment scenario provided in this application embodiment;

[0019] Figure 7This is a schematic diagram comparing the loopback mapping effect provided in the embodiments of this application with the map before and after loopback adjustment;

[0020] Figure 8 This is a schematic diagram of the structure of the semantic navigation map building device based on a depth camera provided in the embodiments of this application;

[0021] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] With the development of computer technology, indoor service robots have become a key research area in order to reduce human labor intensity and improve quality of life. These robots perform various tasks in indoor environments such as homes and offices, including cleaning, delivery, and companionship. To achieve autonomous navigation and task execution in complex indoor environments, indoor service robots rely on accurate maps for localization and path planning. An accurate map helps the robot identify obstacles and navigate routes, thereby effectively completing tasks.

[0024] Currently, most indoor service robots on the market use LiDAR and cameras as their primary sensors. LiDAR, due to its high precision and reliability, is widely used in map building and localization / navigation. LiDAR can accurately measure distances and shapes in the environment by emitting lasers and receiving reflected signals, generating high-precision environmental maps.

[0025] However, current map-building schemes using LiDAR and cameras as sensors still have the following drawbacks: LiDAR equipment is expensive, often accounting for a large portion of the overall robot cost. The large volume of data generated by LiDAR requires significant computing resources for processing and analysis, further increasing the system's cost and complexity.

[0026] The high cost of existing LiDAR technology and its reliance on computing resources limit the overall cost of indoor service robots, hindering their widespread adoption and application. To improve the accessibility and adoption rate of indoor service robots, it is necessary to explore more economical sensors and algorithms to reduce system costs while ensuring performance.

[0027] In view of the problems existing in the prior art, this application provides a method for constructing a semantic grid navigation map using a depth camera. This application aims to study a method for constructing a semantic navigation map for indoor robots. Since the cost of a camera is much lower than that of a laser sensor, using a camera as the primary sensor for localization and mapping helps to significantly reduce system costs.

[0028] The following section, in conjunction with the accompanying drawings, provides a summary of the overall implementation process of the semantic navigation map construction method involved in this application. Figure 1 This is a schematic diagram illustrating the overall implementation flow of the semantic navigation map construction method based on a depth camera provided in this application embodiment. For example... Figure 1 As shown, the overall implementation process of this depth camera-based semantic navigation map construction method can include:

[0029] This application proposes a method for constructing semantic navigation maps using depth cameras. First, it describes how to convert depth maps into single-line laser data, followed by a probabilistic grid map, which has the function of removing dynamic obstacles. Then, it introduces the concept of a sub-map, which includes the global map, and proposes using loop closure to correct pose, thereby constructing a globally consistent navigation map. Next, it introduces the indoor 3D object recognition network FCAF3D, which takes point clouds as input to obtain 3D detection boxes, converts the single-line lasers within the detection boxes into single-line lasers with semantic information, and assigns matching single-line laser semantic data to each grid point.

[0030] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] Figure 2 This is a flowchart illustrating the semantic navigation map construction method based on a depth camera provided in an embodiment of this application. Figure 2 The depth camera-based semantic navigation map construction method can be executed by a chip mounted on the robot. For example... Figure 2 As shown, the semantic navigation map construction method based on depth cameras can specifically include:

[0032] S201 uses a depth camera to acquire depth images and image data of the environment, and generates point cloud data based on the depth images and image data;

[0033] S202 uses a 3D object detection model to process point cloud data in order to identify different types of objects in the environment and generate point cloud data with semantic labels.

[0034] S203, convert point cloud data with semantic labels into single-line laser data with semantic labels, and insert the single-line laser data with semantic labels into the local semantic sub-map;

[0035] S204: When the local semantic submap meets the predetermined conditions, the local semantic submap is inserted into the submap queue and the local semantic submap is stitched together to form a global map.

[0036] S205. During the map generation process, loop closure detection technology is used to correct the robot's pose by identifying the previous scene the robot has passed through. The corrected pose is then applied to the local semantic sub-map and the global map to obtain the final semantic navigation map.

[0037] In some embodiments, point cloud data is processed using a 3D object detection model, including:

[0038] Point cloud data is input into a predetermined 3D object detection model, and a fully convolutional feature pyramid network is used in the 3D object detection model to perform object detection based on multi-level features.

[0039] Point cloud data is transformed into a regular voxel grid through voxelization, and features are extracted from the voxel grid using convolution operations.

[0040] An anchor-free method is used to detect objects by predicting their center point and bounding box parameters, and outputs 3D object boxes corresponding to different types of objects.

[0041] Specifically, the point cloud data acquired by the depth camera is first input into a predetermined 3D object detection model. The 3D object detection model adopts the FCAF (Fully Convolutional Anchorless 3D Object Detection) method, which performs multi-level feature extraction through a fully convolutional feature pyramid network (FPN), significantly improving the object detection effect.

[0042] Furthermore, the point cloud data is voxelized into a regular voxel mesh. The voxelization process transforms continuous 3D point cloud data into discrete voxel meshes, enabling subsequent convolutional operations to extract features more efficiently. After voxelization, multi-level features are extracted from the voxel mesh using 3D convolutional operations. These features are processed within the multi-level structure of a Feature Pyramid Network (FPN) to capture feature information at different scales and levels.

[0043] Furthermore, unlike traditional 3D object detection methods that rely on predefined anchor points, the FCAF3D method employs an anchor-free detection approach. It performs detection by predicting the object's center point and bounding box parameters, avoiding the computational complexity and parameter tuning issues associated with predefined anchor points. Specifically, after feature extraction, the FCAF3D method directly predicts the object's center point coordinates and bounding box parameters using a convolutional neural network. These predictions are then post-processed to output 3D object boxes corresponding to different object types.

[0044] In one example, the acquired 3D point cloud data is input into the FCAF3D 3D object detection model. The point cloud data is voxelized into a regular voxel mesh. Features are extracted from the voxel mesh through 3D convolutional operations, and multi-level feature processing is performed using a fully convolutional Feature Pyramid Network (FPN). Anchor-free detection is adopted, and the center point and bounding box parameters of the object are predicted through a convolutional neural network. The predicted center point and bounding box parameters are post-processed to output the 3D object boxes corresponding to different types of objects.

[0045] Through the method described in this embodiment, this application can efficiently and accurately identify different types of objects in the environment and generate three-dimensional object boxes with semantic labels, providing high-precision environmental perception capabilities for indoor service robots.

[0046] It should be noted that this application utilizes the capabilities of an RGBD camera to reconstruct high-precision 3D point cloud data from depth images, thereby achieving accurate identification of different types of objects in the environment. The RGBD camera not only provides information from the color image but also distance information at each pixel, thus generating an accurate 3D point cloud. This point cloud data can precisely locate and describe the shape, size, and position of objects, providing crucial scene understanding capabilities for autonomous driving, robotics, and augmented reality applications.

[0047] Different types of objects are identified and labeled accordingly, accurately reflecting their category information. Converting labeled 3D point cloud data into labeled monoline laser data further simplifies the data processing workflow while preserving key semantic information of the objects. Monoline laser data offers excellent spatiotemporal resolution, making it suitable for environmental mapping and scene analysis.

[0048] During the mapping process, single-line laser data with semantic information is introduced, and each grid cell is assigned a corresponding semantic label. This method not only improves the accuracy and efficiency of mapping but also provides important data support for subsequent intelligent decision-making such as path planning and obstacle avoidance. Furthermore, dynamic objects such as pedestrians can be subtracted from the point cloud to remove their influence during the mapping process.

[0049] A well-designed 3D object detection algorithm is crucial. Point cloud-based 3D object detection aims to simultaneously locate and identify 3D objects within a given set of 3D points. As a core technology for 3D scene understanding, it is widely used in autonomous driving, robotics, and AR. While 2D methods are suitable for dense, fixed-size arrays, 3D methods are challenged by irregular, unstructured 3D data of arbitrary volume. Therefore, 2D data processing techniques cannot be directly applied to 3D object detection, leading to the need for creative approaches to 3D data processing. Convolutional 3D object detection methods suffer from scalability issues: large-scale scenes either require impractical amounts of computational resources or take too much time to process. Other methods choose voxel data representation and employ sparse convolution; however, these methods address the scalability issue at the expense of detection accuracy. These object detection methods can provide accurate estimations and good scalability. In addition to scalability and accuracy, an ideal 3D object detection method should also handle objects of arbitrary shape and size without additional modifications and manually tuned hyperparameters.

[0050] In some embodiments, the 3D object detection model includes a backbone network, a neck network, and a head network. The 3D object detection model utilizes a fully convolutional feature pyramid network to perform object detection based on multi-level features, including:

[0051] In the backbone network, a high-dimensional sparse residual network structure is used to sparsely modify the residual network, and sparse 3D convolutional operations are used to process irregular and unstructured 3D point cloud data.

[0052] In the neck network, a simplified general sparse decoder is used to process each feature level using sparse transposed 3D convolutions and 3D convolutions, and the input data is filtered by probability masks using pruning layers.

[0053] The head network consists of three parallel sparse convolutional layers, where the sparse convolutional layers share weights between feature levels, and each location generates classification probability, bounding box regression parameters, and centrality at output.

[0054] Specifically, the acquired 3D point cloud data is input into the 3D object detection model. The input data includes 3D spatial coordinates and related depth information. The backbone network adopts a high-dimensional sparse residual network structure, namely HDResNet, which is a sparse modification of the traditional ResNet.

[0055] Furthermore, all two-dimensional convolutional operations are replaced by sparse three-dimensional convolutional operations, enabling the processing of irregular, unstructured three-dimensional point cloud data. This network structure is primarily designed for processing three-dimensional data, effectively utilizing spatial information and local features.

[0056] Furthermore, the neck network employs a simplified GSDN (Generalized Sparse Decoder Network) decoder. Each feature level is processed through sparse transposed 3D convolutions and 3D convolutions. The kernel size of the transposed sparse 3D convolution operation is 2, which increases the number of non-zero values ​​while maintaining network efficiency. To control memory usage, GSDN uses pruning layers to filter the input data through probability masks.

[0057] Furthermore, FCAF3D's header design is simple and lightweight, consisting of three parallel sparse convolutional layers that share weights between feature levels. At each location (x, y, z), a classification probability p, bounding box regression parameters δ, and centrality c are generated at output. This design is similar to the header structure of FCOS (Fully Convolutional One-Stage Object Detection), but is adapted to the characteristics of 3D data.

[0058] In one example, the processing procedure for a 3D object detection model is as follows:

[0059] Point cloud data input: Input the collected 3D point cloud data into the FCAF3D 3D object detection model.

[0060] Backbone network: The input 3D point cloud data is processed using the HDResNet structure to extract preliminary features.

[0061] Neck network: A simplified GSDN decoder is used to perform sparse transposed 3D convolution and 3D convolution processing on each feature level, and pruning layers are used to filter out unimportant data.

[0062] Head network: It processes the extracted features through three parallel sparse convolutional layers, generating classification probability, bounding box regression parameters and centrality at each location.

[0063] Anchor-free detection method: Detection is performed by predicting the center point and bounding box parameters of the object, avoiding the computational complexity and parameter tuning problems of the traditional anchor point method.

[0064] Output results: The model outputs the 3D object bounding boxes corresponding to different types of objects.

[0065] The method described in this embodiment, utilizing the FCAF3D method, can efficiently and accurately process 3D point cloud data, identify different types of objects in the environment, and generate 3D object boxes with semantic labels, providing high-precision environmental perception capabilities for indoor service robots.

[0066] In some embodiments, converting semantically labeled point cloud data into semantically labeled single-line laser data includes:

[0067] Select the sampling height range of the depth image and determine from which row of the image to start as laser data;

[0068] Calculate the angle between each pixel and the central ray based on the camera intrinsic parameters, and calculate the angular range for converting the depth image into single-line laser data;

[0069] Determine the angle increment between two adjacent laser beams, convert the depth value corresponding to each pixel into a laser detection distance value, and map the depth value into a one-dimensional array based on the calculated angle and angle increment to generate single-line laser data;

[0070] Based on the image pixel coordinates and camera intrinsic parameters, calculate the corresponding coordinates of each pixel in the camera coordinate system, and calculate the distance of the laser detection point;

[0071] By traversing the sampling area of ​​the depth image, all valid depth data is converted into single-line laser data;

[0072] During the conversion process, the semantic labels of the identified objects are assigned to the corresponding single-line laser data in order to preserve the semantic information of the data.

[0073] Specifically, first, the sampling height range (scan_height) of the depth image is selected, and the starting row (offset) of the laser data is determined from which row of the image.

[0074] Furthermore, the angle between each pixel and the central ray is calculated based on camera intrinsic parameters (including the camera's focal length and optical axis offset). Angles on the left side of the image center are positive, and angles on the right side are negative.

[0075] Furthermore, the angular range for converting the depth image into single-line laser data is calculated using camera intrinsic parameters, including the angle between the leftmost laser beam and the center laser beam, and the angle between the rightmost laser beam and the center laser beam. Based on these angles, the angular increment value between two adjacent laser beams is determined.

[0076] Furthermore, the depth value corresponding to each pixel is converted into a laser detection distance value. Based on the calculated included angle and angle increment value, these depth values ​​are mapped into a one-dimensional array to generate single-line laser data.

[0077] Furthermore, based on the image pixel coordinates and camera intrinsic parameters, the corresponding coordinates of each pixel in the camera coordinate system are calculated, and the distance to the laser detection point is calculated.

[0078] Furthermore, by traversing the sampling area of ​​the depth image, all valid depth data is converted into single-line laser data.

[0079] Furthermore, during the conversion process, semantic labels for the identified objects are assigned to the corresponding single-line laser data to preserve the semantic information of the data. These semantic labels can help provide more environmental information in subsequent map building and navigation.

[0080] In one example, the following is a detailed process for implementing the above steps:

[0081] Select the range of image rows in the depth image that can be used as laser data, and determine the starting row offset. For example, starting from the offset+1th row, use the depth values ​​of a certain range of "flattened" areas on the image as the laser spot distance.

[0082] Based on the image pixel coordinates and camera intrinsic parameters, the angle between each pixel and the central ray is calculated. Angles on the left side of the image center are positive, and those on the right side are negative. These angles are used to determine the laser beam corresponding to each pixel.

[0083] The angular range for converting the depth image into single-line laser data is calculated, specifically the angles between the leftmost and rightmost laser beams and the center laser beam. Then, the angular increments between adjacent laser beams are calculated, and these angular ranges and increments are used to map the depth values ​​to the laser data.

[0084] The depth value of each pixel is converted into a laser detection distance value, and these depth values ​​are mapped into a one-dimensional array based on the calculated included angle and angle increment value to generate single-line laser data.

[0085] Using camera intrinsics, the coordinates of each pixel in the camera coordinate system are calculated. For example, given the depth value and camera intrinsics, the coordinates of that point in the camera coordinate system can be calculated, and the distance to the laser detection point can be further calculated.

[0086] Traverse the sampling area of ​​the depth image, convert all valid depth data into single-line laser data, and generate a frame of simulated lidar scanning data within the sampling area of ​​the depth image.

[0087] During the conversion process, the semantic labels of the identified objects are assigned to the corresponding single-line laser data to ensure that each laser data point contains the corresponding semantic information.

[0088] The following section provides a detailed explanation of the overall implementation process of converting a depth map into a single-line laser point cloud, using a specific example. This process may include the following:

[0089] A depth map is a grayscale image where each pixel corresponds to a depth value representing the distance from the camera to a point in the scene. Depth maps can be generated using a depth sensor or a monocular camera combined with depth estimation algorithms to represent the three-dimensional structure of the environment.

[0090] The parameters used in this embodiment are explained in Table 1, where row, col, and f are... x f y c y c x This is internal camera information, provided by the camera manufacturer.

[0091] In Table 1:

[0092]

[0093] Figure 3 This is a schematic diagram illustrating the conversion of a depth map into single-line laser parameters, provided in an embodiment of this application. For example... Figure 3 As shown, the angle range for converting the depth map into a single-line laser can be calculated using camera intrinsic parameters:

[0094]

[0095] The angle between two adjacent laser beams:

[0096]

[0097] Table 1. Depth Map to Laser Conversion Parameters

[0098]

[0099]

[0100] Converting a depth map to single-line laser data involves selecting a certain range of "flattened" depth values ​​on the image as the laser beam distance. Figure 4 This is a schematic diagram illustrating the range of a depth map converted into a single-line laser, provided in an embodiment of this application. For example... Figure 4 As shown in the figure, scan height It is the number of image rows that can be used as laser data, and offset indicates that the laser data starts from the offset+1th row.

[0101] Calculate the angle θ between each pixel and the central ray based on the image pixel coordinates and camera intrinsic parameters, such as... Figure 4 As shown, the angle on the left side of the center of the image is positive, and the angle on the right side is negative:

[0102]

[0103] Figure 5This is a schematic diagram illustrating the conversion of depth data within a sampling area of ​​a depth image frame, as provided in an embodiment of this application, into laser data. For example... Figure 5 As shown, the optical center O is the origin of the laser coordinate system, which coincides with the origin of the camera coordinate system. The thick red line represents the set of laser points, and p is a point on the depth image. Given the depth and camera intrinsic parameters, the coordinates P(X,Y) of this point in the camera coordinate system can be calculated using formula (2-6). Then, the distance between points O and P can be calculated using formula (2-7).

[0104]

[0105] OP corresponds to a laser beam emitted by the lidar, with O as the starting point and P as the ending point. Its index can be calculated according to equation (2-5). By traversing the sequence, the lidar scanning data simulated by the sampling area of ​​a depth image frame can be obtained.

[0106] Through the above embodiments, the present invention can efficiently convert point cloud data with semantic tags into single-line laser data with semantic tags, providing accurate environmental information and semantic recognition capabilities for map building and navigation of indoor service robots.

[0107] In some embodiments, during map generation, loop closure detection technology is used to correct the robot's pose by identifying previous scenes traversed by the robot, and the corrected pose is applied to the local semantic sub-map and the global map, including:

[0108] Loop closure detection technology is introduced during the map generation process to identify whether the robot has passed through previously visited scenes and to identify loop closures by measuring similar images;

[0109] When the robot re-enters a previously visited scene, constraints are imposed on the robot's current position to correct for accumulated errors;

[0110] The pose of the first point cloud frame inserted into the submap is used as the initial pose of the submap. The poses of all historical keyframes between loop closure keyframes are corrected. The historical keyframes include old loop closure keyframes and new loop closure keyframes.

[0111] Update and correct the pose of the submap, and insert the submap into the global map based on the corrected pose.

[0112] Specifically, during map generation, loop closure detection technology is introduced to identify whether the robot has revisited previously visited scenes. Loop closure detection identifies loops by comparing the image currently observed by the robot with images from previous scenes and measuring their similarity.

[0113] When the robot re-enters a previously visited scene, loop closure detection imposes constraints on the robot's current position. These constraints correct for pose deviations caused by long-term accumulated errors, ensuring the robot's positioning accuracy.

[0114] The pose corresponding to the first frame of point cloud inserted into the submap is used as the initial pose of the submap. Each submap has its own initial pose when it is generated, which is determined by the point cloud data of the first frame.

[0115] Loop closure detection technology corrects the poses of all historical keyframes between loop closure keyframes. These historical keyframes include both older and newer loop closure keyframes. By correcting the poses of historical keyframes, offsets in the map caused by accumulated errors can be corrected.

[0116] Based on the loop closure detection results, the pose of the sub-map is updated and corrected. The corrected sub-map pose ensures the accurate position of each sub-map within the global map.

[0117] Based on the corrected submap pose, the submap is inserted into the global map. The global map is composed of multiple submaps; by correcting the poses of the submaps, the accuracy and consistency of the global map can be ensured.

[0118] To address the problem of long-term error accumulation, loop closure detection technology is introduced. When the robot re-enters a previously visited scene, loop closure detection eliminates accumulated errors by identifying whether the robot has traversed the previous scene and measuring similar images. Loop closure detection imposes constraints on its current position, thereby correcting accumulated errors to the correct location.

[0119] Loop closure detection not only corrects errors in long-term localization but also enables relocalization. By associating with historical data, it allows the robot to accurately locate itself in previously explored scenes. As the system runs, many sub-maps are generated, each corresponding to a pose. In the program, the pose of the first frame of the point cloud inserted into the sub-map is defined as the pose of the sub-map. Since loop closure detection corrects the poses of all historical keyframes between the old and new loop closure keyframes, the poses of the sub-maps within this range are also corrected. Since the global map is formed by stitching together sub-maps, changing the position of the sub-map inserted into the global map based on the new sub-map pose achieves the effect of the global map being re-corrected due to loop closure detection.

[0120] By using the method described in this embodiment, this application can use loop closure detection technology to correct the robot's pose during the map generation process, ensuring the consistency and accuracy of the local semantic sub-map and the global map, thereby improving the accuracy of robot navigation and positioning.

[0121] It's important to note that a submap is a map of a local area, containing a fixed number of point cloud frames. When the number of inserted point cloud frames reaches a predetermined value, the submap is considered complete and no further new point cloud frames are accepted. New submap creation: When a new point cloud frame enters the system, if the current submap is full, the system creates a new blank submap and inserts subsequent point cloud frames into it, repeating this process. The global map is formed by stitching together multiple submaps. As point cloud data continues to be input, the number of submaps generated by the system increases, and the global map expands accordingly.

[0122] When inserting new point cloud frames, if the boundary point clouds of these frames exceed the current submap's range, the system adjusts the submap's size to include the new data, thus achieving incremental updates. Visual odometry-based localization and mapping utilize IMU integral constraints and visual observations within a sliding window for short-term localization. Because these methods rely on short-term constraints, errors accumulate over time, leading to a decrease in long-term localization and mapping accuracy. To address the problem of long-term error accumulation, loop closure detection is introduced. Loop closure detection eliminates accumulated errors by identifying whether the robot has traversed a previously visited scene and measuring similar images. When the robot re-enters a previously visited scene, loop closure detection imposes constraints on its current position, correcting accumulated errors to the correct location.

[0123] Loop closure detection can not only correct errors in long-term positioning, but also enable repositioning. By associating with historical data, it allows robots to accurately locate themselves in previously explored scenes.

[0124] As the system runs, many sub-maps are generated, each corresponding to a pose. In the program, the pose of the first frame of the point cloud into which the sub-map is inserted is defined as the pose of the sub-map. Since loop closure corrects the poses of all historical keyframes between the old and new loop closure keyframes, the poses of the sub-maps within this range will also be corrected. Since the global map is formed by stitching together sub-maps, changing the position of the sub-map inserted into the global map according to the new sub-map pose can achieve the effect of the global map being corrected due to the loop closure.

[0125] To verify the loop effect, the main part of the experimental scenario was a ring structure composed of four long corridor segments. Figure 6 This is a schematic diagram of the route trajectory based on the matching results of keyframe feature points in a circular corridor positioning experiment scenario provided in this application embodiment. For example... Figure 6As shown, the red line is the positioning trajectory output by the Visual Inertial Odometry (VIO), and the blue line is the positioning trajectory adjusted by the loopback module. The blue dot is the starting point of the robot's movement, moving counterclockwise. Multiple loopbacks occur when the robot returns to the vicinity of its starting position after circling once. The loopback module optimizes the pose graph of all historical keyframes and outputs the corrected positioning trajectory. Statistics show that the robot experienced a total of 13 loopbacks after returning to the vicinity of the starting point. Actual measurements show that the VIO positioning error is 2.71m, which is relatively large, while the positioning error after loopback is 0.09m.

[0126] in addition, Figure 7 This is a schematic diagram comparing the loopback mapping effect provided in the embodiments of this application with the map before and after loopback adjustment. Figure 7 As shown, this demonstrates the submap-based mapping effect in a circular corridor scene. Figure 7 The diagram on the left shows that the maps that should have overlapped did not have loop closure optimization. The accumulated positioning error after running for a long time caused the corridors that should have overlapped to not overlap when returning to the vicinity of the starting point. Figure 7 The diagram on the right shows the adjusted map. The green line represents the positioning trajectory output by VIO, and the red line represents the trajectory formed by connecting the keyframe poses output after loop closure adjustment. Utilizing loop closure optimization to further adjust the poses of the sub-maps also results in corresponding adjustments to the global map. The diagram shows that the map after loop closure adjustment is significantly improved compared to the original map, and is more consistent with the actual scene.

[0127] It's important to note that map construction relies on location information, and errors in the location results will lead to a mismatch between the map and the real environment. Using a map that doesn't reflect reality for navigation will cause obstacles to be misaligned on the navigation map, potentially leading to collisions between the robot and these obstacles.

[0128] In some embodiments, the method further includes:

[0129] During map generation, the environment is divided into several small grids for mapping. Each grid represents a local area and stores the probability information of whether there are obstacles in the local area.

[0130] Set the size and resolution of the grid so that each grid corresponds to a fixed area in the real environment, and assign a probability value to each grid to represent the probability that there is an obstacle in the grid;

[0131] When the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. As time goes by and the observation is continuously updated, the probability value will gradually decrease and return to the obstacle-free state.

[0132] Based on different probability values, the state of the grid is divided into three categories: with obstacles, without obstacles, and unexplored.

[0133] Specifically, during map generation, the entire environment is divided into several small grids for mapping. Each grid represents a local area and stores the probability information of whether obstacles exist within that local area.

[0134] Furthermore, the size and resolution of the grid are set so that each grid corresponds to a fixed area in the real-world environment. Assuming a resolution of 0.05 meters, each grid corresponds to a 5cm x 5cm area in the real-world environment. This high-resolution grid setup allows for detailed environmental modeling, enabling the robot to accurately perceive its surroundings.

[0135] Furthermore, each grid cell is assigned a probability value representing the probability that an obstacle exists within that cell. Based on the different probability values, the grid cell states are divided into three categories:

[0136] Obstacles present: Probability value close to 1.

[0137] No obstacles: probability value close to 0.

[0138] Unexplored: The probability value is in the middle range.

[0139] Furthermore, when the sensor detects a dynamic obstacle, the probability value of the relevant grid cell temporarily increases. Over time and with continuous updates, these probability values ​​gradually decrease, returning to an obstacle-free state. This dynamic probability update mechanism enables the probabilistic grid map to resist interference from dynamic obstacles and effectively handle dynamic changes in the environment.

[0140] In one example, probabilistic grid mapping (PGM) is a technique for representing a robot's environment by dividing the entire environment into several small grids. Each grid represents a local area and stores the probability information of whether obstacles exist within that area. Assuming a resolution of 0.05 meters, each grid corresponds to a 5cm × 5cm area in the real-world environment. This resolution allows for detailed environment modeling, enabling the robot to accurately perceive its surroundings.

[0141] Each grid cell stores a probability value, representing the probability that an obstacle exists within that cell. Based on these probability values, the grid cells can be categorized into three states:

[0142] Obstacles present: A probability value close to 1 indicates that there are obstacles in the area.

[0143] No obstacles: The probability value is close to 0, indicating that there are no obstacles in the area.

[0144] Unexplored: The probability value is in the middle range, indicating that the area has not yet been explored.

[0145] When a sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. However, over time and with continuous updates to the observations, these probability values ​​will gradually decrease, returning to an obstacle-free state. Therefore, PGM has the ability to resist interference from dynamic obstacles, removing their influence and avoiding misinterpreting dynamic changes as static obstacles.

[0146] Through the method described in this embodiment, this application can effectively utilize probabilistic grid maps for environmental modeling and dynamic obstacle handling during map generation, ensuring the robot's environmental perception capabilities and map accuracy.

[0147] Probability Updates and Dynamic Obstacle Handling: Impact of Dynamic Obstacles: When a sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. However, as time passes and observations are continuously updated, these probability values ​​will gradually decrease, returning to an obstacle-free state. Therefore, PGM has the ability to resist interference from dynamic obstacles, removing their influence and avoiding misinterpreting dynamic changes as static obstacles.

[0148] OpenLORIS Dataset Experiment: Performance of probabilistic grid maps on the OpenLORIS dataset. White grids represent no obstacles, and black grids represent obstacles. The red curve represents the sensor's current obstacle observation, and the pink dashed box represents the observation of dynamic obstacles.

[0149] When there are no obstacles: When the sensor does not detect a person, the grid is white (no obstacles).

[0150] When dynamic obstacles appear: When a dynamic obstacle (such as a person) enters the sensor's observation range, the sensor's observed values ​​for these grids turn red. However, after the person leaves the sensor's observation range, the state of these grids does not change to occupied; they remain white. This demonstrates that PGM can maintain the static characteristics of the environment and has a strong resistance to interference from dynamic obstacles.

[0151] Static environment mapping: In a static environment, multiple observations of obstacles by the sensor gradually update the relevant grid status to black (occupied state). This cumulative probability update mechanism effectively improves the stability and accuracy of the map.

[0152] Probabilistic grid maps effectively address the impact of dynamic obstacles on static maps through a flexible probabilistic update mechanism. This type of map exhibits strong anti-interference capabilities and adaptability, providing reliable support for robot localization and mapping in dynamic environments. Experimental results validate the effectiveness and reliability of probabilistic grid maps.

[0153] In some embodiments, the state of the grid is divided into three categories based on different probability values: obstructed, unobstructed, and unexplored, including:

[0154] When the sensor does not detect an obstacle, the grid state is updated to "no obstacle". When a dynamic obstacle enters the sensor's observation range, the grid's observation value is updated to the sensor's current obstacle observation value. When a dynamic obstacle leaves the sensor's observation range, the grid state is remarked as "no obstacle". In a static environment, when the sensor observes an obstacle multiple times, the state of the relevant grid is gradually updated to "obstacle present".

[0155] Specifically, in the initial stage of map generation, all grid cells are set to an unexplored state, meaning that the probability value of each grid cell is in the middle range.

[0156] Furthermore, state updates when there are no obstacles: When the sensor does not detect any obstacles, the grid state is updated to "obstacle-free," indicating that there are no obstacles within the grid area. For example, when a robot moves in an empty room and does not detect any obstacles, the grid state will be updated to "obstacle-free."

[0157] Furthermore, regarding the handling of dynamic obstacles: when a dynamic obstacle (such as a person) enters the sensor's observation range, the sensor updates the observation values ​​of these grids to the current obstacle observation values. The grid state is temporarily updated to the dynamic obstacle state, i.e., the probability value increases. Specifically, when someone enters the robot's sensor's observation range, the state of these grids turns red, indicating that an obstacle is currently present.

[0158] Furthermore, state recovery after dynamic obstacles leave: When a dynamic obstacle leaves the sensor's observation range, the grid's state does not immediately change to occupied; instead, it is remarked as obstacle-free, indicating that no obstacle exists in the area. In this way, PGM can maintain the static characteristics of the environment and has a strong resistance to interference from dynamic obstacles.

[0159] Furthermore, state updates in static environments: In static environments, when sensors repeatedly observe obstacles, the state of the relevant grids is gradually updated to indicate the presence of obstacles. Specifically, when the robot repeatedly observes the presence of obstacles (such as tables or walls) at fixed locations, the state of these grids gradually turns black, indicating the presence of obstacles in the area and a relatively stable state.

[0160] For example, in a specific instance, when there are no obstacles: when the sensor does not detect a person, the grid is white (no obstacles).

[0161] When dynamic obstacles appear: When a dynamic obstacle (such as a person) enters the sensor's observation range, the sensor's observed values ​​for these grids turn red. However, after the person leaves the sensor's observation range, the state of these grids does not change to occupied; they remain white. This demonstrates that PGM can maintain the static characteristics of the environment and has a strong resistance to interference from dynamic obstacles.

[0162] Static environment mapping: In a static environment, multiple observations of obstacles by the sensor gradually update the relevant grid status to black (occupied state). This cumulative probability update mechanism effectively improves the stability and accuracy of the map.

[0163] Through the method described in this embodiment, this application can dynamically update the state of the grid based on the observation information of the sensor during the map generation process, and effectively handle the influence of dynamic obstacles, ensuring the accuracy and stability of the map.

[0164] According to the technical solution provided in the embodiments of this application, the technical solution of this application has at least the following advantages:

[0165] In the map construction process, a probabilistic grid map (PGM) method was employed to effectively remove moving objects. By continuously updating each grid cell in real time, the influence of detected obstacles can be dynamically removed. This method, through a flexible probabilistic update mechanism, effectively handles the interference of dynamic obstacles on the static map, making the constructed map more reliable and practical. Using this method, the system can maintain map accuracy in dynamic environments, providing a more stable navigation foundation.

[0166] During the construction of the semantic map, frequently moving objects, such as seats, can be selectively removed. By semantically recognizing and filtering these objects, their interference with navigation path planning can be reduced. Specifically, by adjusting the map, unnecessary influence of these common movable objects on the navigation path can be avoided, thereby generating a more reasonable and optimized navigation trajectory. This method not only simplifies navigation path planning but also reduces the number of path adjustments the robot makes in complex environments, improving navigation efficiency.

[0167] The adjusted semantic map not only improves the rationality of navigation but also provides a more environmentally aware navigation solution. In practical applications, robots can navigate more efficiently in complex indoor environments and adapt to dynamically changing environmental characteristics. Through real-time updates and dynamic adjustments, the system can flexibly respond to various changes in the environment, improving the overall performance and application value of robot navigation. This enables robots to more accurately avoid dynamic obstacles when performing tasks, enhancing the safety and reliability of navigation.

[0168] In summary, the technical solution of this application significantly improves the reliability of map construction, the rationality of navigation paths, and the ability to perceive the environment by adopting probabilistic grid maps and semantic recognition technology, providing an efficient and reliable navigation solution for indoor service robots.

[0169] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0170] Figure 8 This is a schematic diagram of the structure of the semantic navigation map construction device based on a depth camera provided in an embodiment of this application. Figure 8 As shown, the depth camera-based semantic navigation map building device includes:

[0171] The acquisition module 801 is configured to acquire depth images and image data of the environment using a depth camera, and generate point cloud data based on the depth images and image data;

[0172] The recognition module 802 is configured to process point cloud data using a 3D object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels.

[0173] The conversion module 803 is configured to convert point cloud data with semantic labels into single-line laser data with semantic labels, and insert the single-line laser data with semantic labels into a local semantic sub-map.

[0174] The stitching module 804 is configured to insert the local semantic submap into the submap queue and stitch the local semantic submap into a global map when the local semantic submap meets the predetermined conditions.

[0175] The correction module 805 is configured to use loop closure detection technology to correct the robot's pose by identifying the previous scene the robot has passed through during the map generation process, and apply the corrected pose to the local semantic sub-map and the global map to obtain the final semantic navigation map.

[0176] In some embodiments, Figure 8 The recognition module 802 inputs point cloud data into a predetermined 3D object detection model. In the 3D object detection model, a fully convolutional feature pyramid network is used to perform object detection based on multi-level features. The point cloud data is converted into a regular voxel grid through voxelization, and features are extracted from the voxel grid using convolution operations. Anchor-free method is used to perform detection by predicting the center point and bounding box parameters of the object, and outputs the 3D object boxes corresponding to different types of objects.

[0177] In some embodiments, the 3D object detection model includes a backbone network, a neck network, and a head network. Figure 8 The recognition module 802 utilizes a high-dimensional sparse residual network structure in the backbone network to sparsely modify the residual network and uses sparse 3D convolutional operations to process irregular and unstructured 3D point cloud data. In the neck network, based on a simplified general sparse decoder, sparse transposed 3D convolutions and 3D convolutions are used to process each feature level, and pruning layers are used to filter the input data through probability masks. The head network consists of three parallel sparse convolutional layers, where the sparse convolutional layers share weights between feature levels, and each position generates classification probability, bounding box regression parameters, and centrality at output.

[0178] In some embodiments, Figure 8 The conversion module 803 selects the sampling height range of the depth image and determines from which row of the image to start as laser data; it calculates the angle between each pixel and the central ray based on camera intrinsic parameters, and calculates the angle range for converting the depth image into single-line laser data; it determines the angle increment value between two adjacent laser beams, converts the depth value corresponding to each pixel into a laser detection distance value, and maps the depth value into a one-dimensional array based on the calculated angle and angle increment value to generate single-line laser data; it calculates the corresponding coordinates of each pixel in the camera coordinate system based on the image pixel coordinates and camera intrinsic parameters, and calculates the distance to the laser detection point; it converts all valid depth data into single-line laser data by traversing the sampling area of ​​the depth image; during the conversion process, the recognized object semantic labels are assigned to the corresponding single-line laser data to preserve the semantic information of the data.

[0179] In some embodiments, Figure 8 The correction module 805 introduces loop closure detection technology during map generation to identify whether the robot has passed through previously visited scenes and measures similar images to identify loop closures. When the robot re-enters a previously visited scene, constraints are applied to the robot's current position to correct accumulated errors. The pose corresponding to the first frame point cloud inserted into the sub-map is used as the initial pose of the sub-map. The poses of all historical keyframes between loop closure keyframes are corrected, where historical keyframes include old loop closure keyframes and new loop closure keyframes. The pose of the sub-map is updated and corrected, and the sub-map is inserted into the global map based on the corrected sub-map pose.

[0180] In some embodiments, Figure 8During map generation, the mapping module 806 divides the environment into several small grids, each representing a local area and storing the probability information of whether obstacles exist in that area. The grid size and resolution are set so that each grid corresponds to a fixed area in the real environment. A probability value is assigned to each grid, representing the probability of an obstacle being present within it. When the sensor detects a dynamic obstacle, the probability value of the relevant grid temporarily increases. Over time and with continuous updates, the probability value gradually decreases, returning to an obstacle-free state. Based on the different probability values, the grid's state is categorized as having obstacles, being obstacle-free, or unexplored.

[0181] In some embodiments, Figure 8 When the sensor does not detect an obstacle, the mapping module 806 updates the grid state to "no obstacle"; when a dynamic obstacle enters the sensor's observation range, it updates the grid's observation value to the sensor's current obstacle observation value; when a dynamic obstacle leaves the sensor's observation range, it re-marks the grid state as "no obstacle"; in a static environment, when the sensor observes an obstacle multiple times, it gradually updates the state of the relevant grid to "obstacle present".

[0182] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0183] Figure 9 This is a schematic diagram of the structure of the electronic device 9 provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device 9 of this embodiment includes a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, it implements the steps in the various method embodiments described above. Alternatively, when the processor 901 executes the computer program 903, it implements the functions of each module / unit in the various device embodiments described above.

[0184] For example, computer program 903 may be divided into one or more modules / units, which are stored in memory 902 and executed by processor 901 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 903 in electronic device 9.

[0185] Electronic device 9 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 9 may include, but is not limited to, processor 901 and memory 902. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0186] The processor 901 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0187] The memory 902 can be an internal storage unit of the electronic device 9, such as a hard disk or RAM. The memory 902 can also be an external storage device of the electronic device 9, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 902 can include both internal and external storage units of the electronic device 9. The memory 902 is used to store computer programs and other programs and data required by the electronic device. The memory 902 can also be used to temporarily store data that has been output or will be output.

[0188] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0189] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0190] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0191] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0192] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0195] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A semantic navigation map construction method based on a depth camera, characterized in that, include: A depth camera is used to acquire depth images and image data of the environment, and point cloud data is generated based on the depth images and image data; The point cloud data is processed using a 3D object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels. The point cloud data with semantic labels is converted into single-line laser data with semantic labels, and the single-line laser data with semantic labels is inserted into the local semantic sub-map. This includes: selecting the sampling height range of the depth image and determining which row of the image to start using as laser data; calculating the angle between each pixel and the central ray based on camera intrinsics, and calculating the angle range for converting the depth image into single-line laser data; determining the angle increment value between two adjacent laser beams, converting the depth value corresponding to each pixel into a laser detection distance value, and mapping the depth value into a one-dimensional array based on the calculated angle and angle increment value to generate single-line laser data; calculating the corresponding coordinates of each pixel in the camera coordinate system based on the image pixel coordinates and camera intrinsics, and calculating the distance to the laser detection point; converting all valid depth data into single-line laser data by traversing the sampling area of ​​the depth image; wherein, during the conversion process, the identified object semantic labels are assigned to the corresponding single-line laser data to preserve the semantic information of the data; When the local semantic sub-map meets the predetermined conditions, the local semantic sub-map is inserted into the sub-map queue and the local semantic sub-map is stitched together to form a global map; During map generation, loop closure detection technology is used to correct the robot's pose by identifying the previous scenes it has passed through. The corrected pose is then applied to the local semantic sub-map and the global map to obtain the final semantic navigation map.

2. The method according to claim 1, characterized in that, The process of processing the point cloud data using a 3D object detection model includes: The point cloud data is input into a predetermined 3D object detection model, in which a fully convolutional feature pyramid network is used to perform object detection based on multi-level features. Point cloud data is transformed into a regular voxel grid through voxelization, and features are extracted from the voxel grid using convolution operations. An anchor-free method is used to detect objects by predicting their center point and bounding box parameters, and outputs 3D object boxes corresponding to different types of objects.

3. The method according to claim 2, characterized in that, The 3D object detection model includes a backbone network, a neck network, and a head network. This model utilizes a fully convolutional feature pyramid network to perform object detection based on multi-level features, including: In the backbone network, a high-dimensional sparse residual network structure is used to sparsely modify the residual network, and sparse three-dimensional convolution operations are used to process irregular and unstructured three-dimensional point cloud data. The neck network is based on a simplified general sparse decoder, which processes each feature level using sparse transposed 3D convolutions and 3D convolutions, and uses pruning layers to filter input data through probability masks. The head network consists of three parallel sparse convolutional layers, where the sparse convolutional layers share weights between feature levels, and each position generates classification probability, bounding box regression parameters, and centrality at output.

4. The method according to claim 1, characterized in that, In the map generation process, loop closure detection technology is used to correct the robot's pose by identifying the previous scene it has traversed, and the corrected pose is applied to the local semantic sub-map and the global map, including: Loop closure detection technology is introduced during the map generation process to identify whether the robot has passed through previously visited scenes and to identify loop closures by measuring similar images; When the robot re-enters a previously visited scene, constraints are imposed on the robot's current position to correct for accumulated errors; The pose corresponding to the first frame point cloud inserted into the submap is used as the initial pose of the submap. The poses of all historical keyframes between loop closure keyframes are corrected. The historical keyframes include old loop closure keyframes and new loop closure keyframes. Update and correct the pose of the submap, and insert the submap into the global map based on the corrected pose.

5. The method according to claim 1, characterized in that, The method further includes: During the map generation process, the environment is divided into several small grids for map construction. Each grid represents a local area and stores the probability information of whether there are obstacles in the local area. Set the size and resolution of the grid so that each grid corresponds to a fixed area in the real environment, and assign a probability value to each grid to represent the probability that an obstacle exists in the grid; When the sensor detects a dynamic obstacle, the probability value of the relevant grid will temporarily increase. As time goes by and the observation is continuously updated, the probability value will gradually decrease and return to the obstacle-free state. Based on the different probability values, the state of the grid is divided into three categories: with obstacles, without obstacles, and unexplored.

6. The method according to claim 5, characterized in that, The step of classifying the state of the grid into three categories—with obstacles, without obstacles, and unexplored—based on the different probability values ​​includes: When the sensor does not detect an obstacle, the state of the grid is updated to "no obstacle". When a dynamic obstacle enters the sensor's observation range, the observation value of the grid is updated to the sensor's current obstacle observation value. When the dynamic obstacle leaves the sensor's observation range, the state of the grid is remarked as "no obstacle". In a static environment, when the sensor observes an obstacle multiple times, the state of the relevant grid is gradually updated to "obstacle present".

7. A semantic navigation map construction device based on a depth camera, characterized in that, include: The acquisition module is configured to acquire depth images and image data of the environment using a depth camera, and generate point cloud data based on the depth images and image data; The recognition module is configured to process the point cloud data using a 3D object detection model to identify different types of objects in the environment and generate point cloud data with semantic labels. The conversion module is configured to convert the point cloud data with semantic labels into single-line laser data with semantic labels, and insert the single-line laser data with semantic labels into a local semantic sub-map; This includes: selecting the sampling height range of the depth image and determining which row of the image to start using as laser data; calculating the angle between each pixel and the central ray based on camera intrinsics, and calculating the angle range for converting the depth image into single-line laser data; determining the angle increment value between two adjacent laser beams, converting the depth value corresponding to each pixel into a laser detection distance value, and mapping the depth value into a one-dimensional array based on the calculated angle and angle increment value to generate single-line laser data; calculating the corresponding coordinates of each pixel in the camera coordinate system based on the image pixel coordinates and camera intrinsics, and calculating the distance to the laser detection point; converting all valid depth data into single-line laser data by traversing the sampling area of ​​the depth image; wherein, during the conversion process, the identified object semantic labels are assigned to the corresponding single-line laser data to preserve the semantic information of the data; The stitching module is configured to insert the local semantic sub-map into the sub-map queue and stitch the local semantic sub-map into a global map when the local semantic sub-map meets a predetermined condition. The correction module is configured to use loop closure detection technology to correct the robot's pose by identifying the previous scene the robot has passed through during the map generation process, and apply the corrected pose to the local semantic sub-map and the global map to obtain the final semantic navigation map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images