Three-dimensional dense surface element mapping method based on SLAM and its system and electronic equipment

Abstract

A 3D dense surface element mapping method based on SLAM and its system and electronic device. The 3D dense surface element mapping method based on SLAM includes the following steps: pre-processing the incoming image frame data and posture data to obtain synchronized image frame posture information and corresponding image information, wherein the image information includes aligned grayscale images and depth images; performing superpixel segmentation, surface element extraction and fusion processing on the image information in combination with the image frame posture information in the same thread to obtain an updated surface element map; and performing structured mesh processing on the updated surface element map to output the triangular mesh data of the dense map in real time.

Description

Technical Field

[0001] The present invention relates to the field of SLAM technology, and in particular to a 3D dense surface element mapping method based on SLAM, a system thereof and electronic equipment. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology is the core technology for positioning and navigation of mobile robots, virtual reality and augmented reality devices. Its algorithm framework is becoming more and more perfect, and it is gradually turning from theoretical research to mature market applications. At present, the classic SLAM algorithm framework based on visual sensors or based on the coupling of visual sensors and inertial measurement units (IMU) usually includes modules such as front-end extraction and tracking of image features, back-end optimization estimation of spatial point positions and their own postures, loop detection correction of offsets, and establishment of three-dimensional maps, and has tended to be finalized. For example, back-end optimization mainly includes two types of methods: filtering optimization and nonlinear optimization. To this day, the representative bundle optimization (Bundle Adjustment), pose graph and factor graph algorithms in this nonlinear optimization have obvious advantages and have become the mainstream application algorithms for back-end sliding window optimization and global optimization of SLAM solutions.

[0003] Since estimating and reconstructing the surrounding three-dimensional environment is one of the basic capabilities of autonomous robots to achieve advanced tasks such as safe navigation and operation, and is also a bridge for the wearer to perceive and interact with the surrounding environment in wearable devices such as AR and VR, in order to make the reconstructed three-dimensional map available in mobile robot platforms or wearable devices, the mapping algorithm needs to meet some basic requirements: first, the three-dimensional reconstructed map must densely cover the environment to provide sufficient information for navigation or interaction; second, the mapping module should have good scalability and operating efficiency, such as covering three-dimensional reconstruction of scenes of different scales such as rooms and even streets, and can be deployed in limited CPU / GPU computing resources; finally, the reconstructed three-dimensional map needs to be globally consistent to eliminate the pose estimation drift caused by camera movement.

[0004] However, the mainstream 3D map reconstruction method is usually based on voxels to represent the 3D environment. It obtains the surface position by storing and updating the truncated signed distance, and then uses the marching cube algorithm to obtain the 3D environment grid data. However, the problem with this method is the lack of flexibility, especially when the SLAM system performs keyframe pose graph optimization or loop detection, which leads to pose correction, the voxel map cannot be deformed accordingly. For example, KinectFusion is a pioneering work to achieve real-time 3D reconstruction and pose estimation of RGBD images. Its principle is to represent the 3D environment through voxels with truncated signed distance field information, and the CUDA general parallel computing architecture is required to realize the accelerated operation of 3D point cloud and ICP matching.

[0005] Another method of representing the 3D environment by using face elements can flexibly transform the map according to the change of posture, thereby maintaining the map by using the posture optimization information of the SLAM system. Figure 1 However, most 3D map reconstruction methods require computing resources such as GPUs to process a large number of 3D point cloud operations, which makes it difficult to deploy on mobile platforms with limited computing resources. For example, SurfelMeshing extracts surface elements from the pixels of RGBD images, and then realizes 3D surface reconstruction and structured mesh acquisition, but this requires GPU acceleration to realize surface reconstruction and smoothing denoising.

[0006] In addition, in the SLAM scheme based on nonlinear optimization, the use of sliding windows to read a fixed number of continuous image frames for local optimization can quickly calculate the pose of ordinary image frames; at the same time, by building key frames, fast loop detection and global optimization can be performed, which can be fed back to the pose optimization of ordinary frames in the sliding window to achieve globally consistent state information. Therefore, the 3D map reconstruction module not only needs to obtain the pose of ordinary image frames for 3D point cloud extraction and fusion, but also needs to obtain the pose optimization information of key frames after global optimization to maintain the consistency of the 3D map, in order to reduce the map drift caused by the cumulative error of pose estimation. In this way, the Bundle Fusion of the voxel method realizes the corresponding global optimized map update through the inverse integration and reintegration of RGBD image frames, while the Elastic Fusion of the facet method realizes the global consistency of the map by adding a closed loop detection module, and then performs map deformation. However, these two 3D map reconstruction schemes also need to use GPU for parallel acceleration to meet real-time computing. Summary of the invention

[0007] An advantage of the present invention is that it provides a SLAM-based three-dimensional dense surface element mapping method and its system and electronic equipment, which can realize real-time simultaneous positioning and deployment of three-dimensional dense map reconstruction solutions in a limited computing resource scenario.

[0008] Another advantage of the present invention is that it provides a SLAM-based three-dimensional dense surface element mapping method and its system and electronic device. In one embodiment of the present invention, the SLAM-based three-dimensional dense surface element mapping method can utilize the global optimization of the SLAM system to solve the problem of maintaining the global consistency of the three-dimensional map, while realizing single-threaded real-time processing and fusion of new image frame information on the CPU. It is expected to serve as the underlying algorithm of augmented reality devices and mobile robots, and facilitate the opening of upper-level applications such as virtual-reality integration and robot navigation operations.

[0009] Another advantage of the present invention is that it provides a SLAM-based three-dimensional dense surface element mapping method and its system and electronic device. In one embodiment of the present invention, the SLAM-based three-dimensional dense surface element mapping method can first perform superpixel segmentation on the grayscale and depth images for each frame of ordinary image fusion process, and then extract the surface elements and fuse them with the global map, so as to greatly reduce the amount of calculation and improve the real-time performance of the method.

[0010] Another advantage of the present invention is that it provides a SLAM-based three-dimensional dense surface element mapping method and its system and electronic device. In one embodiment of the present invention, the SLAM-based three-dimensional dense surface element mapping method can realize real-time superpixel segmentation of grayscale images and depth images on a single thread, and then extract and fuse surface element maps, which helps to realize real-time processing and updating of surface element maps.

[0011] Another advantage of the present invention is that it provides a 3D dense surface element mapping method based on SLAM, a system and an electronic device thereof. In one embodiment of the present invention, the 3D dense surface element mapping method based on SLAM can deform the surface elements by obtaining the key frame poses globally optimized by the SLAM backend so as to maintain a globally consistent dense map.

[0012] Another advantage of the present invention is that it provides a SLAM-based three-dimensional dense surface element mapping method and its system and electronic device. In one embodiment of the present invention, the SLAM-based three-dimensional dense surface element mapping method can extract surface elements by superpixel segmentation of grayscale images and depth images, so that the surface element map establishment process can be calculated on the CPU, which is convenient for deployment on an embedded platform.

[0013] Another advantage of the present invention is to provide a 3D dense surface element mapping method based on SLAM and its system and electronic device. In one embodiment of the present invention, the 3D dense surface element mapping method based on SLAM can optimize and improve the SLIC superpixel segmentation and surface element extraction process in parallel, so that the surface element map can be processed and updated in real time on a single CPU thread.

[0014] Another advantage of the present invention is to provide a 3D dense surface element mapping method based on SLAM, a system thereof, and an electronic device, wherein in order to achieve the above advantages, the present invention does not need to adopt a complex structure and a huge amount of calculation, and has low requirements on software and hardware. Therefore, the present invention successfully and effectively provides a solution, not only providing a 3D dense surface element mapping method based on SLAM, a system thereof, and an electronic device, but also increasing the practicality and reliability of the 3D dense surface element mapping method based on SLAM, a system thereof, and an electronic device.

[0015] In order to achieve at least one of the above advantages or other advantages and purposes, the present invention provides a 3D dense surface element mapping method based on SLAM, comprising the steps of:

[0016] Preprocessing incoming image frame data and pose data to obtain synchronized image frame pose information and corresponding image information, wherein the image information includes aligned grayscale images and depth images;

[0017] In the same thread, the image information is combined with the image frame pose information to perform superpixel segmentation, surfel extraction and fusion processing to obtain an updated surfel map; and

[0018] The updated facet map is subjected to structured mesh processing to output triangular mesh data of a dense map in real time.

[0019] According to an embodiment of the present application, the step of performing superpixel segmentation, bin extraction and fusion processing on the image information in combination with the image frame pose information in the same thread to obtain an updated bin map includes the steps of:

[0020] Performing super-pixel segmentation processing on the grayscale image in combination with the depth image to obtain super-pixel data of a new image frame;

[0021] Extracting facets from the incoming image frame according to the superpixel data of the incoming image frame to obtain facets to be fused in the incoming image frame; and

[0022] According to the to-be-fused surfels of the new image frame, a surfel update process is performed on the current surfel map to obtain an updated surfel map.

[0023] According to an embodiment of the present application, the step of performing superpixel segmentation processing on the grayscale image in combination with the depth image to obtain superpixel data of the incoming image frame includes the steps of:

[0024] Initializing the image grids in the depth image and the grayscale image to obtain initial superpixels of a new image frame;

[0025] Classifying the pixels in the new image frame according to the initial superpixels of the new image frame to obtain classified pixels of the new image frame; and

[0026] According to the classified pixel points of the new image frame, the initial superpixel of the new image frame is iteratively updated to obtain the superpixel data of the new image frame with converged data.

[0027] According to an embodiment of the present application, the step of classifying the pixels in the incoming image frame according to the initial superpixel of the incoming image frame to obtain the classified pixels of the incoming image frame includes the steps of:

[0028] Traverse the image grid to extract the current pixel point from the current grid;

[0029] In response to the current pixel being marked as invalid, directly skipping the current grid;

[0030] In response to the current pixel being not marked as invalid, dividing the current grid into a plurality of sub-grids to determine a current super-pixel for which a distance metric value needs to be calculated; and

[0031] By calculating the distance metric value between the current pixel and the determined superpixel, the current superpixel with the minimum distance metric value to the current pixel is selected for classification to obtain the corresponding classified pixel.

[0032] According to an embodiment of the present application, the calculation formula of the distance metric value is implemented as follows:

[0033] D=[|x i -u x |+|y i -u y |]×F S +|I i -u I |×F C +|disp i -u disp |×F d ×S d , where u x ,u y ,u I and u disp Represent the pixel coordinates, grayscale value and disparity of the current pixel respectively; x i ,y i ,I i and disp i Represent the pixel coordinates, grayscale value and disparity of the current superpixel respectively; F S 、F C 、F d and Sd are the corresponding weight coefficients respectively.

[0034] According to an embodiment of the present application, the step of iteratively updating the initial superpixel of the incoming image frame according to the classified pixel points of the incoming image frame to obtain the superpixel data of the incoming image frame with data convergence includes the steps of:

[0035] Traversing the image grid to extract the current superpixel from the current grid;

[0036] In response to the current superpixel being marked as invalid, directly skipping the current grid;

[0037] In response to the current superpixel not being marked as invalid, by traversing the pixel points in the current grid and the surrounding grids, solving the average values ​​of the pixel coordinates, grayscale values, and depth values ​​of the pixel points classified into the current superpixel, so as to update the attributes of the current superpixel; and

[0038] Repeat the above pixel classification step and the above super-pixel updating step until the super-pixel data of the new image frame with data convergence is obtained.

[0039] According to an embodiment of the present application, the step of extracting a surface element from the incoming image frame according to the superpixel data of the incoming image frame to obtain the surface element to be fused of the incoming image frame includes the steps of:

[0040] Traversing the superpixels that are not marked as invalid, retrieving the pixel points that are classified into the current superpixel within the range around the center of the current superpixel, and storing the pixel coordinates and depth values ​​of the pixel points with valid depth values ​​in the container as internal points;

[0041] In response to the number of inliers in the container being less than a number threshold, skipping the current superpixel; and

[0042] In response to the number of inner points in the container being greater than or equal to the number threshold, the three-dimensional spatial coordinates of the inner points are calculated through a back-projection model based on camera parameters and the pixel coordinates and depth values ​​of the inner points, so as to use the plane formed by the inner points in the three-dimensional space as the face element of the new image frame.

[0043] According to an embodiment of the present application, the step of performing a surfel update process on the current surfel map according to the surfels to be fused of the new image frame to obtain an updated surfel map comprises the steps of:

[0044] Traversing all existing bins in the current bin map to transform and project the three-dimensional coordinates of the current existing bins into the image coordinate system of the new image frame;

[0045] In response to the image coordinates of the current existing bin exceeding the image range or the corresponding superpixel label being marked as invalid, skipping the current existing bin;

[0046] In response to the superpixel corresponding to the current existing surface element being not marked as invalid and the depth value and the normal vector between the two satisfying the threshold judgment condition, fusing the current existing surface element with the corresponding surface element to be fused by weighted averaging to obtain a fused surface element; and

[0047] The current surfel map is updated and expanded according to the fused surfel and the surfels in the new image frame that are not involved in the fusion, so as to obtain the updated surfel map.

[0048] According to an embodiment of the present application, the step of preprocessing the incoming image frame data and pose data to obtain synchronized image frame pose information and corresponding image information, wherein the image information includes aligned grayscale images and depth images, comprises the steps of:

[0049] When the incoming image frame data is a binocular image, binocular epipolar correction is first performed to obtain a dedistorted image, and then a disparity image corresponding to the left eye image is calculated by stereo matching to obtain the depth image by binocular camera baseline calculation; and

[0050] When the incoming image frame data is a color image and a depth sensor image, the color image is first converted into a grayscale image and dedistorted, and then aligned according to the external parameters between the cameras and the internal parameters of the RGB sensor to obtain an aligned depth image and disparity image.

[0051] According to another aspect of the present application, an embodiment of the present application further provides a SLAM-based three-dimensional dense surface element mapping system, including:

[0052] a preprocessing module, configured to preprocess incoming image frame data and posture data to obtain synchronized image frame posture information and corresponding image information, wherein the image information includes aligned grayscale images and depth images;

[0053] A facet processing module, used to perform superpixel segmentation, facet extraction and fusion processing on the image information in combination with the image frame pose information in the same thread to obtain an updated facet map; and

[0054] A mesh processing module is used to perform structured mesh processing on the updated surface element map to output the triangular mesh data of the dense map in real time.

[0055] According to one embodiment of the present application, the facet processing module includes a superpixel segmentation module, a facet extraction module and a facet update module which are communicatively connected to each other, wherein the superpixel segmentation module is used to perform superpixel segmentation processing on the grayscale image in combination with the depth image to obtain superpixel data of an incoming image frame; wherein the facet extraction module is used to extract facets from the incoming image frame according to the superpixel data of the incoming image frame to obtain facets to be fused of the incoming image frame; wherein the facet update module is used to perform facet update processing on the current facet map according to the facets to be fused of the incoming image frame to obtain an updated facet map.

[0056] According to one embodiment of the present application, the superpixel segmentation module includes an initialization module, a classification module and an iterative update module that are communicatively connected to each other, wherein the initialization module is used to initialize the image grids in the depth image and the grayscale image to obtain the initial superpixels of the new image frame; wherein the classification module is used to classify the pixel points in the new image frame according to the initial superpixels of the new image frame to obtain the classified pixel points of the new image frame; wherein the iterative update module is used to iteratively update the initial superpixels of the new image frame according to the classified pixel points of the new image frame to obtain the superpixel data of the new image frame with data convergence.

[0057] According to one embodiment of the present application, the classification module is further used to: traverse the image grid to extract the current pixel point from the current grid; in response to the current pixel point being marked as invalid, directly skip the current grid; in response to the current pixel point not being marked as invalid, divide the current grid into multiple sub-grids to determine the current superpixel that needs to calculate the distance metric value; and by calculating the distance metric value between the current pixel point and the determined superpixel, classify the current superpixel with the minimum distance metric value with the current pixel point to obtain the corresponding classified pixel point.

[0058] According to an embodiment of the present application, the calculation formula of the distance metric value is implemented as follows:

[0059] where u x ,u y ,u I and u disp Represent the pixel coordinates, grayscale value and disparity of the current pixel respectively; x i ,y i ,I i and disp i Represent the pixel coordinates, grayscale value and disparity of the current superpixel respectively; F S 、F C 、F dand S d are the corresponding weight coefficients respectively.

[0060] According to one embodiment of the present application, the iterative update module is further used to: traverse the image grid to extract the current superpixel from the current grid; in response to the current superpixel being marked as invalid, directly skip the current grid; in response to the current superpixel not being marked as invalid, by traversing the pixel points in the current grid and its surrounding grids, solve the pixel coordinates, grayscale values ​​and depth values ​​of the pixel points classified into the current superpixel to update the properties of the current superpixel; and repeat the above-mentioned pixel classification step and the above-mentioned superpixel update step until the superpixel data of the new image frame with data convergence is obtained.

[0061] According to one embodiment of the present application, the face element extraction module includes a pixel point retrieval module, a superpixel skipping module and a back projection calculation module that are communicatively connected to each other, wherein the pixel point retrieval module is used to traverse superpixels that are not marked as invalid, retrieve pixel points classified into the current superpixel within the range around the center of the current superpixel, and store the pixel coordinates and depth values ​​of the pixel points with valid depth values ​​in a container as internal points; wherein the superpixel skipping module is used to skip the current superpixel in response to the number of internal points in the container being less than a quantity threshold; wherein the back projection calculation module is used to calculate the three-dimensional spatial coordinates of the internal point through a back projection model according to camera parameters and the pixel coordinates and depth values ​​of the internal point in response to the number of internal points in the container being greater than or equal to the quantity threshold, so as to use the plane formed by the internal point in the three-dimensional space as the face element of the new image frame.

[0062] According to an embodiment of the present application, the facet update module includes a facet projection module, a facet skipping module, a facet fusion module and an update and extension module that are communicatively connected to each other, wherein the facet projection module is used to traverse all existing facets in the current facet map to convert and project the three-dimensional coordinates of the current existing facets to the image coordinate system of the new image frame; wherein the facet skipping module is used to skip the current existing facet in response to the image coordinates of the current existing facet exceeding the image range or the corresponding superpixel label being marked as invalid; wherein the facet fusion module is used to fuse the current existing facet with the corresponding facet to be fused by weighted averaging in response to the superpixel corresponding to the current existing facet not being marked as invalid and the depth value and normal vector between the two satisfying the threshold judgment condition, so as to obtain the fused facet; wherein the update and extension module is used to update and expand the current facet map according to the fused facet and the facets not involved in the fusion in the new image frame, so as to obtain the updated facet map.

[0063] According to another aspect of the present application, an embodiment of the present application further provides an electronic device, including:

[0064] at least one processor for executing instructions; and

[0065] A memory communicatively connected to the at least one processor, wherein the memory has at least one instruction, wherein the instruction is executed by the at least one processor to enable the at least one processor to perform some or all steps in a 3D dense surface element mapping method based on SLAM, wherein the 3D dense surface element mapping method based on SLAM comprises the steps of:

[0066] Preprocessing incoming image frame data and pose data to obtain synchronized image frame pose information and corresponding image information, wherein the image information includes aligned grayscale images and depth images;

[0067] In the same thread, the image information is combined with the image frame pose information to perform superpixel segmentation, surfel extraction and fusion processing to obtain an updated surfel map; and

[0068] The updated facet map is subjected to structured mesh processing to output triangular mesh data of a dense map in real time.

[0069] Further objects and advantages of the present invention will be fully apparent from an understanding of the following description and the accompanying drawings.

[0070] These and other objects, features and advantages of the present invention will be more fully understood from the following detailed description, accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 4 is a schematic diagram of a framework of a SLAM three-dimensional mapping module according to an embodiment of the present invention.

[0072] Figure 2 It is a flowchart of a SLAM-based three-dimensional dense surface element mapping method according to an embodiment of the present invention.

[0073] Figure 3 A schematic flow chart of the preprocessing step in the SLAM-based three-dimensional dense surface element mapping method according to the above embodiment of the present invention is shown.

[0074] 4A to 4F A schematic flow chart of the facet processing step in the SLAM-based three-dimensional dense facet mapping method according to the above embodiment of the present invention is shown.

[0075] Figure 5AAn example of performing superpixel initialization on a grayscale image according to the SLAM-based three-dimensional dense surface element mapping method according to the above embodiment of the present application is shown.

[0076] Figure 5B An example of performing superpixel initialization on a depth image according to the SLAM-based three-dimensional dense surface element mapping method according to the above embodiment of the present application is shown.

[0077] Fig. 6A An example of traversal using superpixels as elements in the SLAM-based three-dimensional dense surface element mapping method according to the above embodiment of the present application is shown.

[0078] Figure 6B An example of traversal using pixel points as elements in the SLAM-based three-dimensional dense surface element mapping method according to the above embodiment of the present application is shown.

[0079] Figure 6C An example of traversal using a grid as an element by the SLAM-based three-dimensional dense surface element mapping method according to the above embodiment of the present application is shown.

[0080] Figure 7 An example of sampling within a grid according to the SLAM-based three-dimensional dense surface element mapping method according to the above embodiment of the present application is shown.

[0081] Figure 8 It is a schematic block diagram of the SLAM-based three-dimensional dense surface element mapping system according to an embodiment of the present invention.

[0082] Fig. 9 A block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0083] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0084] In the present invention, the term "one" in the claims and the specification should be understood as "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. Unless it is clearly indicated in the disclosure of the present invention that the number of the element is only one, the term "one" cannot be understood as unique or single, and the term "one" cannot be understood as a limitation on the quantity.

[0085] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0086] In today's real-time simultaneous positioning and mapping systems, a sparse map consisting of feature points corresponding to three-dimensional space points is usually created. In order to provide enough information for navigation or interaction, the three-dimensional map must densely cover the environment, that is, obtain a dense map. At the same time, most three-dimensional map reconstruction solutions require computing resources such as GPUs to achieve accelerated calculations, and require additional closed-loop detection modules and reconstruction strategies to ensure the real-time and global consistency of dense map reconstruction. However, the complex calculation process makes it difficult to deploy three-dimensional dense map algorithms on mobile platforms with limited computing resources, resulting in the slow development of augmented reality devices and mobile robot-related applications.

[0087] Therefore, in order to maintain a tight model representation of the scene while the simultaneous positioning and mapping system is running on a mobile platform with limited computing resources, this application proposes a tightly coupled simultaneous positioning and facet mapping solution based on key frame construction classes such as back-end nonlinear optimization, which realizes real-time 3D map reconstruction based on superpixel segmentation and facet extraction, and maintains the global consistency of the reconstructed map through the key frame pose optimization information of the back-end of the SLAM system. In addition, this application also improves and optimizes the image SLIC superpixel segmentation and facet extraction process, so that the 3D mapping module can realize real-time segmentation of grayscale images and depth images on a single thread, and then extract and update the facet map, and the facet map indexed and organized by the key frames in SLAM can be deformed according to the key frame pose optimization to meet the ground Figure 1 In addition, the present application can also incrementally construct map grids by structural triangulation, thereby achieving real-time dense map Figure 3 Corner grid data output.

[0088] Specifically, the positioning module in the simultaneous positioning and mapping system involved in the present invention is constructed based on the current mainstream back-end nonlinear optimization SLAM algorithm framework, and a three-dimensional map reconstruction module is designed to realize the dense map construction function. For example, Figure 1As shown in the figure, the positioning module includes a front-end thread for extracting and tracking image feature points, and a back-end sliding window optimization thread for calculating and optimizing the pose of continuous ordinary image frames by using a sliding window method. At the same time, the ordinary frames are extracted as key frames and transmitted to the back-end global optimization thread for loop detection and global optimization to obtain globally consistent camera poses and feature point spatial positions, which are fed back to the sliding window to improve the accuracy of local optimization. The mapping module is divided into three threads, wherein thread one is responsible for preprocessing the image and pose data transmitted by the positioning module, obtaining synchronized ordinary image frame poses and corresponding grayscale images and depth images; thread two is responsible for extracting new image frame information and integrating it into the global facet map; thread three is responsible for obtaining and outputting structured grids from the updated facet map.

[0089] In other words, the main technical solution of the present application is to obtain the pose information of the ordinary image frame by sliding window optimization in the SLAM backend, and then combine the key frame pose update information of the backend global optimization, and the sensor image data to output to the mapping module. The mapping module first processes the input sensor image data to obtain aligned grayscale images and depth images; at this time, if the key frame pose update information after global optimization is passed into the SLAM backend, the sub-image of the corresponding key frame is deformed. Then, the mapping module extracts superpixels on the aligned grayscale image and depth image, and extracts face elements from the superpixels in combination with the image frame pose information, and then merges them into the global face element map. Finally, a structured grid is incrementally constructed and output from the updated face element map.

[0090] Schematic method

[0091] Reference to the attached drawings of the specification Figures 2 to 7 As shown, a 3D dense surface element mapping method based on SLAM according to an embodiment of the present invention is explained. Specifically, Figure 2 As shown, the SLAM-based three-dimensional dense surface element mapping method may include the following steps:

[0092] S100: preprocessing incoming image frame data and posture data to obtain synchronized image frame posture information and corresponding image information, wherein the image information includes aligned grayscale images and depth images;

[0093] S200: performing superpixel segmentation, surfel extraction and fusion processing on the image information in combination with the image frame pose information in the same thread to obtain an updated surfel map; and

[0094] S300: Performing structured mesh processing on the updated surfel map to output triangular mesh data of a dense map in real time.

[0095] It is worth noting that, since the three-dimensional dense surface element mapping method based on SLAM of the present application is to extract the surface element by superpixel segmentation of the grayscale image and the depth image, so that the three-dimensional mapping module can realize the real-time segmentation of the grayscale image and the depth image on a single thread (i.e., the same thread) and then extract and fuse the surface element map, the three-dimensional dense surface element mapping method based on SLAM of the present application enables the surface element map establishment process to be calculated on the CPU without the need for computing resources such as GPU, which is convenient for deployment on embedded mobile platforms. At the same time, the three-dimensional dense surface element mapping method based on SLAM of the present application realizes real-time three-dimensional map reconstruction based on superpixel segmentation and surface element extraction, so as to maintain the global consistency of the reconstructed map through the key frame pose optimization information of the back end of the SLAM system.

[0096] In addition, if Figure 1 As shown, the front-end visual odometer of the SLAM system positioning module can track the feature points of the previous frame on the continuous image frames input by the image sensor through the pyramid LK optical flow method. When the number of tracked feature points is difficult to maintain tracking stability, a suitable number of feature points are newly extracted on the current image frame, and then the image frame and feature point data are transmitted to the back end; the back end can estimate and optimize the spatial positions of a fixed number of continuous image frames and their feature points by means of sliding window local optimization. When a new image frame to be processed (i.e., a new image frame) is transmitted, the oldest image frame in the sliding window is marginalized. At the same time, when the back-end data is transmitted, a suitable ordinary image frame is extracted as a key frame and stored in the key frame database. When a loop is detected or a new key frame is stored, a global optimization is performed to nonlinearly optimize the state variables including all key frames and their feature points; then, the globally consistent optimization results are fed back to the sliding window optimization for correction.

[0097] In other words, the SLAM backend of the present application can calculate and optimize the pose of continuous ordinary image frames in a sliding window manner to obtain the pose estimate of the current ordinary image frame; at the same time, compared with the sliding window optimization that must be performed every time a new ordinary image frame is passed in, the global optimization is generally performed only when a loop is detected in the key frame database or a new key frame is stored. After the sliding window optimization is completed, the local optimization thread outputs the pose of the ordinary image frame and its reference key frame sequence number as pose data and inputs it into the map building module receiving thread; if global optimization occurs at this time, since the key frame pose after global optimization will be passed in to the local optimization thread, the optimized pose of the key frame and the optimized key frame sequence number will be output from the local optimization thread to the receiving thread of the map building module; at the same time, after the back-end global optimization is executed, if it is detected that a key frame contains too much repeated observation information, the key frame will be deleted from the key frame database to improve the global optimization efficiency, and the redundant key frame sequence number will also be passed in to the map building module. At the same time, the image data corresponding to the ordinary image frame is obtained and passed in as image data.

[0098] According to the above-mentioned embodiments of the present application, in the step S100 of the SLAM-based three-dimensional dense surface mapping method of the present application, after receiving new image data, different preprocessing processes need to be performed according to the different output sensor image data. This is because the front end of the SLAM positioning module only needs to perform dedistortion operations on the pixel coordinates of the feature points when extracting feature points. Therefore, in order to obtain aligned dedistorted grayscale images and depth images, specific preprocessing operations need to be performed on the new images.

[0099] For example, Figure 3 As shown, the step S100 of the SLAM-based three-dimensional dense surface element mapping method of the present application may include the following steps:

[0100] S110: when the incoming image frame data is a binocular image, firstly perform binocular epipolar correction to obtain a dedistorted image, and then calculate a disparity image corresponding to the left-eye image by stereo matching, so as to obtain the depth image by binocular camera baseline calculation; and

[0101] S120: When the incoming image frame data is a color image and a depth sensor image, the color image is first converted into a grayscale image and dedistorted, and then aligned according to the external parameters between the cameras and the internal parameters of the RGB sensor to obtain an aligned depth image and disparity image.

[0102] Preferably, in the step S120 of the SLAM-based three-dimensional dense surface element mapping method of the present application, when the SLAM positioning module is in RGBD mode (that is, the incoming image frame data is a color image and a depth sensor image), its registration model is implemented as follows:

[0103] [u, v, d] c =K_c×T_c_d×K_d.inv()×[u, v, d] d ; where [u,v,d] d The pixel coordinates and corresponding depth values ​​in the depth sensor coordinate system are converted into a three-dimensional point cloud through the depth camera intrinsic parameter inverse matrix Kd.inv(), and then further calculated through the inter-camera external parameter T_c_d. and the RGB camera intrinsic parameter K_c to obtain the depth value [u, v, d] of the pixel point coordinates in the RGB image system. c .

[0104] It is worth noting that, after receiving new pose data, the step S100 of the SLAM-based three-dimensional dense face mapping method of the present application needs to find the pre-processed grayscale image and depth image (or disparity image) corresponding to the pose, and combine it with the reference key frame number to pass it into the processing thread for subsequent processing. It can be understood that if the pose after key frame optimization and the optimized key frame number are passed in, the face element map is deformed, wherein the face element map can be composed of sub-graphs indexed by each key frame number, and the face element in each sub-graph is indexed by the reference key frame number when it was extracted. Therefore, the present application finds the face element in the sub-graph corresponding to the pose optimization key frame number, and updates the face element coordinates and normal vectors according to the pose changes before and after the key frame optimization; at the same time, the state of these updated face elements is changed to change (i.e., change), which means that the grid needs to be updated again when the grid is structured to avoid direct deformation of the face element causing map discontinuity. In addition, if a redundant key frame number is passed in, since the redundant key frame is defined as having more repeated visual observation information and less valid visual observation information, the face element extracted using it as the index should also be redundant and be deleted directly.

[0105] In particular, according to the above embodiments of the present application, Figure 4A As shown, the step S200 of the SLAM-based three-dimensional dense surface element mapping method of the present application may include the steps of:

[0106] S210: performing super-pixel segmentation processing on the grayscale image in combination with the depth image to obtain super-pixel data of a new image frame;

[0107] S220: extracting surfels from the incoming image frame according to the superpixel data of the incoming image frame to obtain surfels to be fused of the incoming image frame; and

[0108] S230: performing a surfel update process on the current surfel map according to the surfels to be fused of the new image frame to obtain an updated surfel map.

[0109] Preferably, if Figure 4BAs shown, the step S210 of the SLAM-based three-dimensional dense surface element mapping method of the present application may include the steps of:

[0110] S211: Initializing the image grids in the depth image and the grayscale image to obtain initial superpixels of a new image frame;

[0111] S212: classifying the pixels in the new image frame according to the initial superpixels of the new image frame to obtain classified pixels of the new image frame; and

[0112] S213: performing iterative updating processing on the initial superpixels of the new image frame according to the classified pixels of the new image frame to obtain superpixel data of the new image frame with converged data.

[0113] Exemplarily, the step S210 of the SLAM-based three-dimensional dense surface element mapping method of the present application can perform superpixel segmentation processing on the grayscale image and the depth image based on the SLIC image superpixel segmentation principle, and the i-th superpixel segmented has the attribute SP i :[x i ,y i , I i , d i ], where x i and i are the pixel coordinates of the i-th superpixel in the current image frame, I i and d i are the grayscale value and depth value of the i-th superpixel in the current image frame respectively. Specifically, Figure 5A and Figure 5B As shown, with Block-size as the side length of the square grid, the grayscale image and the depth image are divided into equal distances, and the superpixels are initialized at the center of each grid, wherein the pixel coordinates of the initial superpixel are equal to the pixel coordinates of the center pixel of the grid, the grayscale value of the initial superpixel is equal to the grayscale value corresponding to the center pixel on the grayscale image, and the depth value of the initial superpixel is equal to the depth value corresponding to the center pixel on the depth image. It can be understood that if the depth value corresponding to the current center pixel on the depth image is missing or does not exist, the present application can re-search for a valid depth value within the range of the grid and its surrounding grids (i.e. [-block-size, +block-size]) and assign it to the superpixel attribute SP i .

[0114] It is worth noting that since the depth image obtained by the binocular camera through stereo matching usually has defects and holes, and the field of view of the depth sensor (such as TOF camera, etc.) is much smaller than that of the color camera, most areas of the aligned depth image do not have depth values. Therefore, when no valid depth value can be found within the range of the grid, the step S211 of the SLAM-based three-dimensional dense surface element mapping method of the present application can be used to replace the current initial superpixel SP i All pixels within the current grid are marked as invalid. Figure 5A and Figure 5B The gray area in the image can be directly skipped during the subsequent superpixel segmentation and surface element extraction to improve efficiency.

[0115] In addition, the step S212 of the SLAM-based three-dimensional dense surface element mapping method of the present application can classify the pixels by the k-means clustering algorithm. For example, the pixels in the center surrounding range of the initial superpixel (i.e., [-block-size, +block-size]) are first calculated by the SLIC superpixel segmentation algorithm to obtain a five-dimensional feature vector [x i ,y i , I i , a i , b i ]; then, the pixel point is classified into the superpixel with the minimum distance metric value to obtain the corresponding classified pixel point.

[0116] Exemplarily, the calculation formula of the distance metric value may be implemented as, but not limited to, the following formulas (1), (2) and (3):

[0117]

[0118]

[0119]

[0120] As can be seen from the above: the distance metric mainly calculates the Euclidean distance of space and color, and at the same time divides by N respectively C and N S To normalize.

[0121] Correspondingly, when superpixel segmentation is performed on the grayscale image and the depth image, the calculation formula of the corresponding distance metric value may be, but is not limited to, as shown in the following equations (4) and (5):

[0122]

[0123]

[0124] where u x ,u y ,u I ,u d Represent the pixel coordinates, grayscale value and depth value of the pixel point respectively. It can be understood that when there is a valid depth value for the pixel point and the superpixel, the pixel point is calculated and classified according to formula (5), and when there is no valid depth value for either the pixel point or the superpixel, the pixel point is calculated and classified according to formula (4).

[0125] It is worth noting that since the above distance metric value calculation is performed on all pixels on the grayscale image and the depth image, in order to accelerate the computational efficiency of the pixel classification process, the present application has made the following improvements: First, for the above formula (5), the Euclidean distance between the inverse depth of the pixel and the superpixel is calculated by floating-point division to perform superpixel segmentation on the grayscale image and the depth image. The present application proposes to directly calculate the distance metric value on the grayscale image and the disparity image to save the division operation and reduce the computational complexity. At the same time, the disparity image itself is a sixteen-bit integer, which is convenient for fixed-point integerization; secondly, since the superpixels without valid depth values ​​are marked as invalid in the implementation scheme of the present application and do not need to be calculated, the present application uniformly calculates the distance metric value by formula (5), and after calculating the depth distance, multiplies it by a coefficient S determined by whether the pixel has a valid depth value. d , to reduce conditional judgments and facilitate parallelization; third, change the remaining floating-point operations to integer operations, that is, the remaining coordinates and grayscale distance calculations in the formula are fixed-point to sixteen-bit integers, and the normalization coefficients are removed to convert them into integer multiplication, which helps to reduce floating-point operations and facilitate SIMD parallelization; finally, since the Euclidean distance needs to calculate the square, the sixteen-bit integer multiplication needs to put the result in a thirty-two-bit integer first.

[0126] In summary, the SLAM-based three-dimensional dense surface element mapping method of the present application further proposes to change the calculation of Euclidean distance to the calculation of Manhattan distance, that is, to calculate the absolute value of the distance difference, which helps to improve the calculation efficiency, wherein the calculation formula of the distance metric value is preferably implemented as the following formula (6):

[0127] D=[|x i -u x |+|y i -u y |]×F S +|I i -u I |×F C +|disp i -u disp |×Fd ×S d (6)

[0128] where u x ,u y ,u I and u disp Represent the pixel coordinates, grayscale value and disparity of the current pixel respectively; x i ,y i ,I i and disp i Represent the pixel coordinates, grayscale value and disparity of the current superpixel respectively; F S 、F C 、F d and S d are the corresponding weight coefficients respectively.

[0129] It is worth mentioning that in the pixel classification process of SLIC superpixel segmentation, it involves solving and calculating the distance metric between each pixel and the surrounding 4-connectivity superpixel center. There are usually two strategies: one is Fig. 6A As shown, first traverse and inspect all superpixels to obtain the current superpixel SP i , and then continue to traverse all pixels in the range [-block-size, +block-size] around the superpixel to calculate the current pixel P i and the current superpixel SP i The distance metric value is, if the current pixel point P i The stored minimum distance metric is updated, and the pixel is classified into the superpixel SP i , the superpixel number it is classified to is assigned to the superpixel number; secondly, Figure 6B As shown, all pixels on the image are traversed. Since the local search range of superpixels is [-block-size, +block-size], the distance from the pixel point P is found. i The distance metrics of the four superpixels initialized in the range of [-block-size, +block-size] are calculated respectively and the pixel point P is classified according to the calculated minimum value. i .

[0130] However, although the above strategy 1 is convenient for extracting pixels around superpixels for SIMD parallel computing, its drawback is that some pixels on the final image may not be classified into superpixels, which makes the segmentation result discontinuous and needs to be improved through algorithms such as connected region marking. Although the above strategy 2 can ensure that all pixels are classified into appropriate superpixels, it is necessary to find the four closest superpixels first, which is not conducive to parallel computing. Therefore, Figure 4C As shown, the step S212 of the SLAM-based three-dimensional dense surface element mapping method of the present application may include the steps of:

[0131] S2121: traverse the image grid to extract the current pixel point from the current grid;

[0132] S2122: In response to the current pixel being marked as invalid, directly skipping the current grid;

[0133] S2123: In response to the current pixel being not marked as invalid, dividing the current grid into a plurality of sub-grids to determine a current super-pixel for which a distance metric value needs to be calculated; and

[0134] S2124: By calculating the distance metric value between the current pixel and the determined superpixel, the current superpixel with the minimum distance metric value to the current pixel is selected for classification to obtain the corresponding classified pixel.

[0135] For example, Figure 6C As shown, traverse the image grid, taking the current grid block-size*block-size as an example, if the extracted pixel point is invalid, skip the grid directly, otherwise divide the current grid into four sub-grids. Assume that the initialization superpixel SP of the current grid i The number of superpixels in each row of the image is sp_stride, then the superpixels in the upper left subgrid that need to calculate the distance metric have the same number, which is i, i-1, i-sp_stride, and i-sp_stride-1, and so on. At the same time, the pixels in the upper left and upper right subgrids need to calculate the distance values ​​of the superpixels with numbers i and i-sp_stride, and the pixels in the lower left and lower right subgrids need to calculate the distance values ​​of the superpixels with numbers i and i+sp_stride, so that it is easy to take out the continuous pixels in each row in the current (sub) grid for parallel calculation.

[0136] According to the above embodiments of the present application, Figure 4D As shown, the step S213 of the SLAM-based three-dimensional dense surface element mapping method of the present application may include the steps of:

[0137] S2131: traverse the image grid to extract a current superpixel from the current grid;

[0138] S2132: In response to the current superpixel being marked as invalid, directly skipping the current grid;

[0139] S2133: In response to the current superpixel not being marked as invalid, by traversing the pixel points in the current grid and the surrounding grids, solving the average values ​​of the pixel coordinates, grayscale values, and depth values ​​of the pixel points classified into the current superpixel, so as to update the attributes of the current superpixel; and

[0140] S2134: Repeat the pixel classification step and the superpixel update step until the superpixel data of the new image frame with data convergence is obtained.

[0141] It is worth noting that after the pixels of the image are classified, a superpixel update step is required. For example, first, the image grid is traversed, and if the superpixel SP corresponding to the current grid i If the current grid corresponds to a superpixel SP i If it is not marked as invalid, the pixels in the range of [-block-size, +block-size] are traversed and classified into the current superpixel SP i The pixel coordinates, grayscale and depth values ​​of the pixel points are averaged, and its attributes [x i ,y i , I i , d i ]; Finally, according to the SLIC superpixel segmentation principle, the above steps S212 and S213 will be repeatedly calculated several times in the hope of convergence. In particular, the SLAM-based three-dimensional dense surface element mapping method of the present application further adopts row sampling in the calculation process of steps S212 and S213, which will reduce the time consumption of pixel classification and superpixel update steps while not affecting SIMD parallel computing. For example, Figure 7 As shown in the figure, assuming that row_stride (i.e., sampling interval) is 2, when the number of iterations is odd, the pixels in the odd rows of the grid are classified, and when the superpixel is updated, the pixels in the odd rows of the grid are also retrieved for update; similarly, the situation is similar when the number of iterations is even.

[0142] According to the above embodiment of the present application, after the SLIC superpixel segmentation algorithm of the grayscale image combined with the depth (or disparity) image is completed in step S210, the step S220 of the present application extracts the surface element based on the superpixel, which will greatly save the calculation amount of the three-dimensional surface element. In order to speed up the surface element extraction process of the new image frame, as Figure 4EAs shown, the step S220 of the SLAM-based three-dimensional dense surface element mapping method of the present application may include the steps of:

[0143] S221: traverse the superpixels that are not marked as invalid, retrieve the pixel points classified into the current superpixel within the range around the center of the current superpixel, and store the pixel coordinates and depth values ​​of the pixel points with valid depth values ​​in the container as internal points;

[0144] S222: In response to the number of internal points in the container being less than a number threshold, skipping the current superpixel; and

[0145] S223: In response to the number of inner points in the container being greater than or equal to the number threshold, the three-dimensional spatial coordinates of the inner points are calculated through a back-projection model according to the camera parameters and the pixel coordinates and depth values ​​of the inner points, so as to use the plane formed by the inner points in the three-dimensional space as the face element of the new image frame.

[0146] It is worth noting that since the spatial points on the plane should satisfy the plane equation: ax i +by i +cz i =d, where coefficients a, b and c are the plane normal vectors, so the normal vector can be obtained by SVD decomposition of a matrix composed of any four point clouds. In order to maximize the number of interior points in the plane equation, the present application may adopt the groupSAC algorithm, first calculating the parallax histogram distribution of the interior points, and then classifying and randomly sampling the interior points according to the distribution to solve the plane equation with the largest number of interior points, and finally performing SVD decomposition of the matrix composed of all groupSAC interior points to obtain the plane equation, that is, to obtain the face element normal vector. It can be understood that the three-dimensional coordinates of the face element in the present application can be obtained by back-projection of the superpixel center coordinates and corrected according to its normal vector.

[0147] In addition, the surface element Surfel has the property [p S ,n S ,I S ,σ S ,r S ,f S ], which respectively represent three-dimensional coordinates, normal vector, gray value, confidence, radius and reference key frame, wherein the three-dimensional coordinates and normal vector of the face element extracted from the new image frame are obtained through the above step S223, the gray value is equal to the gray value of the superpixel corresponding to the face element, the confidence is 1, the radius is the distance of the face element farthest from the surrounding eight-connectivity (8-connectivity), and the reference key frame is the reference key frame number of the corresponding image frame. The fused face element map is to fuse the face element extracted from the new image frame with the face element of the current face element map. Specifically, Figure 4FAs shown, the step S230 of the SLAM-based three-dimensional dense surface element mapping method of the present application may include the steps of:

[0148] S231: traversing all existing bins in the current bin map to transform and project the three-dimensional coordinates of the current existing bins into the image coordinate system of the new image frame;

[0149] S232: In response to the image coordinates of the current existing bin exceeding the image range or the corresponding superpixel being marked as invalid, skipping the current existing bin;

[0150] S233: In response to the superpixel corresponding to the current existing surfel not being marked as invalid and the depth value and the normal vector between the two satisfying the threshold judgment condition, fusing the current existing surfel with the corresponding surfel to be fused by weighted averaging to obtain a fused surfel; and

[0151] S234: updating and expanding the current surfel map according to the fused surfels and the surfels in the new image frame that are not involved in the fusion, so as to obtain the updated surfel map.

[0152] For example, all existing surfels in the current surfel map are traversed. j , transform and project its 3D coordinates to the image coordinate system of the current image frame. If it exceeds the image range or the corresponding pixel is marked as invalid, it is skipped; otherwise, find the superpixel SP to which the pixel is classified i As a surface element Surfel i , calculate whether the depth value and normal vector between the two meet the threshold judgment condition, and if not, the existing surface element Surfel j The confidence of is minus one. If it is satisfied, the newly extracted surfel is fused with the corresponding existing surfel in the current surfel map by weighted average. The specific formulas (7), (8), (9) and (10) are as follows:

[0153]

[0154]

[0155]

[0156] σ j =min{σ j +w,σ max} (10)

[0157] The weight w is obtained by dividing the number of valid pixels classified into the face element Surfeli by block-size*block-size; and the radius of the face element after fusion is taken as the smaller value of the two radii. It can be understood that if the confidence of a face element in the face element map is less than 0, the face element is deleted; at the same time, in the face element map, the state mark of the face element fused with the face element extracted from the new image frame is changed to change, indicating that the grid needs to be updated when the grid is structured; and the remaining newly extracted face elements that have not been fused with the face element map are marked as free, indicating that the part of the structured grid corresponding to the current face element map needs to be expanded.

[0158] According to the above-mentioned embodiments of the present application, in the step S300 of the SLAM-based three-dimensional dense surface element mapping method of the present application, after the surface element map is updated, the surfaces marked as change and free (that is, the updated surface element map) are passed to the next thread to complete the structured mesh update and output, wherein the triangular mesh output frequency is relatively lower than the input image frame rate.

[0159] It is worth noting that the facets in the updated facet map are preferably stored in an octree data structure, so that other facets within the facet radius can be quickly and accurately retrieved. Specifically, in the incremental meshing process, the facets around the facet marked as free are searched and local incremental triangulation is performed; and in the mesh update process, the meshes with the facets marked as change as vertices are deleted, and these facets are re-triangulated.

[0160] In addition, based on the above typical implementation methods, a specific example of a nonlinear optimization SLAM system and a 3D map reconstruction module is built, and the effect of running a binocular or RGBD data set on a PC platform or an embedded platform is tested. The results show that the SLIC superpixel segmentation and facet extraction optimization strategy proposed in this application can realize the processing of incoming image frames in real time on a single thread, and can combine the global optimization keyframe pose results of the SLAM system to obtain a facet map with global consistency.

[0161] In summary, due to the complex computational process in the prior art, it is difficult to deploy three-dimensional dense map algorithms on mobile platforms with limited computing resources, resulting in a slowdown in the development of applications related to augmented reality devices and mobile robots. Therefore, the present application proposes a simultaneous positioning and facet mapping solution based on key frame construction classes such as back-end nonlinear optimization, which realizes real-time three-dimensional map reconstruction based on superpixel segmentation and facet extraction, while maintaining the global consistency of the reconstructed map through the key frame pose optimization information of the back-end of the SLAM system. In addition, on the one hand, by improving and optimizing the SLIC image superpixel segmentation and facet extraction process, the three-dimensional mapping module can realize real-time segmentation of grayscale images and depth images on a single thread to extract facets. On the other hand, the three-dimensional facet map indexed and organized by key frames can be quickly deformed according to the key frame pose optimization to meet the ground. Figure 1 In particular, the present application also incrementally constructs a three-dimensional map by means of structural gridding, while compensating for the discontinuity defects caused by the facet map.

[0162] Schematic system

[0163] Reference to the attached drawings of the specification Figure 8 As shown, a 3D dense surface element mapping system based on SLAM according to an embodiment of the present invention is explained. Specifically, Figure 8 As shown, the SLAM-based three-dimensional dense face mapping system 1 may include: a preprocessing module 10, which is communicatively connected to each other, for preprocessing incoming image frame data and pose data to obtain synchronized image frame pose information and corresponding image information, wherein the image information includes aligned grayscale images and depth images; a face processing module 20, which is used to perform superpixel segmentation, face extraction and fusion processing on the image information in combination with the image frame pose information in the same thread to obtain an updated face map; and a meshing processing module 30, which is used to perform structured meshing processing on the updated face map to output triangular mesh data of the dense map in real time.

[0164] More specifically, if Figure 8 As shown, the facet processing module 20 may include a superpixel segmentation module 21, a facet extraction module 22 and a facet update module 23 which are communicatively connected to each other, wherein the superpixel segmentation module 21 is used to perform superpixel segmentation processing on the grayscale image in combination with the depth image to obtain superpixel data of the new image frame; wherein the facet extraction module 22 is used to extract facets from the new image frame according to the superpixel data of the new image frame to obtain facets to be fused of the new image frame; wherein the facet update module 23 is used to perform facet update processing on the current facet map according to the facets to be fused of the new image frame to obtain an updated facet map.

[0165] In one example of this application, Figure 8 As shown, the superpixel segmentation module 21 includes an initialization module 211, a classification module 212 and an iterative update module 213 which are communicatively connected to each other, wherein the initialization module 211 is used to initialize the image grids in the depth image and the grayscale image to obtain the initial superpixels of the new image frame; wherein the classification module 212 is used to classify the pixel points in the new image frame according to the initial superpixels of the new image frame to obtain the classified pixel points of the new image frame; wherein the iterative update module 213 is used to iteratively update the initial superpixels of the new image frame according to the classified pixel points of the new image frame to obtain the superpixel data of the new image frame with data convergence.

[0166] In one example of the present application, the classification module 212 is further used to: traverse the image grid to extract the current pixel point from the current grid; in response to the current pixel point being marked as invalid, directly skip the current grid; in response to the current pixel point not being marked as invalid, divide the current grid into multiple sub-grids to determine the current superpixel that needs to calculate the distance metric value; and by calculating the distance metric value between the current pixel point and the determined superpixel, classify the current superpixel with the minimum distance metric value with the current pixel point to obtain the corresponding classified pixel point.

[0167] Preferably, the calculation formula of the distance metric value is implemented as:

[0168] where u x ,u y ,u I and u disp Represent the pixel coordinates, grayscale value and disparity of the current pixel respectively; x i ,y i ,I i and disp i Represent the pixel coordinates, grayscale value and disparity of the current superpixel respectively; F S 、F C 、F d and S d are the corresponding weight coefficients respectively.

[0169] In one example of the present application, the iterative update module 213 is further used to: traverse the image grid to extract the current superpixel from the current grid; in response to the current superpixel being marked as invalid, directly skip the current grid; in response to the current superpixel not being marked as invalid, by traversing the pixel points in the current grid and its surrounding grids, solve the pixel coordinates, grayscale values ​​and depth values ​​of the pixel points classified into the current superpixel to update the properties of the current superpixel; and repeat the above-mentioned pixel classification step and the above-mentioned superpixel update step until the superpixel data of the new image frame with data convergence is obtained.

[0170] In one example of this application, Figure 8 As shown, the face element extraction module 22 includes a pixel point retrieval module 221, a superpixel skipping module 222 and a back projection calculation module 223 which are communicatively connected to each other, wherein the pixel point retrieval module 221 is used to traverse the superpixels that are not marked as invalid, retrieve the pixel points classified into the current superpixel within the range around the center of the current superpixel, so as to store the pixel coordinates and depth values ​​of the pixel points with valid depth values ​​in a container as internal points; wherein the superpixel skipping module 222 is used to skip the current superpixel in response to the number of internal points in the container being less than a quantity threshold; wherein the back projection calculation module 223 is used to calculate the three-dimensional spatial coordinates of the internal point through a back projection model according to the camera parameters and the pixel coordinates and depth values ​​of the internal point in response to the number of internal points in the container being greater than or equal to the quantity threshold, so as to use the plane formed by the internal point in the three-dimensional space as the face element of the new image frame.

[0171] In one example of this application, Figure 8 As shown, the facet update module 23 includes a facet projection module 231, a facet skipping module 232, a facet fusion module 233 and an update expansion module 234 which are communicatively connected to each other, wherein the facet projection module 231 is used to traverse all existing facets in the current facet map to convert and project the three-dimensional coordinates of the current existing facets to the image coordinate system of the new image frame; wherein the facet skipping module 232 is used to skip the current existing facet in response to the image coordinates of the current existing facet exceeding the image range or the corresponding superpixel mark being marked as invalid; wherein the facet fusion module 233 is used to fuse the current existing facet with the corresponding facet to be fused by weighted averaging in response to the superpixel corresponding to the current existing facet not being marked as invalid and the depth value and normal vector between the two satisfying the threshold judgment condition, so as to obtain the fused facet; wherein the update expansion module 234 is used to update and expand the current facet map according to the fused facet and the facets not involved in the fusion in the new image frame, so as to obtain the updated facet map.

[0172] Schematic electronic equipment

[0173] Below, reference Fig. 9 An electronic device according to an embodiment of the present invention is described below. Fig. 9 As shown, the electronic device 90 includes one or more processors 91 and a memory 92 .

[0174] The processor 91 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions. In other words, the processor 91 includes one or more physical devices configured to execute instructions. For example, the processor 91 may be configured to execute instructions as part of the following: one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform tasks, implement data types, convert the state of one or more components, implement technical effects, or otherwise obtain desired results.

[0175] The processor 91 may include one or more processors configured to execute software instructions. In addition or in lieu thereof, the processor 91 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processor of the processor 91 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel and / or distributed processing. The various components of the processor 91 may optionally be distributed on two or more separate devices, which may be remotely located and / or configured to perform collaborative processing. Various aspects of the processor 91 may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0176] The memory 92 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute the program instructions to implement some or all of the steps in the above-mentioned schematic method of the present invention described above, and / or other desired functions.

[0177] In other words, the memory 92 includes one or more physical devices configured to store machine-readable instructions that can be executed by the processor 91 to implement the methods and processes described herein. When implementing these methods and processes, the state of the memory 92 can be changed (for example, to store different data). The memory 92 may include removable and / or built-in devices. The memory 92 may include optical storage (for example, CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (for example, RAM, EPROM, EEPROM, etc.) and / or magnetic storage (for example, hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The memory 92 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable and / or content addressable devices.

[0178] It is understood that the memory 92 includes one or more physical devices. However, various aspects of the instructions described herein may alternatively be propagated through a communication medium (e.g., electromagnetic signals, optical signals, etc.) that is not held by a physical device for a limited period of time. Various aspects of the processor 91 and the memory 92 may be integrated together into one or more hardware logic components. These hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application-specific integrated circuits (PASIC / ASIC), program and application-specific standard products (PSSP / ASSP), systems on chip (SOCs), and complex programmable logic devices (CPLDs).

[0179] In one example, if Fig. 9 As shown, the electronic device 90 may also include an input device 93 and an output device 94, which are interconnected by a bus system and / or other forms of connection mechanisms (not shown). For example, the input device 93 may be, for example, a camera module for collecting image data or video data, etc. As another example, the input device 93 may include or dock with one or more user input devices such as a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input device 93 may include or dock with a selected natural user input (NUI) component. Such component parts may be integrated or peripheral, and the transduction and / or processing of input actions may be processed on or off the board. Example NUI components may include microphones for language and / or speech recognition; infrared, color, stereoscopic display and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for evaluating brain activity and / or body movement; and / or any other suitable sensors.

[0180] The output device 94 can output various information to the outside, including classification results, etc. The output device 94 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0181] Of course, the electronic device 90 can also further include the communication device, wherein the communication device can be configured to couple the electronic device 90 to one or more other computer devices. The communication device can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or a wide area network. In certain embodiments, the communication device can allow the electronic device 90 to send a message to other devices and / or receive a message from other devices via a network such as the Internet.

[0182] It will be appreciated that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above process may be changed.

[0183] Of course, to simplify, Fig. 9 Only some of the components related to the present invention in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.

[0184] It should also be noted that in the device, apparatus and method of the present invention, each component or each step can be decomposed and / or reassembled. Such decomposition and / or reassembly should be regarded as equivalent solutions of the present invention.

[0185] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0186] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

Claims

1. A 3D dense surface element mapping method based on SLAM, characterized in that: Includes steps: Preprocessing incoming image frame data and pose data to obtain synchronized image frame pose information and corresponding image information, wherein the image information includes aligned grayscale images and depth images; In the same thread, the image information is combined with the image frame pose information to perform superpixel segmentation, surface element extraction and fusion processing to obtain an updated surface element map; as well as Performing structured mesh processing on the updated surface element map to output triangular mesh data of a dense map in real time; The step of performing superpixel segmentation, facet extraction and fusion processing on the image information in combination with the image frame pose information in the same thread to obtain an updated facet map comprises the following steps: Performing super-pixel segmentation processing on the grayscale image in combination with the depth image to obtain super-pixel data of a new image frame; Extracting facets from the incoming image frame according to the superpixel data of the incoming image frame to obtain facets to be fused in the incoming image frame; and According to the to-be-fused surfels of the new image frame, a surfel update process is performed on the current surfel map to obtain an updated surfel map; The step of extracting facets from the incoming image frame according to the superpixel data of the incoming image frame to obtain facets to be fused of the incoming image frame comprises the following steps: Traversing the superpixels that are not marked as invalid, retrieving the pixel points that are classified into the current superpixel within the range around the center of the current superpixel, and storing the pixel coordinates and depth values ​​of the pixel points with valid depth values ​​in the container as internal points; In response to the number of inliers in the container being less than a number threshold, skipping the current superpixel; and In response to the number of inner points in the container being greater than or equal to the number threshold, the three-dimensional spatial coordinates of the inner points are calculated through a back-projection model based on camera parameters and the pixel coordinates and depth values ​​of the inner points, so as to use the plane formed by the inner points in the three-dimensional space as the face element of the new image frame.

2. The SLAM-based three-dimensional dense surface element mapping method according to claim 1, wherein: The step of performing super-pixel segmentation processing on the grayscale image in combination with the depth image to obtain super-pixel data of the incoming image frame comprises the steps of: Initializing the image grids in the depth image and the grayscale image to obtain initial superpixels of a new image frame; Classifying the pixels in the new image frame according to the initial superpixel of the new image frame to obtain classified pixels of the new image frame; as well as According to the classified pixel points of the new image frame, the initial superpixel of the new image frame is iteratively updated to obtain the superpixel data of the new image frame with converged data.

3. The SLAM-based three-dimensional dense surface element mapping method according to claim 2, wherein: The step of classifying the pixels in the new image frame according to the initial superpixel of the new image frame to obtain classified pixels of the new image frame comprises the steps of: Traverse the image grid to extract the current pixel point from the current grid; In response to the current pixel being marked as invalid, directly skipping the current grid; In response to the current pixel being not marked as invalid, dividing the current grid into a plurality of sub-grids to determine a current super-pixel for which a distance metric value needs to be calculated; as well as By calculating the distance metric value between the current pixel and the determined superpixel, the current superpixel with the minimum distance metric value to the current pixel is selected for classification to obtain the corresponding classified pixel.

4. The SLAM-based three-dimensional dense surface element mapping method according to claim 3, wherein: The calculation formula of the distance metric is implemented as: D = [|x i - u x | + |y i - u y |] × F S + |I i - u I | × F C + |disp i - u disp | × F d × S d where u x , u y , u I and u disp Represent the pixel coordinates, grayscale value and disparity of the current pixel respectively; x i ,y i ,I i and disp i Represent the pixel coordinates, grayscale value and disparity of the current superpixel respectively; F S 、F C 、F d and S d are the corresponding weight coefficients respectively.

5. The SLAM-based three-dimensional dense surface element mapping method according to claim 4, wherein: The step of iteratively updating the initial superpixel of the new image frame according to the classified pixel points of the new image frame to obtain the superpixel data of the new image frame with data convergence comprises the steps of: Traversing the image grid to extract the current superpixel from the current grid; In response to the current superpixel being marked as invalid, directly skipping the current grid; In response to the current superpixel not being marked as invalid, by traversing the pixel points in the current grid and the surrounding grids, solving the average values ​​of the pixel coordinates, grayscale values, and depth values ​​of the pixel points classified into the current superpixel, so as to update the attributes of the current superpixel; and Repeat the above pixel classification step and the above super-pixel updating step until the super-pixel data of the new image frame with data convergence is obtained.

6. The SLAM-based three-dimensional dense surface element mapping method according to any one of claims 1 to 5, wherein: The step of performing a surfel update process on the current surfel map according to the surfels to be fused of the new image frame to obtain an updated surfel map comprises the following steps: Traversing all existing bins in the current bin map to transform and project the three-dimensional coordinates of the current existing bins into the image coordinate system of the new image frame; In response to the image coordinates of the current existing bin exceeding the image range or the corresponding superpixel label being marked as invalid, skipping the current existing bin; In response to the superpixel corresponding to the current existing surfel not being marked as invalid and the depth value and the normal vector between the two satisfying the threshold judgment condition, fusing the current existing surfel with the corresponding surfel to be fused by weighted averaging to obtain a fused surfel; as well as The current surfel map is updated and expanded according to the fused surfel and the surfels in the new image frame that are not involved in the fusion, so as to obtain the updated surfel map.

7. The SLAM-based three-dimensional dense surface element mapping method according to any one of claims 1 to 5, wherein: The step of preprocessing the incoming image frame data and posture data to obtain synchronized image frame posture information and corresponding image information, wherein the image information includes aligned grayscale images and depth images, comprises the steps of: When the incoming image frame data is a binocular image, binocular epipolar correction is first performed to obtain a dedistorted image, and then the disparity image corresponding to the left eye image is calculated by stereo matching to obtain the depth image by binocular camera baseline calculation; and When the incoming image frame data is a color image and a depth sensor image, the color image is first converted into a grayscale image and dedistorted, and then aligned according to the external parameters between the cameras and the internal parameters of the RGB sensor to obtain an aligned depth image and disparity image.

8. A 3D dense surface element mapping system based on SLAM, characterized in that: comprising communicatively connected to each other: a preprocessing module, configured to preprocess incoming image frame data and posture data to obtain synchronized image frame posture information and corresponding image information, wherein the image information includes aligned grayscale images and depth images; A facet processing module, used for performing superpixel segmentation, facet extraction and fusion processing on the image information in combination with the image frame pose information in the same thread to obtain an updated facet map; as well as a grid processing module, used for performing structured grid processing on the updated surface element map to output the triangular grid data of the dense map in real time; Wherein, the face element processing module comprises a superpixel segmentation module, a face element extraction module and a face element update module which are communicatively connected to each other, wherein the superpixel segmentation module is used to perform superpixel segmentation processing on the grayscale image in combination with the depth image to obtain superpixel data of the new image frame; wherein the face element extraction module is used to extract face elements from the new image frame according to the superpixel data of the new image frame to obtain face elements to be fused of the new image frame; wherein the face element update module is used to perform face element update processing on the current face element map according to the face elements to be fused of the new image frame to obtain an updated face element map; The face extraction module includes a pixel retrieval module, a superpixel skipping module and a back-projection calculation module which are communicatively connected to each other, wherein the pixel retrieval module is used to traverse superpixels that are not marked as invalid, retrieve pixel points classified into the current superpixel within the range around the center of the current superpixel, and store the pixel coordinates and depth values ​​of the pixel points with valid depth values ​​in a container as internal points; wherein the superpixel skipping module is used to skip the current superpixel in response to the number of internal points in the container being less than a quantity threshold; wherein the back-projection calculation module is used to calculate the three-dimensional spatial coordinates of the internal point through a back-projection model according to camera parameters and the pixel coordinates and depth values ​​of the internal point in response to the number of internal points in the container being greater than or equal to the quantity threshold, so as to use the plane formed by the internal point in the three-dimensional space as the face of the new image frame.

9. The SLAM-based three-dimensional dense surface element mapping system according to claim 8, wherein: The superpixel segmentation module includes an initialization module, a classification module and an iterative update module which are communicatively connected to each other, wherein the initialization module is used to initialize the image grids in the depth image and the grayscale image to obtain initial superpixels of a new image frame; The classification module is used to classify the pixels in the new image frame according to the initial superpixels of the new image frame to obtain classified pixels of the new image frame; The iterative update module is used to iteratively update the initial superpixel of the new image frame according to the classified pixel points of the new image frame to obtain superpixel data of the new image frame with data convergence.

10. The SLAM-based three-dimensional dense surface element mapping system according to claim 8 or 9, wherein: The facet update module comprises a facet projection module, a facet skip module, a facet fusion module and an update expansion module which are communicatively connected to each other, wherein the facet projection module is used to traverse all existing facets in the current facet map to transform and project the three-dimensional coordinates of the current existing facets to the image coordinate system of the new image frame; The face element skipping module is used to skip the current face element in response to the image coordinates of the current face element exceeding the image range or the corresponding superpixel label being marked as invalid; the face element fusion module is used to fuse the current face element with the corresponding face element to be fused by weighted averaging in response to the superpixel corresponding to the current face element not being marked as invalid and the depth value and normal vector between the two satisfying the threshold judgment condition to obtain a fused face element; the update and expansion module is used to update and expand the current face element map according to the fused face element and the face elements not involved in the fusion in the new image frame to obtain the updated face element map.

11. An electronic device, characterized in that: include: at least one processor for executing instructions; and A memory communicatively connected to the at least one processor, wherein the memory has at least one instruction, wherein the instruction is executed by the at least one processor so that the at least one processor performs the steps in the SLAM-based three-dimensional dense surface element mapping method as described in any one of claims 1 to 7.

Images