Three-dimensional map reconstruction method, mobile device, and computer-readable storage medium

Abstract

The present invention discloses a three-dimensional map reconstruction method, a self-mobile device, and a computer-readable storage medium. The three-dimensional map reconstruction method is applied to a self-mobile device and includes: determining an RGB image, a depth map, and an odometer pose at the same moment; determining a current local map based on the RGB image, depth map, and odometer pose at the same moment; matching the current local map with a global map to correct the odometer pose; updating the current local map based on the corrected odometer pose and a sliding window strategy; and fusing the updated current local map with the RGB image to update the global map. This three-dimensional map reconstruction method can reduce memory usage and computing power requirements during runtime and can be applied to mobile devices.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional map reconstruction method, a mobile device, and a computer-readable storage medium. Background Art

[0002] Today, the application of 3D reconstruction functions in self-moving devices (such as robots) is becoming increasingly widespread and mature. The principle of 3D reconstruction is: relying on sensors to perceive the surrounding environment, while describing the robot's own position and environment. By matching the robot's current observations with historical maps, the robot's current latest position can be determined. Once the latest position is determined, the latest observations can be applied to the current position to update and supplement the map.

[0003] However, the 3D map reconstruction methods in related technologies require high computing power, are time-consuming, and need to be processed offline or rely on an additional image processing device, a GPU (Graphics Processing Unit). They cannot be run in real time on mobile devices with limited computing power and memory. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To achieve the above-mentioned objectives, an embodiment of the first aspect of the present invention proposes a three-dimensional map reconstruction method, which is applied to a mobile device. The method includes: determining an RGB image, a depth map, and an odometer pose at the same moment; determining a current local map based on the RGB image, the depth map, and the odometer pose at the same moment; matching the current local map with a global map to correct the odometer pose; updating the current local map based on the corrected odometer pose and a sliding window strategy; and fusing the updated current local map with the RGB image to update the global map.

[0006] To achieve the above-mentioned objectives, an embodiment of the second aspect of the present invention proposes a mobile device, including an imaging device, a memory, a controller, and a three-dimensional reconstruction program stored in the memory and executable on the controller. When the controller executes the three-dimensional reconstruction program, the above-mentioned three-dimensional map reconstruction method is implemented.

[0007] To achieve the above objectives, a third embodiment of the present invention provides a computer-readable storage medium having a three-dimensional reconstruction program stored thereon. When the three-dimensional reconstruction program is executed by a processor, the three-dimensional map reconstruction method described above is implemented.

[0008] The 3D map reconstruction method, mobile device, and computer-readable storage medium of the embodiments of the present invention can, after determining the RGB image, depth map, and odometry pose at the same time, determine the current local map based on the RGB image, depth map, and odometry pose at the same time, and then match the current local map with the global map to correct the odometry pose to ensure the accuracy of the odometry pose and prevent the error of the odometry pose from affecting the 3D reconstruction. Moreover, after correcting the odometry pose, the current local map is also moved and updated based on the corrected odometry pose. Furthermore, the updated current local map is fused with the RGB image to update the global map, thereby significantly reducing the memory usage and computing power requirements during the 3D reconstruction process and improving the computational speed of the 3D map reconstruction method. This allows the 3D map reconstruction method of the embodiments of the present invention to be directly applied on mobile devices without the need for additional devices such as GPUs and high-end CPUs (Central Processing Units). Moreover, since the local map is constructed and used to update the global map, the computing power requirements for 3D reconstruction can be further reduced.

[0009] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of a mobile device according to an example of the present invention;

[0011] Figure 2 is a flow chart of a three-dimensional map reconstruction method according to an embodiment of the present invention;

[0012] Figure 3 is a flow chart of a three-dimensional map reconstruction method according to another embodiment of the present invention;

[0013] Figure 4 is a flow chart of a three-dimensional map reconstruction method according to another embodiment of the present invention;

[0014] Figure 5 is a flow chart of a three-dimensional map reconstruction method according to another embodiment of the present invention;

[0015] Figure 6 This is a flowchart of a three-dimensional map reconstruction method according to another embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0017] The following describes a three-dimensional map reconstruction method, a mobile device, and a computer-readable storage medium according to embodiments of the present invention with reference to the accompanying drawings.

[0018] Figure 1 FIG. 4 is a flowchart of a three-dimensional map reconstruction method according to an embodiment of the present invention.

[0019] In an embodiment of the present invention, the three-dimensional map reconstruction method is applied to a self-moving device, which may be, for example, an unmanned delivery vehicle, a drone, a warehouse robot, a shopping mall service robot, a food delivery robot, a sweeping robot, a sweeping and mopping robot, a floor washing robot, or other robots. Another example may be a handheld three-dimensional reconstruction scanner, a fixed three-dimensional reconstruction scanning device, a head-mounted AR device, or other device that can obtain RGB images, depth maps, and odometer postures.

[0020] As an example, the self-moving device is a sweeping and mopping robot. The sweeping and mopping robot 100 can be as follows: Figure 1 See Figure 1 The sweeping and mopping robot includes: a distance measuring component 102, a control device 104, a walking component 106, and an imaging device 108. The imaging device 108 is arranged at the front end of the sweeping and mopping robot, and can be, for example, an RGB camera and a depth camera, and then an RGB image is acquired by the RGB camera, and a depth image is acquired by the depth camera. At the same time, the distance measuring component 102 can be, for example, an odometer, and then the odometer posture is acquired through the distance measuring component 102 to obtain the current posture information of the sweeping and mopping robot. The posture information includes three-dimensional coordinate information and posture information. The three-dimensional coordinate information specifically includes the coordinates corresponding to the x-axis, y-axis, and z-axis. The posture information of the self-mobile device is the orientation information of the sweeping and mopping robot, which specifically includes pitch, yaw, and roll.

[0021] like Figure 1 As shown, the sweeping and mopping robot can be used to implement a three-dimensional map reconstruction method, which includes the following steps:

[0022] S11, determine the RGB image, depth image and odometry pose at the same moment.

[0023] Specifically, a depth map at the current moment and several RGB maps at adjacent moments are obtained; wherein the depth map and each of the several RGB maps carry a corresponding timestamp; the timestamp difference between each RGB map and the depth map in the several RGB maps is calculated; if the timestamp difference is not greater than a first time threshold and the number of RGB maps is one, the RGB map and the depth map are regarded as the RGB map and depth map at the same moment, and the posture corresponding to the RGB map is used as the odometer posture at the same moment; if the timestamp difference is not greater than the first time threshold and the number of RGB maps is at least two, the RGB map and the depth map corresponding to the latest timestamp are regarded as the RGB map and depth map at the same moment, and the timestamp of the depth map is used as the reference to determine the odometer posture at the same moment by linear interpolation.

[0024] Specifically, the timestamp of the most recently obtained depth map can be obtained, and several RGB maps of adjacent moments can be obtained based on the timestamp, and then the timestamp difference between the timestamps of the several RGB maps and the timestamp of the depth map can be calculated to obtain several timestamp differences, and the calculation results can be compared with the first time threshold. If only one timestamp difference is less than or equal to the first time threshold, the RGB map corresponding to the timestamp difference is regarded as the RGB map at the same time as the depth map, and the posture corresponding to the RGB map is used as the odometer posture at the same time as the depth map. If there are multiple timestamp differences less than or equal to the first time threshold, the most recently obtained RGB map among the multiple RGB maps corresponding to the multiple timestamp differences is obtained, and it is regarded as the RGB map at the same time as the depth map, and the odometer posture at the same time is determined by linear interpolation based on the timestamp of the depth map.

[0025] As an example, assuming the timestamp of the most recently acquired depth image is 1s, several RGB images at adjacent moments are acquired based on this timestamp. The differences between the timestamps of these RGB images and 1s are calculated to obtain several timestamp differences. Assuming the first time threshold is set to 0.2s, timestamp differences of 0.2s or less are obtained from these timestamp differences. If only one timestamp difference is less than or equal to 0.2s, the RGB image corresponding to this timestamp difference is considered to be the RGB image at the same moment as the depth image, and the pose corresponding to this RGB image is used as the odometry pose at the same moment as the depth image. If at least two timestamp differences exist that are less than or equal to 0.2s, the RGB image corresponding to the most recent timestamp is considered to be the RGB image at the same moment as the depth image. For example, if two RGB images exist with timestamps of 0.9s and 1.1s, the RGB image with the timestamp of 1.1s is considered to be the RGB image at the same moment as the depth image. The odometry pose at the same moment is then determined using linear interpolation, based on the timestamp of this depth image.

[0026] The above method of using the timestamp of the depth map as a reference to determine the odometer pose at the same time by linear interpolation can be found in Figure 3 , including the following steps:

[0027] S111, obtaining a first RGB image and a second RGB image having the smallest timestamp difference with the depth image; wherein the first RGB image carries a first timestamp and first pose information, and the second RGB image carries a second timestamp and second pose information.

[0028] Specifically, a first RGB image and a second RGB image with the smallest timestamp difference from the depth image are obtained, wherein the odometry pose of the first RGB image is (X1, Y1, Z1, roll1, pitch1, yaw1), and the odometry pose of the second RGB image is (X2, Y2, Z2, roll2, pitch2, yaw2), and the first timestamp of the first RGB image is earlier than the timestamp of the depth image, and the second timestamp of the second RGB image is later than the timestamp of the depth image. The above (X1, Y1, Z1) and (X2, Y2, Z2) are three-dimensional coordinate information representing the position of the mobile device corresponding to the timestamps of the first RGB image and the second RGB image, respectively. The above (roll1, pitch1, yaw1) and (roll2, pitch2, yaw2) are attitude information of the mobile device corresponding to the timestamps of the first RGB image and the second RGB image, respectively. The heading angle in the attitude information represents the angle of rotation along the X-axis, the yaw in the attitude information represents the angle of rotation along the Y-axis, and the roll in the attitude information represents the angle of rotation along the Z-axis.

[0029] S112 : Obtain a first weight between the first RGB image and the depth image and a second weight between the second RGB image and the depth image according to a proportional relationship between the first timestamp and the second timestamp.

[0030] Specifically, a first difference between the first timestamp and the timestamp of the depth map is obtained, and a second difference between the second timestamp and the timestamp of the depth map is obtained, and then, based on the principle that the smaller the difference, the greater the weight, the first weight Ai and the second weight Bi are obtained according to the ratio between the first difference and the second difference.

[0031] As an example, assuming the timestamp of the depth map is 1s, and the first timestamp and the second timestamp are 0.8s and 1.2s respectively, since the differences between 0.8s, 1.2s and 1s are all 0.2s, the first weight Ai and the second weight Bi are both 50%.

[0032] S113 , obtaining a theoretical odometer pose based on the first pose information and the first weight and the second pose information and the second weight, as the odometer pose at the same moment.

[0033] Specifically, after obtaining the first weight Ai and the second weight Bi, the odometer pose (X, Y, Z, roll, pitch, yaw) at the same time as the depth map can be calculated according to the following formula:

[0034] X=Ai*X1+Bi*X2,

[0035] Y=Ai*Y1+Bi*Y2,

[0036] Z=Ai*Z1+Bi*Z2,

[0037] roll=Ai*roll1+Bi*roll2,

[0038] pitch=Ai*pitch1+Bi*pitch2,

[0039] yaw=Ai*yaw1+Bi*yaw2。

[0040] This allows us to preprocess the RGB image, depth map, and odometry pose after acquiring them to obtain the RGB image, depth map, and odometry pose at the same moment. Since the RGB image only determines the color of map points, a slight difference in timestamps has little impact on the result. However, the depth map determines the location of map points and has a greater impact on reconstruction accuracy. Therefore, using the depth map's timestamp as a benchmark can achieve higher reconstruction accuracy. Furthermore, by unifying the timestamps of the RGB image, depth map, and odometry pose, sensors installed on mobile devices, such as cameras, no longer require specialized hardware; instead, standard cameras can be used, reducing costs and facilitating maintenance.

[0041] S12, determining the current local map based on the RGB image, depth image, and odometer pose at the same moment.

[0042] It should be noted that the computing module on the mobile device obtains a set of RGB images and depth images at the same time every preset time, and records it as a frame, and the frame carries the odometry pose. Every time the computing module obtains a frame, it adds the frame to the sliding window. If the sliding window is full at this time, the frame with the earliest timestamp in the sliding window is first moved out of the sliding window, and then the newly obtained frame is added to the sliding window. As an example, assuming that the size of the sliding window is 10, and the frames currently in the sliding window are 90 to 100 frames; when the 101st frame is obtained, the 90th frame is moved out of the sliding window, and the 101st frame is added to the sliding window.

[0043] Specifically, when a mobile device acquires a new frame, it adds the new frame to the sliding window and records the map points within the sliding window as the map points of the current local map. As an example, assuming the sliding window size is 10, and the frames currently within the sliding window are 90-100, when the 101st frame is acquired, the 101st frame is added to the sliding window, and the map points in frames 91-101 become the map points of the current local map.

[0044] S13, matching the current local map with the global map to correct the odometer posture.

[0045] Specifically, a current local map corresponding to the odometry pose is obtained; based on each map point in the current local map, a point cloud set that matches the corresponding map point is searched in the global map; a least squares plane is constructed based on the point cloud set, and a first correction value of the odometry pose is calculated based on the distance from each map point to the corresponding least squares plane; the odometry pose is corrected according to the first correction value to obtain a corrected odometry pose.

[0046] Specifically, since the odometry pose may contain errors during the operation of a mobile device, especially after long periods of operation, the errors in the odometry pose can accumulate, resulting in large errors in the odometry pose. To address the impact of this error, after determining the current local map, the portion of the global map corresponding to the current local map can be determined. Then, using the point-to-plane ICP (Iterative Closest Point) matching technique, the local map and the global map are matched to obtain a correction for the odometry pose. Specifically, for each map point in the current local map, a matching point cloud set is searched in the global map. This point cloud set can, for example, be a preset number of the closest map points. A least-squares plane is then constructed from the point cloud set. By optimizing the distance from each map point to the corresponding least-squares plane, a first correction for the odometry pose is obtained. The odometry pose is then corrected based on the first correction to obtain a corrected odometry pose. This eliminates the impact of odometry pose errors on 3D reconstruction. Among them, the above-mentioned local map and the above-mentioned global map are both composed of map points, and each map point includes its corresponding 3D coordinates (x-axis, y-axis, z-axis coordinates) and color, and also needs to include the normal vector information of the map point.

[0047] For example, suppose a robot entered a room 20 minutes ago and now enters the same room again. The map points obtained by the robot in the room 20 minutes ago are within the global map, while the map points obtained in the room now are within the local map. Using point-to-surface ICP matching technology, the first correction value of the odometry pose is obtained.

[0048] It should be noted that after matching the current local map with the global map to correct the odometer posture, it is necessary to perform the following Figure 4 Steps shown:

[0049] S131, obtaining a relative posture transformation matrix of the odometer posture according to the odometer posture and the corrected odometer posture.

[0050] S132: Obtain coordinate information of each map point in the current local map.

[0051] S133: Multiply the relative posture transformation matrix by the coordinate information of each map point to obtain updated coordinate information of each map point.

[0052] Specifically, after using the correction amount to correct the odometer posture, it is also necessary to obtain the relative posture transformation matrix T_delta of the odometer posture based on the odometer posture before correction and the odometer posture after correction, and then correct the current local map using the following formula: P_new = T_delta*P_old, where P_old is the map point coordinate of the current local map before correction, and P_new is the map point coordinate of the current local map after correction.

[0053] S14, updating the current local map according to the corrected odometer pose and sliding window strategy.

[0054] Specifically, a set of RGB images and depth images at the same moment is considered a frame, which carries the odometry pose. This frame is used as the current frame corresponding to the current moment, and multiple adjacent image frames corresponding to the current frame are obtained. Specifically, after correcting the odometry pose, the current frame corresponding to the current moment is obtained. This current frame includes the RGB image and depth image at the same moment, and this frame carries the corrected odometry pose. Multiple adjacent image frames corresponding to the current frame are obtained, and a sliding window strategy is used to update the current local map based on the adjacent image frames.

[0055] The above method of obtaining multiple adjacent image frames corresponding to the current frame can be found in Figure 5 , including the following steps:

[0056] S141, obtaining the odometer pose of the current frame; wherein the current frame carries a timestamp.

[0057] S142: Acquire multiple adjacent frames within a preset time period of the current frame according to the timestamp corresponding to the current frame.

[0058] S143, performing a preset number of loop detections on the current frame to obtain a number of loop frames, the loop detection comprising: obtaining a target frame based on the current frame, and obtaining a plurality of candidate historical frames of the target frame, and screening out candidate historical frames whose difference is less than a second preset threshold from the plurality of candidate historical frames as loop frames based on the difference between the odometer pose of each candidate historical frame and the odometer pose of the target frame; wherein each candidate historical frame carries a corresponding odometer pose.

[0059] Specifically, after acquiring the current frame, the current frame is first used as the target frame, and a historical frame that is close in time to the current frame is acquired and used as a candidate historical frame. As an example, assuming that the current frame is the 50,000th frame, the target frame is the 50,000th frame, and the candidate historical frame is a frame within the range of (49,000, 50,000). Alternatively, adjacent frames within a preset time period can be removed first, and then the candidate historical frame can be determined from the remaining historical frames. Since each candidate historical frame carries a corresponding odometer pose, and the second preset threshold includes a second preset distance threshold and a second preset angle threshold, the coordinates corresponding to each candidate historical frame can be first acquired, and the coordinates corresponding to the target frame can be compared with the coordinates corresponding to the candidate historical frame, and the candidate historical frame whose coordinate difference is less than the second preset distance threshold can be selected. Then, the roll, pitch, and yaw angles of the target frame are obtained, and the roll, pitch, and yaw angles of the candidate historical frames selected above are obtained. The difference in roll, pitch, and yaw angles between the target frame and the candidate historical frames is calculated. If the angle difference between a candidate historical frame and the target frame is less than a second preset angle threshold, the candidate historical frame is determined to be a loop frame of the target frame.

[0060] Optionally, after selecting a candidate historical frame whose coordinate difference is less than a preset distance threshold, the rotation vector and rotation angle of the target frame and the selected candidate historical frame can be calculated respectively, and the difference between the target frame and the candidate historical frame can be obtained. If the difference corresponding to a candidate historical frame is less than a preset value, the candidate historical frame is determined to be the loop frame of the current frame.

[0061] Furthermore, after obtaining the loop frame using the above method, the loop frame can also be regarded as a new target frame, and the above steps are repeated. Repeat this process a preset number of times to obtain a loop frame of a loop frame, a loop frame of a loop frame of a loop frame, a loop frame of a loop frame of a loop frame, and so on. In this way, it is possible to obtain all loop frames that have similar positions and similar observation angles to the current frame. As an example, assuming that the current frame is the 100th frame, the loop frame is calculated for the 100th frame, and the loop frames are obtained as frames 20 and 21, and then the loop frames of frames 20 and 21 are obtained as frames 0, 1, 2, and 3. Repeat this process 10 times, record all the loop frames obtained, and ensure that the local map contains all loop frames that have similar positions and similar observation angles to the 100th frame. In this way, it is possible to obtain several loop frames by looping, thereby reducing the computing power requirements for three-dimensional reconstruction.

[0062] It should be noted that it can be set that during the operation of the mobile device, each time a frame is obtained, the above method needs to be used to obtain its loop frame, the loop frame of the loop frame, the loop frame of the loop frame of the loop frame... It can also be set that after the loop frame of a frame is obtained, the loop frame corresponding to the frame can be recorded, and then if the loop frame calculated for a certain frame is the frame, the recording result can be directly used to obtain the loop frame corresponding to the frame.

[0063] S144: Use the plurality of loop frames and the plurality of adjacent frames as adjacent image frames of the current frame.

[0064] Specifically, all loop frames obtained by the above method and the multiple adjacent frames within the preset time period of the current frame obtained above are used as adjacent image frames of the current frame, and then the adjacent image frames are added to the sliding window.

[0065] Therefore, it is possible to update the local map using a sliding window strategy.

[0066] Among them, the above-mentioned sliding window strategy is: when multiple adjacent image frames are added to a sliding window of preset size, the map points corresponding to several adjacent image frames in the sliding window are used to generate the current local map, and the map points corresponding to the remaining adjacent image frames not added to the sliding window are added to the global map; wherein, the map points corresponding to the remaining adjacent image frames not added to the sliding window are added to the global map, including: obtaining the remaining adjacent image frames in which the difference between the odometer posture in the image frame in the sliding window and the odometer posture of the current frame is greater than a first preset threshold, and when the remaining adjacent image frames exit the sliding window, the map points corresponding to the remaining adjacent image frames are removed from the current local map, and the map points corresponding to the remaining adjacent image frames are added to the global map.

[0067] Specifically, the first preset threshold includes a first preset distance threshold and a first preset angle threshold. After acquiring the current frame, the position and orientation of the frames within the sliding window are compared with the position and orientation of the current frame. If the difference between the position of a frame within the sliding window and the position of the current frame is greater than the first preset distance threshold, or the difference between the orientation of a frame within the sliding window and the orientation of the current frame is greater than the first preset angle threshold, the frame is determined to be a remaining adjacent image frame, and the remaining adjacent image frame is then removed from the sliding window.

[0068] When obtaining the adjacent image frames of the current frame and adding the adjacent image frames to the sliding window, the map points corresponding to the adjacent image frames are removed from the global map, and the map points corresponding to the adjacent image frames are added to the current local map; when the remaining adjacent image frames with gradually increasing differences exit the sliding window, the map points corresponding to the remaining adjacent image frames are removed from the current local map, and the map points corresponding to the remaining adjacent image frames are added to the global map.

[0069] S15, fusing the updated current local map with the RGB map to update the global map.

[0070] Specifically, see Figure 6 , the updated current local map is merged with the RGB map to update the global map, including the following steps:

[0071] S151, performing image segmentation on the RGB image and dividing it into several super-pixel regions.

[0072] Specifically, a superpixel area is a small area composed of a series of pixels that are adjacent in position and have similar characteristics such as color, brightness, and texture. In an embodiment of the present application, for the current frame, the RGB image can be divided according to the coordinates, colors, and corresponding depth values of the pixels in the RGB image in the frame to obtain a superpixel area. For example, the pixels in the RGB image whose coordinate difference is less than or equal to a preset coordinate threshold, whose color difference is less than or equal to a preset color threshold, and whose depth value difference is less than or equal to a preset depth value threshold can be divided into a superpixel area. For another example, a plurality of coordinate intervals, color intervals, and depth value intervals can be preset, and then the pixels whose coordinates are within a coordinate interval, whose colors are within a color interval, and whose depth values are within a depth value interval can be divided into a superpixel area.

[0073] For each superpixel region, the average color, average depth, and average normal vector of all pixels that make up the region are obtained, and the average color, average depth, and average normal vector are used as the color, depth, and normal vector of the superpixel region.

[0074] Among them, the method for obtaining the normal vector of the above-mentioned pixel point is: obtain several pixel points near a certain pixel point, for example, you can obtain the pixel point closest to the pixel point among the pixels to the right of the pixel point, and obtain the pixel point closest to the pixel point among the pixels below the pixel point; then after obtaining several pixel points near the pixel point, use the pixel point and several pixel points near the pixel point to construct a plane, and obtain the normal vector of the plane. The normal vector of the plane that has the same direction as the camera is the normal vector of the pixel point.

[0075] In this way, the data can be merged to reduce memory usage, and the robustness to noise can be improved in the process of averaging.

[0076] Optionally, when performing image segmentation on the RGB image, the RGB image can also be directly divided into a number of squares, and each square is approximated as a superpixel region. For embedded devices, this method of obtaining superpixel regions is preferably used.

[0077] S152 , based on the corrected odometer pose and the configuration parameters of the image sensor on the mobile device, project each map point in the updated current local map onto the plane where the RGB image is located to obtain the pixel coordinates corresponding to each map point.

[0078] Specifically, the updated current local map is projected onto the RGB image according to the following calculation formula:

[0079] U=K*T*P,

[0080] Among them, K is the configuration parameter of the image sensor, T is the corrected odometry pose, P is each map point in the updated current local map, and U is the pixel coordinate corresponding to each map point.

[0081] S153: When a certain pixel coordinate is within the range of any super-pixel region, the pixel coordinate is compared with the corresponding super-pixel region for feature difference.

[0082] Specifically, for a certain pixel coordinate, if it is within the range of a super-pixel region, the difference in normal vector and depth value between the pixel coordinate and the super-pixel region is compared.

[0083] S154 , obtaining pixel coordinates describing the same environmental feature as the superpixel region based on the comparison result, and integrating the map points corresponding to the pixel coordinates into the corresponding superpixel region.

[0084] Specifically, if the pixel coordinates are close to the normal vector direction of the superpixel area and the depth value difference is not large, then the pixel coordinates and the superpixel area can be considered to be descriptions of the same environmental feature, and the pixel coordinates and the superpixel area can be fused. Specifically, the normal vector, coordinates, and color of the pixel coordinates and the superpixel area are weightedly fused respectively, where the weight value of the pixel coordinate is the sum of the weight values of the historical superpixel areas fused with its corresponding local map point, and the weight value of the superpixel area is the square of the inverse of its depth.

[0085] If there is a superpixel area that has not been fused, the environmental features corresponding to the superpixel area are considered to be the newly observed environmental features of the mobile device, and then the color, normal vector, depth of the superpixel area and the odometry pose corresponding to the superpixel area are used to create a new map point in the global map, thereby supplementing the map.

[0086] This allows the global map to be updated. Furthermore, by reprojecting the local map onto the RGB image during the update process, there's no need to create a new map point for each value in the depth map, significantly reducing memory usage during 3D reconstruction. Furthermore, by fusing pixel coordinates with superpixel regions, errors in the depth map are prevented from affecting the 3D reconstruction. Furthermore, multiple observations can be used to describe the same feature, further improving the accuracy of the 3D reconstruction.

[0087] In one embodiment of the present invention, although the odometry pose must be corrected before moving the current local map, this correction cannot completely eliminate errors. As errors accumulate, vertical and horizontal planes in the global map, such as walls, tables, floors, and cabinets, will gradually become thicker. To address this issue, after updating the global map, the 3D map reconstruction method further includes maintaining the accuracy of the global map.

[0088] Specifically, the normal vector of each map point in the global map is obtained. Based on the normal vector, each map point in the vertical and horizontal planes is obtained. For each map point, the following operation is performed: several points in the global map closest to the map point are obtained, and a least squares plane is constructed using these points. The map point is then moved along the normal vector direction of the least squares plane until it falls on the least squares plane.

[0089] In this way, the global map can be processed to eliminate errors, so that the local map can be better matched with the global map.

[0090] In one embodiment of the present invention, after determining the current local map based on the RGB image, depth map, and odometry pose at the same time, it is further possible to determine whether the odometry pose needs to be corrected according to a preset method. If it is determined that correction is not necessary, step S13 is omitted and the current local map is moved directly according to the odometry pose. For example, the type of odometry used by the mobile device can be obtained. If the odometry used by the mobile device is a type that is prone to errors, such as an IMU (Inertial Measurement Unit) odometry, correction is determined to be required. If the odometry used by the mobile device is a more accurate odometry, such as a laser odometry, correction is determined not to be required.

[0091] In one embodiment of the present invention, when the sliding window strategy is used to update the current local map, the running time of the current mobile device can be obtained. If the running time of the current mobile device is greater than the preset time threshold, the method shown in the above embodiment is used.

[0092] If the current mobile device's running time is less than or equal to a preset time threshold, the mobile device may first retrieve multiple adjacent frames within a preset time period of the current frame based on the timestamp corresponding to the current frame. These frames are then added to a sliding window, and the coordinates corresponding to all historical frames are obtained. The coordinates are then compared with the coordinates corresponding to the current frame and the coordinates corresponding to the historical frames, and the historical frame whose coordinate difference is less than a preset distance threshold is selected. Furthermore, the roll, pitch, and yaw angles of the current frame are obtained, as well as the roll, pitch, and yaw angles of the selected historical frames. The difference in roll, pitch, and yaw angles between the current frame and the historical frames is calculated. If the angle difference between a historical frame and the current frame is less than a preset angle threshold, the historical frame is determined to be an adjacent image frame of the current frame.

[0093] It should be noted that using the running time of the mobile device as the judgment criterion is only a specific example in the embodiment of the present invention and is not limited thereto in practical applications. For example, the judgment criterion may be set to the number of historical frames.

[0094] Therefore, when the mobile device has a short running time and the number of historical frames obtained is not large, the adjacent image frames can be obtained by traversing the historical frames to increase the accuracy of obtaining the adjacent image frames; and when the number of historical frames is large, the adjacent image frames can be obtained by looping, thereby improving the efficiency of obtaining the adjacent image frames.

[0095] In one embodiment of the present invention, before determining the current local map based on the RGB image, depth map, and odometry pose at the same moment, the 3D map reconstruction method further includes: using ICP point cloud registration technology to perform real-time correction of the odometry pose. Specifically, after determining the RGB image, depth map, and odometry pose at the same moment, ICP point cloud registration technology can be used to first correct the odometry pose, and then the corrected odometry pose can be used to determine the current local map.

[0096] In summary, the 3D map reconstruction method of the embodiment of the present invention can, after determining the RGB image, depth map, and odometry pose at the same time, determine the current local map based on the RGB image, depth map, and odometry pose at the same time, and then match the global map based on the current local map to correct the odometry pose to ensure the accuracy of the odometry pose and prevent the error of the odometry pose from affecting the 3D reconstruction. Moreover, after correcting the odometry pose, the current local map is moved and updated based on the corrected odometry pose. Furthermore, the updated current local map is fused with the RGB image to update the global map, thereby significantly reducing the memory usage and computing power requirements during the 3D reconstruction process and improving the computational speed of the 3D map reconstruction method. This allows the 3D map reconstruction method of the embodiment of the present invention to be directly applied to mobile devices without the need for additional GPUs, high-end CPUs, etc., and can be run online and on embedded devices. Moreover, since a local map is constructed and used to update the global map, the computing power requirements for 3D reconstruction can be further reduced.

[0097] Furthermore, the present invention provides a mobile device.

[0098] In an embodiment of the present invention, the mobile device includes an image device, a memory, a controller, and a 3D reconstruction program stored in the memory and executable on the controller. When the controller executes the 3D reconstruction program, the above-mentioned 3D map reconstruction method is implemented.

[0099] The mobile device of the embodiment of the present invention, by implementing the above-mentioned three-dimensional map reconstruction method, can determine the current local map based on the RGB image, depth map, and odometer pose at the same time after determining the RGB image, depth map, and odometer pose at the same time, and match the global map based on the current local map to correct the odometer pose to ensure the accuracy of the odometer pose and avoid the error of the odometer pose from affecting the three-dimensional reconstruction. Moreover, after correcting the odometer pose, the current local map is moved and updated according to the corrected odometer pose. Furthermore, the updated current local map is fused with the RGB image to update the global map, thereby utilizing this fusion to significantly reduce the memory usage and computing power requirements in the three-dimensional reconstruction process, and improve the computing speed of the three-dimensional map reconstruction method, so that the three-dimensional map reconstruction method of the embodiment of the present invention can be directly applied by mobile devices without the need for additional GPU, high-end CPU and other equipment, and can be run online and on embedded devices.

[0100] Furthermore, the present invention provides a computer-readable storage medium.

[0101] In an embodiment of the present invention, a three-dimensional reconstruction program is stored on a computer-readable storage medium, and the three-dimensional reconstruction program implements the above-mentioned three-dimensional map reconstruction method when executed by a processor.

[0102] The computer-readable storage medium of an embodiment of the present invention, when the computer program thereon is executed by a processor, can determine the RGB image, depth map, and odometer pose at the same time, and then determine the current local map based on the RGB image, depth map, and odometer pose at the same time, and match the global map based on the current local map to correct the odometer pose to ensure the accuracy of the odometer pose and avoid the error of the odometer pose from affecting the three-dimensional reconstruction. Moreover, after correcting the odometer pose, the current local map is moved and updated according to the corrected odometer pose. Furthermore, the updated current local map is fused with the RGB image to update the global map, thereby utilizing the fusion to significantly reduce the memory usage and computing power requirements in the three-dimensional reconstruction process, and improve the computing speed of the three-dimensional map reconstruction method, so that the three-dimensional map reconstruction method of the embodiment of the present invention can be directly applied to mobile devices without the need for additional GPU, high-end CPU and other equipment, and can be run online, and can be run on embedded devices.

[0103] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0104] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0105] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0106] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0107] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0108] In the present invention, unless otherwise specified or limited, the terms "installed," "connected," "connect," "fixed," etc. should be understood in a broad sense. For example, they can refer to fixed connection, detachable connection, or integration; mechanical connection, electrical connection; direct connection, or indirect connection through an intermediate medium; internal communication between two components, or interaction between two components, unless otherwise specified. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0109] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0110] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A three-dimensional map reconstruction method, characterized in that: Applied to a mobile device, the method includes: Determine the RGB image, depth image, and odometry pose at the same moment; Determine the current local map based on the RGB image, depth image, and odometry pose at the same moment; Matching the current local map with the global map to correct the odometer pose; Updating the current local map according to the corrected odometer pose and sliding window strategy; Fusing the updated current local map with the RGB map to update the global map; The determining of the RGB image, depth image, and odometer pose at the same moment includes: Obtain a depth map at the current moment and several RGB maps at adjacent moments; wherein each of the depth map and the several RGB maps carries a corresponding timestamp; Calculate the timestamp difference between each RGB image and the depth image in the plurality of RGB images; If the timestamp difference is not greater than the first time threshold and the number of RGB images is one, the RGB image and the depth image are regarded as RGB images and depth images at the same time, and the pose corresponding to the RGB image is used as the odometry pose at the same time; If the timestamp difference is not greater than the first time threshold and the number of RGB images is at least two, the RGB image corresponding to the latest timestamp and the depth image are regarded as the RGB image and depth image at the same time, and the timestamp of the depth image is used as the reference to determine the odometer pose at the same time by linear interpolation.

2. The three-dimensional map reconstruction method according to claim 1, characterized in that: The method of determining the odometer pose at the same moment by linear interpolation based on the timestamp of the depth map includes: Obtain a first RGB image and a second RGB image having the smallest timestamp difference from the depth image; wherein the first RGB image carries a first timestamp and first pose information, and the second RGB image carries a second timestamp and second pose information; Obtaining a first weight between the first RGB image and the depth image and a second weight between the second RGB image and the depth image according to a proportional relationship between the first timestamp and the second timestamp; A theoretical odometer posture is obtained according to the first posture information, the first weight, the second posture information, and the second weight, as the odometer posture at the same moment.

3. The three-dimensional map reconstruction method according to claim 1, characterized in that: Before determining the current local map based on the RGB image, the depth image, and the odometry pose at the same time, the method further includes: The ICP point cloud registration technology is used to perform real-time correction on the odometer posture.

4. The three-dimensional map reconstruction method according to claim 1, characterized in that: The matching of the current local map with the global map to correct the odometer pose includes: Obtaining a current local map corresponding to the odometer pose; According to each map point in the current local map, searching for a point cloud set matching the corresponding map point in the global map; Constructing a least squares plane based on the point cloud set, and calculating a first correction value for the odometer posture based on the distance from each map point to the corresponding least squares plane; The odometer posture is corrected according to the first correction value to obtain a corrected odometer posture.

5. The three-dimensional map reconstruction method according to claim 4, characterized in that: After matching the current local map with the global map to correct the odometer pose, the method further includes: Obtaining a relative posture transformation matrix of the odometer posture according to the odometer posture and the corrected odometer posture; Obtaining coordinate information of each map point in the current local map; The relative posture transformation matrix is multiplied by the coordinate information of each map point to obtain the updated coordinate information of each map point.

6. The three-dimensional map reconstruction method according to claim 1, characterized in that: The updating of the current local map according to the corrected odometer pose and the sliding window strategy includes: Taking a set of RGB images and depth images at the same moment as a frame, and the frame carrying the odometry pose; Taking the frame as the current frame corresponding to the current moment, obtaining multiple adjacent image frames corresponding to the current frame; The sliding window strategy is as follows: when adding a plurality of adjacent image frames to a sliding window of a preset size, the map points corresponding to the corresponding adjacent image frames within the sliding window are used to generate a current local map, and the map points corresponding to the remaining adjacent image frames not included in the sliding window are added to the global map; The step of adding the map points corresponding to the remaining adjacent image frames not included in the sliding window to the global map includes: Obtain the remaining adjacent image frames in which the difference between the odometry pose in the image frames within the sliding window and the odometry pose of the current frame is greater than a first preset threshold. When the remaining adjacent image frames exit the sliding window, remove the map points corresponding to the remaining adjacent image frames from the current local map, and add the map points corresponding to the remaining adjacent image frames to the global map.

7. The three-dimensional map reconstruction method according to claim 6, characterized in that: The adjacent image frames include multiple adjacent frames and multiple historical frames, and obtaining the multiple adjacent image frames corresponding to the current frame includes: Obtaining the odometer pose of the current frame; wherein the current frame carries a timestamp; Acquire a plurality of adjacent frames within a preset time period of the current frame according to a timestamp corresponding to the current frame; Performing a preset number of loop closure detections on the current frame to obtain a plurality of loop closure frames, the loop closure detection comprising: obtaining a target frame based on the current frame, and obtaining a plurality of candidate historical frames of the target frame; and selecting, from the plurality of candidate historical frames, candidate historical frames having a difference value less than a second preset threshold value as loop closure frames based on a difference value between an odometer pose of each candidate historical frame and an odometer pose of the target frame; wherein each candidate historical frame carries a corresponding odometer pose; The plurality of loop frames and the plurality of adjacent frames are all used as adjacent image frames of the current frame.

8. The three-dimensional map reconstruction method according to claim 1, characterized in that: The updated current local map is merged with the RGB map, including: Performing image segmentation on the RGB image to divide it into a plurality of super-pixel regions; Projecting each map point in the updated current local map onto the plane of the RGB image based on the corrected odometry pose and configuration parameters of the image sensor on the mobile device to obtain pixel coordinates corresponding to each map point; When a pixel coordinate is within the range of any superpixel region, comparing the pixel coordinate with the corresponding superpixel region for feature difference; According to the comparison result, pixel coordinates describing the same environmental feature as the super-pixel region are obtained, and the map points corresponding to the pixel coordinates are integrated into the corresponding super-pixel region.

9. The three-dimensional map reconstruction method according to claim 8, characterized in that: The image segmentation is performed on the RGB image to divide it into a plurality of super-pixel regions, including: The RGB image is divided according to the coordinates, colors and corresponding depth values of the pixels in the RGB image to obtain the superpixel area.

10. A self-propelled device, characterized in that: The invention comprises an image device, a memory, a controller and a three-dimensional reconstruction program stored in the memory and executable on the controller. When the controller executes the three-dimensional reconstruction program, the three-dimensional map reconstruction method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium, characterized in that A three-dimensional reconstruction program is stored thereon, and when the three-dimensional reconstruction program is executed by a processor, the three-dimensional map reconstruction method according to any one of claims 1 to 9 is implemented.

Images