Map Processing Method, Device, Cleaning Robot and Medium of a Cleaning Robot

Abstract

The present invention discloses a map processing method, device, cleaning robot and medium for a cleaning robot. The map processing method includes: obtaining a three-dimensional map, and determining a reference wall region and a wall region to be filled at a preset height in the three-dimensional map; determining a number of vacant positions in the wall region to be filled; filling the vacant positions at corresponding positions in the wall region to be filled according to the reference color of each position in the reference wall region to obtain a target wall region; and updating the three-dimensional map according to the target wall region and the reference wall region. According to the map processing method, device, cleaning robot and medium of the present invention, the missing parts of the walls in the three-dimensional map can be quickly filled, the display effect of the three-dimensional map can be optimized, and the user experience can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of map processing, and in particular, to a map processing method, device, cleaning robot, and medium for a cleaning robot. Background Art

[0002] With the rapid development of science and technology, artificial intelligence technology has become increasingly mature, and the application of positioning and mapping functions on robots has gradually become widespread and mature, and the accuracy of positioning and mapping has also gradually improved. It relies on sensors to perceive the surrounding environment, and at the same time realizes the description of the robot's own position and the environment. By matching the robot's current observations with the historical mapping, the current latest position of the robot can be determined; after determining the latest position, the latest observations can be applied to the current position to update and supplement the mapping.

[0003] However, in the related art, various implementation schemes in the direction of three-dimensional reconstruction have excessively high requirements for computing power and time consumption. Most of them are offline schemes or rely on GPUs, and it is difficult to achieve real-time operation under the conditions of limited computing power and memory on the mobile robot side. Even if three-dimensional map reconstruction can be performed on the mobile robot side, the final generated three-dimensional model will have missing parts of some walls due to the inability to completely cover the surrounding environment during the operation of the mobile robot, affecting the aesthetics. Summary of the Invention

[0004] The present invention aims to at least solve one of the technical problems in the related art to some extent. To this end, an object of the present invention is to propose a map processing method for a cleaning robot, which can quickly fill the missing parts of the walls in the three-dimensional map, optimize the display effect of the three-dimensional map, and improve the user experience.

[0005] A second object of the present invention is to propose a map processing device for a cleaning robot.

[0006] A third object of the present invention is to propose a cleaning robot.

[0007] A fourth object of the present invention is to propose a computer-readable storage medium.

[0008] To achieve the above object, an embodiment of the first aspect of the present invention proposes a map processing method for a cleaning robot, the map processing method including: obtaining a three-dimensional map, and determining a reference wall region and a wall region to be filled at a preset height in the three-dimensional map; determining a number of vacant positions in the wall region to be filled; filling the vacant positions at corresponding positions in the wall region to be filled according to the reference color of each position in the reference wall region to obtain a target wall region; and updating the three-dimensional map according to the target wall region and the reference wall region.

[0009] According to the map processing method of the present invention, first, a reference wall region at a preset height is determined, and then, according to the reference color at each position of the reference wall region at the preset height, the vacant positions at the corresponding positions of the wall regions to be filled at other heights are filled, and the three-dimensional map is updated, so that the missing parts of the walls in the three-dimensional map can be filled quickly, the display effect of the three-dimensional map is optimized, and the user experience is improved.

[0010] In some embodiments of the present invention, determining the reference wall region at the preset height in the three-dimensional map includes: intercepting the wall region at the preset height in the three-dimensional map; densifying the wall region at the preset height according to the wall points of the three-dimensional map to obtain the reference wall region at the preset height in the three-dimensional map.

[0011] In some embodiments of the present invention, determining the wall regions to be filled in the three-dimensional map includes: determining the upper boundary height and the lower boundary height of the wall according to the z-axis coordinate component of the wall points of the three-dimensional map; determining the first wall region in the three-dimensional map according to the upper boundary height of the wall and the preset height, and determining the second wall region in the three-dimensional map according to the lower boundary height of the wall and the preset height; determining the wall regions to be filled in the three-dimensional map according to the first wall region and the second wall region.

[0012] In some embodiments of the present invention, determining the several vacant positions of the wall regions to be filled includes: traversing each position in each height region of the wall regions to be filled and determining the wall point closest to each position; calculating the first distance between the position and the closest wall point corresponding to the position, and when the first distance is greater than the first threshold, taking the position as the vacant position of the wall regions to be filled.

[0013] In some embodiments of the present invention, after updating the three-dimensional map according to the target wall region and the reference wall region, the method further includes: traversing each map point of the updated three-dimensional map and determining the map point closest to each map point; calculating the second distance between the map point and the closest map point corresponding to the map point, and when the second distance is greater than the second threshold, inserting a new map point between the map point and the closest map point corresponding to the map point; determining the parameters of the new map point according to the parameters of the map point and the parameters of the closest map point corresponding to the map point.

[0014] In some embodiments of the present invention, the obtaining of the three-dimensional map includes: determining a current image frame, pose information corresponding to the current image frame, a local map, and camera parameters; segmenting the current image frame to obtain a plurality of superpixel regions; according to the pose information and the camera parameters, projecting each map point in the local map onto the pixel plane of the current image frame to obtain projection coordinate information corresponding to each map point; according to the projection coordinate information, normal vector information, and depth information of each map point, determining a plurality of map points having environmental feature consistency with a certain superpixel region; integrating the plurality of map points into the corresponding superpixel region to update the local map, and updating the three-dimensional map according to the updated local map.

[0015] In some embodiments of the present invention, the current image frame includes a color image and a depth image with timestamp alignment. The segmenting the current image frame to obtain a plurality of superpixel regions includes: dividing pixel points in the color image whose coordinate difference is less than or equal to a preset coordinate threshold, color difference is less than or equal to a preset color threshold, and depth value difference is less than or equal to a preset depth value threshold into one superpixel region.

[0016] To achieve the above object, an embodiment of the second aspect of the present invention provides a map processing device for a cleaning robot. The map processing device includes: an obtaining module, configured to obtain a three-dimensional map and determine a reference wall region and a wall region to be filled at a preset height in the three-dimensional map; a first determining module, configured to determine a plurality of vacant positions in the wall region to be filled; a filling module, configured to fill the vacant positions at corresponding positions in the wall region to be filled according to the reference color at each position in the reference wall region to obtain a target wall region; and an updating module, configured to update the three-dimensional map according to the target wall region and the reference wall region.

[0017] According to the map processing device of the cleaning robot of the present invention, first, a reference wall region at a preset height is determined, and then the vacant positions at corresponding positions in the wall region to be filled at other heights are filled according to the reference color at each position in the reference wall region at the preset height, and the three-dimensional map is updated, so that the missing part of the wall in the three-dimensional map can be filled quickly, the display effect of the three-dimensional map is optimized, and the user experience is improved.

[0018] To achieve the above object, an embodiment of the third aspect of the present invention provides a cleaning robot, which includes the map processing device described in the above embodiment.

[0019] For the cleaning robot according to the present invention, first, a reference wall area at a preset height is determined, and then, according to the reference color at each position of the reference wall area at the preset height, the vacant positions at the corresponding positions of the to-be-filled wall areas at other heights are filled, and the three-dimensional map is updated, so that the missing parts of the walls in the three-dimensional map can be filled quickly, the display effect of the three-dimensional map can be optimized, and the user experience can be improved.

[0020] To achieve the above object, a fourth aspect embodiment of the present invention provides a computer-readable storage medium, on which a map processing program is stored. When the map processing program is executed by a processor, the map processing method described in any of the above embodiments is implemented.

[0021] For the computer-readable storage medium according to the present invention, first, a reference wall area at a preset height is determined, and then, according to the reference color at each position of the reference wall area at the preset height, the vacant positions at the corresponding positions of the to-be-filled wall areas at other heights are filled, and the three-dimensional map is updated, so that the missing parts of the walls in the three-dimensional map can be filled quickly, the display effect of the three-dimensional map can be optimized, and the user experience can be improved.

[0022] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where:

[0024] Figure 1 is a schematic flowchart of a map processing method according to an embodiment of the present invention;

[0025] Figure 2 is a schematic scene diagram of a map processing method according to an embodiment of the present invention;

[0026] Figure 3 is a schematic flowchart of a map processing method according to another embodiment of the present invention;

[0027] Figure 4 is a schematic flowchart of a map processing method according to another embodiment of the present invention;

[0028] Figure 5 is a schematic flowchart of a map processing method according to another embodiment of the present invention;

[0029] Figure 6 is a schematic flowchart of a map processing method according to another embodiment of the present invention;

[0030] Figure 7Schematic diagram of the effect of densification processing in the map processing method according to an embodiment of the present invention;

[0031] Figure 8 Schematic flowchart of the map processing method according to another embodiment of the present invention;

[0032] Figure 9 Scene schematic diagram of the map processing method according to another embodiment of the present invention;

[0033] Figure 10 Schematic diagram of the effect of pose correction in the map processing method according to an embodiment of the present invention;

[0034] Figure 11 Structural block diagram of the map processing device according to an embodiment of the present invention;

[0035] Figure 12 Structural block diagram of the cleaning robot according to an embodiment of the present invention;

[0036] Figure 13 Structural block diagram of the electronic device according to an embodiment of the present invention. Detailed implementation manners

[0037] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0038] To clearly illustrate the map processing method, device, cleaning robot and medium of the cleaning robot in the embodiments of the present invention, the following will be described in conjunction with Figure 1 The schematic flowchart of the map processing method shown. As Figure 1 shown, the map processing method of the cleaning robot in the embodiments of the present invention includes the following steps:

[0039] S11: Obtain a three-dimensional map, and determine a reference wall area and a wall area to be filled at a preset height in the three-dimensional map;

[0040] S13: Determine a plurality of vacant positions in the wall area to be filled;

[0041] S15: Fill the vacant positions at the corresponding positions in the wall area to be filled according to the reference color of each position in the reference wall area to obtain a target wall area;

[0042] S17: Update the three-dimensional map according to the target wall area and the reference wall area.

[0043] According to the map processing method of the present invention, first, a reference wall area at a preset height is determined, and then, according to the reference color at each position of the reference wall area at the preset height, the vacant positions at the corresponding positions of the wall areas to be filled at other heights are filled, and the three-dimensional map is updated, so that the missing parts of the walls in the three-dimensional map can be filled quickly, the display effect of the three-dimensional map can be optimized, and the user experience can be improved.

[0044] Specifically, the cleaning robot includes but is not limited to a floor-sweeping robot, a mopping robot, a sweeping and mopping integrated robot, a lawn trimming robot, etc. The obtained three-dimensional map can be a map generated in real time by the cleaning robot, or a three-dimensional map pre-stored in the cloud server or the cleaning robot, which is not limited here. There are vacant positions in the wall area of the obtained three-dimensional map, which affect the visual experience. The forms of the three-dimensional map include but are not limited to a point cloud map, a unified-size voxel map, a variable-size voxel map, a curved surface map, etc.

[0045] The three-dimensional map can include multiple map points and vacant positions. A map point can be understood as a position in the three-dimensional map with known coordinate information, color information, and normal vector information, that is, the coordinate information, color information, and normal vector information of each map point are determined and known. A vacant position can be understood as a position in the three-dimensional map with unknown coordinate information, color information, and normal vector information, that is, the coordinate information, color information, and normal vector information of each vacant position are empty and unknown.

[0046] In the three-dimensional map, the map points corresponding to the ground in the actual indoor environment can be called ground points, that is, the ground points can represent the ground in the actual indoor environment. In the three-dimensional map, the map points corresponding to the walls in the actual indoor environment can be called wall points, that is, the wall points can represent the walls in the actual indoor environment. In the three-dimensional map, the map points corresponding to the furniture in the actual indoor environment can be called furniture points, that is, the furniture points can represent the furniture in the actual indoor environment. It can be understood that a certain height area of the three-dimensional map can include both wall points and furniture points at the same time, and a certain height area of the three-dimensional map can only include wall points and not furniture points. By selecting a suitable preset height, it can be ensured that the preset height area of the three-dimensional map basically only includes wall points and not furniture points, which is convenient for determining the reference wall area.

[0047] The area where the wall points are located can be called the wall area. The reference wall area can be the wall area at the preset height in the three-dimensional map that has not been processed, that is, the reference wall area includes the initial wall points at the preset height in the three-dimensional map. The reference wall area can also be the new wall area obtained after processing the wall area at the preset height in the three-dimensional map, that is, the reference wall area includes the initial wall points at the preset height and the newly added wall points at the preset height in the three-dimensional map.

[0048] There are many wall points in the reference wall area. Any wall point in the reference wall area can be used as a reference point for the vacant position of the wall area to be filled in the vertical direction of the position of the wall point, that is, the coordinates of any wall point in the reference wall area can be used as the reference coordinates of the vacant position of the corresponding position, the color of any wall point in the reference wall area can be used as the reference color of the vacant position of the corresponding position, and the normal vector of any wall point in the reference wall area can be used as the reference normal vector of the vacant position of the corresponding position, so that the vacant position in the wall area to be filled can be filled according to the reference wall area. The corresponding position is the position of the wall area to be filled in the vertical direction of the position of the wall point in the reference wall area.

[0049] The wall area to be filled can be understood as the wall area at other heights in the three-dimensional map except for the preset height. The target wall area can be understood as the wall area to be filled in which at least part of the vacant position is filled.

[0050] Filling can be understood as determining the coordinate information, color information and normal vector information of the vacant position, and then generating a new wall point at the vacant position.

[0051] In some embodiments, please combine Figure 2 In step S15, the vacant positions of the corresponding positions of the wall area to be filled are filled vertically upward and vertically downward at each position of the reference wall area at a preset height, that is, the reference color of the wall point at each position in the reference wall area is directly used as the color of the vacant position of the corresponding position of the wall area to be filled, the reference normal vector of the wall point at each position in the reference wall area is directly used as the normal vector of the vacant position of the corresponding position of the wall area to be filled, and the coordinates of the vacant position of the corresponding position of the wall area to be filled are determined according to the reference coordinates of the wall point at each position in the reference wall area, and new wall points are generated at the vacant positions according to the colors, normal vectors and coordinates.

[0052] In some embodiments, after step S17, the map processing method further includes: displaying the updated three-dimensional map. In this way, the updated three-dimensional map can be presented to the user to show the details of the indoor environment and improve the user experience.

[0053] See also Figure 3 In some embodiments of the present invention, step S11 includes:

[0054] S111: intercepting a wall area of ​​a preset height in the three-dimensional map;

[0055] S113: performing densification processing on the wall area of ​​the preset height according to the wall points of the three-dimensional map to obtain a reference wall area of ​​the preset height in the three-dimensional map.

[0056] Thus, new wall points are added to the wall points in the wall area with a preset height in the three-dimensional map, so as to obtain a reference wall area with a larger number of wall points, so as to better fill the vacant positions in the wall area to be filled.

[0057] Specifically, in some embodiments, the preset height is greater than the lower boundary height of the door frame and less than the upper boundary height of the door frame. Thus, the reference wall area includes the vacant positions corresponding to the door frame area, and when filling the wall area to be filled according to the reference wall area, it is possible to avoid filling the door frame area with a wall. It can be understood that during the process of filling the wall area to be filled according to the reference wall area, if a certain position in the reference wall area is a vacant position, the corresponding position in the wall area to be filled cannot be filled, that is, if the corresponding position in the wall area to be filled was originally a vacant position, this position remains vacant after filling, and if the corresponding position in the wall area to be filled was originally a wall point, then this wall point remains unchanged after filling. In one example, considering that the map points in the wall area with a height of 1.6 meters are very likely to be wall points and less likely to be furniture points, and considering that the height of the door frame is usually 2 meters, the wall area with a height of 1.6 meters can include the vacant positions corresponding to the door frame area, so the preset height is set to 1.6 meters.

[0058] Further, in step S113, the densification process may include interpolation processing, that is, interpolating the vacant positions in the wall area with a preset height according to several initial wall points in the three-dimensional map that are closer to the vacant positions in the wall area with a preset height, so as to generate new wall points at the vacant positions in the wall area with a preset height, where the color of the new wall points is the weighted average of the colors of these several (for example, 3) initial wall points in the three-dimensional map. The weight value can be determined according to the distance between the initial wall point and the new wall point. The closer the distance, the greater the weight value, and the smaller the distance, the greater the weight value. In some embodiments, the kdtree algorithm can be used to determine several initial wall points in the three-dimensional map that are closer to the vacant positions in the wall area with a preset height.

[0059] It should be noted that since the interpolation process generates new wall points using the initial wall points, and the newly generated wall points are not used for interpolation processing, the interpolation process will not generate new wall points at all the vacant positions within the original door frame area, so that at least some of the vacant positions in the door frame area can be retained, avoiding the vacant positions in the door frame area being filled with wall points during the filling process.

[0060] Please refer to Figure 4 , in some embodiments of the present invention, step S11 includes:

[0061] S115: Determine the upper boundary height and the lower boundary height of the wall according to the z-axis coordinate component of the wall points on the three-dimensional map;

[0062] S117: Determine the first wall area in the three-dimensional map according to the upper boundary height of the wall and the preset height, and determine the second wall area in the three-dimensional map according to the lower boundary height of the wall and the preset height;

[0063] S119: Determine the wall area to be filled in the three-dimensional map according to the first wall area and the second wall area.

[0064] In this way, the wall area to be filled can be determined.

[0065] Specifically, since the coordinate information of each wall point on the three-dimensional map is known, the maximum value and the minimum value of the z-axis coordinate component can be determined by traversing the z-axis coordinate components of each wall point. Furthermore, the upper boundary height of the wall is determined according to the maximum value of the z-axis coordinate component, and the lower boundary height of the wall is determined according to the minimum value of the z-axis coordinate component.

[0066] Furthermore, the area between the preset height and the upper boundary height of the wall is used as the first wall area, the area between the preset height and the lower boundary height of the wall is used as the second wall area, and the first wall area and the second wall area are used as the wall area to be filled.

[0067] In one example, the preset height is 1.6 meters, the upper boundary height of the wall is 3 meters, and the lower boundary height of the wall is 0. Then, the area with a height greater than 1.6 meters and less than 3 meters is used as the first wall area, and the area with a height greater than 0 and less than 1.6 meters is used as the second wall area.

[0068] Please refer to Figure 5 , in some embodiments of the present invention, step S13 includes:

[0069] S131: Traverse each position in each height area of the wall area to be filled, and determine the wall point closest to each position;

[0070] S133: Calculate the first distance between the position and the closest wall point corresponding to the position, and when the first distance is greater than the first threshold, use the position as the vacant position of the wall area to be filled.

[0071] In this way, the vacant position of the wall area to be filled can be determined.

[0072] Specifically, the kdtree algorithm can be used to determine the wall point closest to each position in each height area of the wall area to be filled, and the first distance between each position and the corresponding closest wall point.

[0073] In one example, the first threshold is set to 1 cm. If the first distance between a certain position in the wall area to be filled and the nearest wall point is greater than 1 cm, then this position is determined as the vacant position in the wall area to be filled.

[0074] In one example, the first threshold is set to 2 cm. If the first distance between a certain position in the wall area to be filled and the nearest wall point is greater than 2 cm, then this position is determined as the vacant position in the wall area to be filled.

[0075] Please refer to Figure 6 , in some embodiments of the present invention, after step S17, the method further includes:

[0076] S21: Traverse each map point of the updated three-dimensional map and determine the nearest map point to each map point;

[0077] S23: Calculate the second distance between the map point and the nearest map point corresponding to the map point, and when the second distance is greater than the second threshold, insert a new map point between the map point and the nearest map point corresponding to the map point;

[0078] S25: Determine the parameters of the new map point according to the parameters of the map point and the parameters of the nearest map point corresponding to the map point.

[0079] In this way, global densification processing is performed on the updated three-dimensional map to further optimize the details of the three-dimensional map and improve the display effect of the three-dimensional map (as Figure 7 shown). It can be understood that since different environmental features (such as tables, wardrobes, floors, walls) are observed at different distances, angles, and frequencies from the sensor when the three-dimensional map is initially generated, the map points in some areas of the three-dimensional map are sparse. Although the map points in this part of the area are sparse, the complete structural information is retained, which can characterize the corresponding environmental features to a certain extent. For example, the actual desktop includes 2000 points, while there are only 1000 map points corresponding to the desktop in the three-dimensional map. These 1000 sparsely distributed map points retain the complete structural information of the desktop. Although they can characterize the desktop, the display effect is poor. Through the method of this embodiment, the number of map points corresponding to the desktop in the three-dimensional map can be increased, so that the number of map points corresponding to the desktop after processing is greater than 1000, thereby improving the display effect of the desktop in the three-dimensional map.

[0080] Specifically, the kdtree algorithm can be used to determine the nearest map point to each map point of the updated three-dimensional map and the second distance between every two nearest map points.

[0081] In one example, the second threshold is set to 1 cm. If the second distance between every two closest map points is greater than 1 cm, a new map point is inserted between these two map points.

[0082] In one example, the second threshold is set to 2 cm. If the second distance between every two closest map points is greater than 2 cm, a new map point is inserted between these two map points.

[0083] The parameters of the inserted new map point may include coordinate information and color information. The parameters of the inserted new map point can be obtained by linear interpolation based on the parameters of the two map points.

[0084] Please refer to Figure 8 , in some embodiments of the present invention, step S11 includes:

[0085] S31: Determine the current image frame, the pose information, the local map, and the camera parameters corresponding to the current image frame;

[0086] S33: Segment the current image frame to obtain a plurality of superpixel regions;

[0087] S35: According to the pose information and the camera parameters, project each map point in the local map onto the pixel plane of the current image frame to obtain the projection coordinate information corresponding to each map point;

[0088] S37: According to the projection coordinate information, the normal vector information, and the depth information of each map point, determine several map points that have environmental feature consistency with a certain superpixel region;

[0089] S39: Incorporate several map points into the corresponding superpixel region to update the local map, and update the 3D map according to the updated local map.

[0090] In this way, it is possible to achieve low CPU / memory occupancy, real-time, and high-precision 3D reconstruction on the cleaning robot side to obtain a 3D map. It can be understood that due to limitations such as the detection distance of the sensor, the observation direction of the robot, and the walking route of the robot, only partial regions of a certain environmental feature (such as a table, a wardrobe, the ground, a wall) may be observed when generating the 3D map, resulting in a situation where part of the wall is missing in the 3D map, affecting the aesthetics.

[0091] Specifically, the current image frame may include a color image and a depth image with timestamp alignment. The color image can be obtained by an RGB camera, and the depth image can be obtained by a depth camera. When the timestamp difference between the color image and the depth image is less than a set value, it can be considered that the timestamps of the color image and the depth image are aligned.

[0092] The pose information is the position and orientation of the cleaning robot at the time when the timestamp of the current image frame is aligned. The position can be represented by three-dimensional coordinates (x, y, z). The orientation can be represented by pitch, yaw, and roll.

[0093] The pose information can be obtained through the pose signal output by the odometer sensor. When the timestamp corresponding to the pose signal cannot be aligned with the timestamp of the current image frame, the pose signal that is earlier than the timestamp of the current image frame and the pose signal that is later than the timestamp of the current image frame can be interpolated to obtain the pose signal that is aligned with the timestamp of the current image frame, and the pose information can be determined based on the interpolated pose signal; when the timestamp corresponding to the pose signal is aligned with the timestamp of the current image frame, the pose information can be directly determined based on the pose signal with the aligned timestamp. In this way, the accuracy of the data is guaranteed, the error is reduced, and errors in the generated three-dimensional map are avoided.

[0094] Odometer sensors include but are not limited to laser odometers, IMU odometers, visual odometers, etc. It is worth noting that the pose signal output by the laser odometer has no errors and is relatively accurate, while the pose signals output by the IMU odometer and visual odometer have errors. The type of odometer sensor can be used to determine whether the obtained pose signal has errors.

[0095] When there are errors in the posture signal output by the odometer sensor, the local map can be determined based on the nearest and most recently acquired adjacent image frames of a preset number of frames (for example, 10 frames). In this way, errors in the generated three-dimensional map can be avoided as much as possible. When there are no errors in the posture signal output by the odometer sensor and the running time of the cleaning robot is short, the image frame with basically the same posture information as the current image frame in the historical frame can be determined based on the loop detection algorithm and used as the loop frame, and the local map can be determined based on the adjacent image frames and the loop frames; when there are no errors in the posture signal output by the odometer sensor and the running time of the cleaning robot is long, the loop frame and the loop frame of the loop frame can be determined based on the breadth-first search algorithm, and the local map can be determined based on the adjacent image frames, the loop frame and the loop frame of the loop frame. In this way, the accuracy of the generated three-dimensional map is guaranteed. In an example, image frames with a loop relationship such as Figure 9 shown.

[0096] Furthermore, when there is an error in the posture signal output by the odometer sensor, the posture of the local map can be corrected based on the point-surface ICP matching technology. The correction effect is as follows: Figure 10As shown below. That is, traverse each map point in the local map in sequence, find several map points in the 3D map that are closest to this map point, use them to construct a least-squares plane, record the distance from this map point in the local map to this least-squares plane and the normal vector of this least-squares plane, correct the pose information by minimizing the distances from all map points in the local map to the corresponding least-squares planes, and then obtain the relative pose transformation matrix before and after pose correction: T_delta = T_cur * T_pre -1 , for the coordinate P of each map point in the local map, multiply it by this transformation matrix T_delta to correct the position of the local map. The corrected coordinate P_new of the local map can be expressed by the following formula: P_new = T_delta * P_old.

[0097] According to the pose information T and the camera parameter K, project each map point P in the local map onto the pixel plane u of the current image frame: u = K * T * P to obtain the projection coordinate information corresponding to each map point. When the projection coordinate information of a certain map point matches a certain superpixel region, further compare the normal vector information and depth information of this map point and this superpixel region. If the differences in their normal vector information and depth information are less than the set value, it can be determined that this map point is a map point with environmental feature consistency with the superpixel region. Furthermore, the normal vector information, coordinate information, and color information of this map point and this superpixel region can be weighted and fused to update the local map. Among them, the weight value of the map point in the local map is the sum of the weight values of the historical superpixel regions that have been fused with it. The weight value of the superpixel region is determined according to the depth information of the superpixel region, and the weight value of the superpixel region is the square of the reciprocal of the depth. It can be understood that the fusion process saves memory and also reduces noise interference.

[0098] For the superpixel regions that have not been fused, they can be determined as the environmental features newly observed in the current image frame. Using their color information, normal vector information, and coordinate information, and then according to the current pose information, rotate them to the global coordinate system to create new map points to supplement the 3D map.

[0099] In some embodiments, considering that when the odometer sensor is an IMU odometer, although enabling the pose correction function can greatly improve the accuracy of the corrected local map and pose information, there will still be a small amount of error accumulation, and drift will still occur after long-term operation. Therefore, it is necessary to maintain the accuracy of the 3D map in real time to ensure the accurate matching of the local map and the 3D map. Further, considering the highly structured operating environment with a large number of vertical and horizontal planes, map points with these features are extracted according to the normal vector information of the map points, and these map points are traversed in turn. According to several nearest map points of these map points, by constructing a least squares plane, these map points are moved along the normal direction of the least squares plane to the corresponding least squares plane, so as to realize the convergence of all map points with respect to the feature center plane. In this way, when performing ICP matching next time, it can effectively avoid the local map being matched to the wrong 3D map position and ensure the accuracy of pose correction.

[0100] In some embodiments of the present invention, the current image frame includes a color image and a depth image with timestamp alignment, and step S33 includes: dividing pixel points in the color image whose coordinate difference is less than or equal to a preset coordinate threshold, color difference is less than or equal to a preset color threshold, and depth value difference is less than or equal to a preset depth value threshold into a superpixel region.

[0101] In this way, by dividing the superpixel region, a large amount of data can be merged, saving computing power and reducing memory occupancy. It can be understood that pixel points with adjacent coordinates, similar colors, and similar depth values usually correspond to the same environmental feature. Therefore, instead of using multiple pixel points for description, a superpixel region can be used to replace multiple pixel points.

[0102] Specifically, calculate the average value of the colors of all pixel points in the superpixel region as the color of the superpixel region, calculate the average value of the depth values of all pixel points in the superpixel region as the depth value of the superpixel region, and calculate the average value of the normal vectors of all pixel points in the superpixel region as the normal vector of the superpixel region, so as to determine the parameters of the superpixel region. In this way, the robustness to noise is improved.

[0103] It can be understood that in other embodiments, step S33 may further include: evenly dividing the current image frame into several rectangular tiles, and taking each rectangular tile as a superpixel region. In this way, the superpixel region can be determined quickly, and when the embedded device runs the method of the embodiments of the present invention, the computing power can be effectively saved and the efficiency can be improved.

[0104] It should be noted that the specific values mentioned above are only used as examples to illustrate the implementation of the present invention in detail, and should not be construed as a limitation of the present invention. In other examples, embodiments, or implementations, other values can be selected according to the present invention, and no specific limitation is made here.

[0105] To implement the above embodiments, an embodiment of the present invention further provides a map processing device for a cleaning robot, which can implement the map processing method of any of the above embodiments. Figure 11 FIG. is a schematic structural diagram of a map processing device according to an embodiment of the present invention. As Figure 11 shown, the map processing device 100 of the cleaning robot proposed by the present invention includes an acquisition module 12, a first determination module 14, a filling module 16, and an update module 18. The acquisition module 12 is configured to acquire a three-dimensional map and determine a reference wall region and a wall region to be filled at a preset height in the three-dimensional map. The first determination module 14 is configured to determine a plurality of vacant positions in the wall region to be filled. The filling module 16 is configured to fill the vacant positions at corresponding positions in the wall region to be filled according to the reference color of each position in the reference wall region to obtain a target wall region. The update module 18 is configured to update the three-dimensional map according to the target wall region and the reference wall region.

[0106] For the map processing device 100 of the cleaning robot according to the embodiment of the present invention, first, a reference wall region at a preset height is determined, and then the vacant positions at corresponding positions in the wall region to be filled at other heights are filled according to the reference color of each position in the reference wall region at the preset height, and the three-dimensional map is updated, so that the missing part of the wall in the three-dimensional map can be quickly filled, the display effect of the three-dimensional map is optimized, and the user experience is improved.

[0107] In some embodiments of the present invention, the acquisition module 12 includes a truncation unit and a densification unit. The truncation unit is configured to truncate the wall region at a preset height in the three-dimensional map. The densification unit is configured to densify the wall region at the preset height according to the wall points of the three-dimensional map to obtain a reference wall region at the preset height in the three-dimensional map.

[0108] In some embodiments of the present invention, the acquisition module 12 includes a first determination unit, a second determination unit, and a third determination unit. The first determination unit is configured to determine the upper boundary height and the lower boundary height of the wall according to the z-axis coordinate component of the wall points of the three-dimensional map. The second determination unit is configured to determine a first wall region in the three-dimensional map according to the upper boundary height of the wall and the preset height, and determine a second wall region in the three-dimensional map according to the lower boundary height of the wall and the preset height. The third determination unit is configured to determine a wall region to be filled in the three-dimensional map according to the first wall region and the second wall region.

[0109] In some embodiments of the present invention, the first determination module 14 includes a traversal unit and a calculation unit. The traversal unit is configured to traverse each position in each height region of the wall region to be filled, and determine the wall point closest to each position. The calculation unit is configured to calculate a first distance between the position and the closest wall point corresponding to the position, and when the first distance is greater than a first threshold, use the position as a vacant position in the wall region to be filled.

[0110] In some embodiments of the present invention, the map processing device 100 further includes a traversal module, a calculation module, and a second determination module. The traversal module is configured to traverse each map point of the updated three-dimensional map and determine the closest map point to each map point. The calculation module is configured to calculate a second distance between the map point and the closest map point corresponding to the map point, and when the second distance is greater than a second threshold, insert a new map point between the map point and the closest map point corresponding to the map point. The second determination module is configured to determine the parameters of the new map point according to the parameters of the map point and the parameters of the closest map point corresponding to the map point.

[0111] In some embodiments of the present invention, the acquisition module 12 includes a fourth determination unit, a segmentation unit, a projection unit, a fifth determination unit, and an update unit. The fourth determination unit is configured to determine the current image frame, the pose information, the local map, and the camera parameters corresponding to the current image frame. The segmentation unit is configured to segment the current image frame to obtain a plurality of superpixel regions. The projection unit is configured to project each map point in the local map onto the pixel plane of the current image frame according to the pose information and the camera parameters to obtain the projection coordinate information corresponding to each map point. The fifth determination unit is configured to determine a plurality of map points having environmental feature consistency with a certain superpixel region according to the projection coordinate information, the normal vector information, and the depth information of each map point. The update unit is configured to incorporate the plurality of map points into the corresponding superpixel region to update the local map, and update the three-dimensional map according to the updated local map.

[0112] In some embodiments of the present invention, the segmentation unit is further configured to divide pixel points in the color image whose coordinate difference is less than or equal to a preset coordinate threshold, color difference is less than or equal to a preset color threshold, and depth value difference is less than or equal to a preset depth value threshold into a superpixel region.

[0113] It should be noted that the above explanations of the implementation manners and beneficial effects of the map processing method are also applicable to the map processing device 100 of the present invention. To avoid redundancy, no detailed elaboration will be made here.

[0114] To implement the above embodiments, an embodiment of the present invention also provides a cleaning robot, which can implement the map processing method of any of the above embodiments. Figure 12 is a schematic structural diagram of a cleaning robot according to an embodiment of the present invention. As Figure 12As shown, the cleaning robot 1000 proposed by the present invention includes the map processing device 100 of the above embodiment.

[0115] For the cleaning robot 1000 according to an embodiment of the present invention, first determine a reference wall area at a preset height, and then fill the vacant positions at the corresponding positions of the wall areas to be filled at other heights according to the reference colors at each position of the reference wall area at the preset height, and update the three-dimensional map, so that the missing parts of the walls in the three-dimensional map can be quickly filled, the display effect of the three-dimensional map can be optimized, and the user experience can be improved.

[0116] It should be noted that the above explanations of the implementation manners and beneficial effects of the map processing method are also applicable to the cleaning robot 1000 of the present invention. To avoid redundancy, no detailed elaboration will be made here.

[0117] To implement the above embodiment, an embodiment of the present invention also proposes an electronic device, which can implement the map processing method of any of the above embodiments. Figure 13 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. As Figure 13 shown, the electronic device 200 proposed by the present invention includes a memory 22, a processor 24, and a map processing program 26 stored on the memory 22 and executable on the processor 24. When the processor 24 executes the map processing program 26, the map processing method of any of the above embodiments is implemented.

[0118] For the electronic device 200 according to an embodiment of the present invention, first determine a reference wall area at a preset height, and then fill the vacant positions at the corresponding positions of the wall areas to be filled at other heights according to the reference colors at each position of the reference wall area at the preset height, and update the three-dimensional map, so that the missing parts of the walls in the three-dimensional map can be quickly filled, the display effect of the three-dimensional map can be optimized, and the user experience can be improved.

[0119] Specifically, the electronic device 200 may be a device including a color camera, a depth camera, and an odometer sensor, such as a cleaning robot, an unmanned delivery vehicle, a drone, a warehousing robot, a shopping mall service robot, a food delivery robot, a handheld three-dimensional reconstruction scanner, a fixed three-dimensional reconstruction scanning device, a head-mounted AR device, etc. At this time, the generation and post-processing of the three-dimensional map can be performed on the same electronic device. The electronic device 200 may also be a device such as a server, a laptop computer, or a personal computer that does not include a color camera, a depth camera, and an odometer sensor. At this time, the three-dimensional map can be generated by other devices, and the electronic device 200 can obtain the three-dimensional map generated by other devices and perform post-processing on the obtained three-dimensional map. Other devices include, but are not limited to, cleaning robots, unmanned delivery vehicles, drones, warehousing robots, shopping mall service robots, food delivery robots, handheld three-dimensional reconstruction scanners, fixed three-dimensional reconstruction scanning devices, head-mounted AR devices, etc.

[0120] To implement the above embodiments, an embodiment of the present invention also provides a computer-readable storage medium, on which a map processing program is stored. When the map processing program is executed by a processor, the map processing method of any of the above embodiments is implemented.

[0121] According to the computer-readable storage medium of the embodiment of the present invention, first determine the reference wall region at a preset height, and then fill the vacant positions at the corresponding positions of the wall regions to be filled at other heights according to the reference colors at each position of the reference wall region at the preset height, and update the three-dimensional map, so that the missing parts of the walls in the three-dimensional map can be filled quickly, the display effect of the three-dimensional map can be optimized, and the user experience can be improved.

[0122] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementing the processFigure 1 one process or multiple processes and / or blocks Figure 1 a device for the functions specified in one block or multiple blocks.

[0124] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the process Figure 1 one process or multiple processes and / or blocks Figure 1 the functions specified in one block or multiple blocks.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0126] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0127] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. are based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0128] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are for descriptive purposes only, and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated in this embodiment. Thus, the features defined with the terms "first", "second", etc. in the embodiments of the present invention may explicitly or implicitly indicate that at least one such feature is included in this embodiment. In the description of the present invention, the meaning of the word "plurality" is at least two or more, such as two, three, four, etc., unless otherwise specifically defined in the embodiment.

[0129] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for processing a map of a cleaning robot, characterized in that, comprising: obtaining a three-dimensional map, and determining a reference wall region and a wall region to be filled at a preset height in the three-dimensional map; determining a plurality of vacant positions in the wall region to be filled; filling the vacant positions at corresponding positions in the wall region to be filled according to the reference color of each position in the reference wall region to obtain a target wall region, where the corresponding position is the position in the wall region to be filled in the vertical direction of the position where the wall point in the reference wall region is located; updating the three-dimensional map according to the target wall region and the reference wall region; the reference wall region is a new wall region obtained after processing the wall region at a preset height in the three-dimensional map; the wall region to be filled is the wall region at other heights except the preset height in the three-dimensional map; the determining a plurality of vacant positions in the wall region to be filled includes: traversing each position in each height region of the wall region to be filled, and determining the wall point closest to each position; calculating a first distance between the position and the closest wall point corresponding to the position, and when the first distance is greater than a first threshold, taking the position as a vacant position in the wall region to be filled.

2. The map processing method according to claim 1, characterized in that, the determining the reference wall region at a preset height in the three-dimensional map includes: intercepting the wall region at a preset height in the three-dimensional map; densifying the wall region at the preset height according to the wall points of the three-dimensional map to obtain the reference wall region at a preset height in the three-dimensional map.

3. The map processing method according to claim 1, characterized in that, the determining the wall region to be filled in the three-dimensional map includes: determining the upper boundary height and the lower boundary height of the wall according to the z-axis coordinate component of the wall points of the three-dimensional map; determining a first wall region in the three-dimensional map according to the upper boundary height of the wall and the preset height, and determining a second wall region in the three-dimensional map according to the lower boundary height of the wall and the preset height; determining the wall region to be filled in the three-dimensional map according to the first wall region and the second wall region.

4. The map processing method according to any one of claims 1-3, characterized in that, after the updating the three-dimensional map according to the target wall region and the reference wall region, the method further includes: traversing each map point of the updated three-dimensional map, and determining the closest map point to each map point; calculating a second distance between the map point and the closest map point corresponding to the map point, and when the second distance is greater than a second threshold, inserting a new map point between the map point and the closest map point corresponding to the map point; determining the parameters of the new map point according to the parameters of the map point and the parameters of the closest map point corresponding to the map point.

5. The map processing method according to claim 1, characterized in that, the obtaining the three-dimensional map includes: Determine the current image frame and the pose information, local map, and camera parameters corresponding to the current image frame; Segment the current image frame to obtain a plurality of superpixel regions; According to the pose information and the camera parameters, project each map point in the local map onto the pixel plane of the current image frame to obtain the projection coordinate information corresponding to each map point; According to the projection coordinate information, normal vector information, and depth information of each map point, determine several map points that have environmental feature consistency with a certain superpixel region; Integrate several map points into the corresponding superpixel region to update the local map, and update the three-dimensional map according to the updated local map.

6. The map processing method according to claim 5, wherein, the current image frame includes a color image and a depth image with timestamp alignment, and the segmenting the current image frame to obtain a plurality of superpixel regions includes: Dividing the pixel points in the color image whose coordinate difference is less than or equal to a preset coordinate threshold, color difference is less than or equal to a preset color threshold, and depth value difference is less than or equal to a preset depth value threshold into a superpixel region.

7. A map processing device for a cleaning robot, wherein, it includes: An acquisition module, configured to acquire a three-dimensional map and determine a reference wall region and a wall region to be filled at a preset height in the three-dimensional map; A first determination module, configured to determine several vacant positions in the wall region to be filled; A filling module, configured to fill the vacant positions at the corresponding positions in the wall region to be filled with the reference color of each position in the reference wall region to obtain a target wall region, where the corresponding position is the position in the wall region to be filled in the vertical direction of the position where the wall point is located in the reference wall region; An update module, configured to update the three-dimensional map according to the target wall region and the reference wall region; The reference wall region is a new wall region obtained after processing the wall region at a preset height in the three-dimensional map; the wall region to be filled is the wall region at other heights in the three-dimensional map except the preset height; The determining several vacant positions in the wall region to be filled includes: traversing each position in each height region of the wall region to be filled and determining the wall point closest to each position; Calculating a first distance between the position and the closest wall point corresponding to the position, and when the first distance is greater than a first threshold, taking the position as the vacant position in the wall region to be filled.

8. A cleaning robot, wherein, the cleaning robot includes the map processing device according to claim 7.

9. A computer-readable storage medium, wherein, a map processing program is stored thereon, and when the map processing program is executed by a processor, it implements the map processing method according to any one of claims 1-6.

Images