Environment Reconstruction Method, System and Mobile Robot Based on SLAM Method

Abstract

The present invention discloses an environment reconstruction method, system and mobile robot based on the SLAM method. The environment reconstruction method based on the SLAM method includes: fusing multiple data sources based on the simultaneous localization and mapping method to construct a 3D semantic map and storing it in a map database; updating the 3D semantic map according to depth measurement and semantic information. By fusing multiple data sources, the present invention can more accurately construct a 3D semantic map of the environment.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robots, and particularly to an environment reconstruction method, system and mobile robot. Background Art

[0002] Current mobile robots, such as floor sweepers, whether based on gyroscopes, cameras or single-line lasers, lack the ability to perceive the real ground environment, or rather, the amount of information is very small, and only discrete two-dimensional sampling information of the spatial structure at a specific height or sparse feature point cloud information on the top space of the floor sweeper can be obtained. Moreover, it is difficult to reconstruct the geometric structure and semantic information of the environment with a height below 10 cm on the ground in the map, resulting in the current floor sweepers being unable to solve the problems of perceiving and colliding with low obstacles (such as slippers, cables, small toys, pet feces, etc.). Summary of the Invention

[0003] Aiming at the defects of the above-mentioned prior art, the purpose of the present invention is to provide an environment reconstruction method, system and mobile robot based on the SLAM (simultaneous localization and mapping) method to solve the above defects or at least partially solve the above defects.

[0004] According to one aspect of the present invention, an environment reconstruction method based on the simultaneous localization and mapping method is provided, including: fusing multiple data sources based on the simultaneous localization and mapping method to construct a 3D semantic map and storing it in a map database; and updating the 3D semantic map according to depth measurement and semantic information.

[0005] In an embodiment of the present invention, the multiple data sources include inertial measurement unit data, wheel encoder data, 2D point cloud data, depth map data, and IR map data.

[0006] In an embodiment of the present invention, the fusion includes: establishing a motion equation according to the motion constraints of a mobile robot to obtain the prior information of the state quantity of the mobile robot at the current moment; establishing an observation equation according to the characteristics of multiple sensors on the mobile robot to obtain the measurement information of the multiple sensors; and fusing the prior information and the measurement information to obtain the fused positioning information.

[0007] In an embodiment of the present invention, the update includes: projecting the depth map data collected by a depth camera to obtain a 3D point cloud, and updating the probability information in the map voxels of the 3D semantic map in combination with the fused positioning information.

[0008] In one embodiment of the present invention, the environment reconstruction method further includes: analyzing and removing the point cloud noise in the 3D point cloud by using the confidence histogram method with the IR map data collected by the depth camera.

[0009] In one embodiment of the present invention, the environment reconstruction method further includes: performing downsampling processing on the 3D point cloud.

[0010] In one embodiment of the present invention, the environment reconstruction method further includes: performing semantic segmentation and edge extraction on an object by using the IR map data collected by the depth camera to obtain the semantic information and edge information of the object, and updating the 3D semantic map by combining the semantic information and the edge information.

[0011] In one embodiment of the present invention, updating the 3D semantic map by combining the semantic information and the edge information includes: performing Bayesian inference according to the semantic information and the edge information to obtain an inference result; and re-projecting the map points of the 3D point cloud according to the inference result to obtain the re-projected 3D point cloud.

[0012] In one embodiment of the present invention, the object is a light-transmitting object, a light-reflecting object, and / or a light-absorbing object.

[0013] In one embodiment of the present invention, the state quantity of the motion equation includes at least one of position, attitude, linear velocity, angular velocity, and acceleration.

[0014] In one embodiment of the present invention, the sensor includes a lidar, a depth camera, an inertial measurement unit, and a wheel encoder.

[0015] In one embodiment of the present invention, the fusion is to fuse the prior information and the measurement information by using a Bayesian recursive estimation algorithm.

[0016] In one embodiment of the present invention, when performing fusion, the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder is used as the initial value of the slow-changing data measured by the lidar and the depth camera, and then Bayesian inference is performed together with a registration result.

[0017] In one embodiment of the present invention, when performing fusion, it further includes: performing pose optimization on the attitude.

[0018] According to another aspect of the present invention, the present invention further provides an environment reconstruction system based on a simultaneous localization and mapping method, including: a map database for storing data, including a plurality of data sources; a processor configured to perform: constructing a 3D semantic map based on the simultaneous localization and mapping method using the plurality of data sources and storing it in the map database; and updating the 3D semantic map according to depth measurement and semantic information.

[0019] In another embodiment of the present invention, the plurality of data sources include inertial measurement unit data, wheel encoder data, 2D point cloud data, depth map data, and IR map data.

[0020] In another embodiment of the present invention, the processor includes: a fusion unit for fusing prior information and measurement information to obtain fused localization information, where the prior information is the prior information of the current state quantity of the mobile robot obtained by establishing a motion equation according to the motion constraints of a mobile robot, and the measurement information is the measurement information of the plurality of sensors obtained by establishing an observation equation according to the characteristics of the plurality of sensors on the mobile robot; and an update unit for using the depth map data collected by a depth camera to project to obtain a 3D point cloud and updating the probability information in the map voxels of the 3D semantic map in combination with the fused localization information.

[0021] In another embodiment of the present invention, the processor further includes: a confidence histogram processing unit for using the IR map data collected by the depth camera to analyze and remove point cloud noise in the 3D point cloud by the confidence histogram method.

[0022] In another embodiment of the present invention, the processor further includes: a downsampling processing unit for performing downsampling processing on the 3D point cloud.

[0023] In another embodiment of the present invention, the processor further includes: a semantic segmentation unit for using the IR map data collected by the depth camera to perform semantic segmentation on an object to obtain semantic information of the object; an edge extraction unit for using the IR map data collected by the depth camera to perform edge extraction on an object to obtain edge information of the object; where the processor updates the 3D semantic map in combination with the semantic information and the edge information.

[0024] In another embodiment of the present invention, the processor further includes: a Bayesian inference unit for performing Bayesian inference according to the semantic information and the edge information to obtain an inference result; and a map point reprojection unit for reprojecting the map points of the 3D point cloud according to the inference result to obtain a reprojected 3D point cloud.

[0025] In another embodiment of the present invention, the object is a light-transmitting object, a light-reflecting object, and / or a light-absorbing object.

[0026] In another embodiment of the present invention, the state variables of the motion equation include at least one of position, attitude, linear velocity, angular velocity, and acceleration.

[0027] In another embodiment of the present invention, the sensors include a lidar, a depth camera, an inertial measurement unit, and a wheel encoder.

[0028] In another embodiment of the present invention, the fusion unit uses the Bayesian recursive estimation algorithm to fuse the prior information and the measurement information.

[0029] In another embodiment of the present invention, when performing fusion, the fusion unit uses the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder as the initial value of the slow-changing data measured by the lidar and the depth camera, and then performs Bayesian inference together with a registration result.

[0030] In another embodiment of the present invention, the processor further includes: a pose optimization unit for optimizing the pose during fusion.

[0031] To achieve the above object, the present invention further provides a mobile robot, including: an environment reconstruction system based on the simultaneous localization and mapping method as described above.

[0032] In yet another embodiment of the present invention, the mobile robot is a floor sweeper.

[0033] By using the fusion of multiple data sources, the present invention can more accurately construct a 3D semantic map of the environment. Moreover, by updating the 3D semantic map according to depth measurement and semantic information, the present invention can achieve the reconstruction of a high-precision environment map, enabling the mobile robot to truly see the home environment, thereby completely solving the problems that existing floor sweepers cannot see low obstacles and cannot avoid pet feces, slippers, socks, cables, etc.

[0034] The present invention can also use the IR image data collected by the depth camera to analyze and remove the point cloud noise with low confidence through the confidence histogram method. At the same time, for light-transmitting, light-reflecting, and light-absorbing objects that cannot be measured, the 3D semantic map can be updated by combining the semantic information and edge information of the objects, thereby achieving the reconstruction of a globally consistent high-precision environment map.

[0035] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other objects, features, and advantages of the present invention more obvious and understandable, the following specifically describes the embodiments of the present invention. Brief Description of the Drawings

[0036] By reading the following detailed description of the preferred embodiments, various other advantages and benefits of the present invention will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0037] Figure 1 A schematic diagram of the environmental reconstruction system based on the simultaneous localization and mapping method of the present invention is shown;

[0038] Figure 2 A schematic diagram of the environmental reconstruction method based on the simultaneous localization and mapping method of the present invention is shown;

[0039] Figure 3 A schematic diagram of the structure of the mobile robot of the present invention is shown. Detailed Embodiments

[0040] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0041] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiment", etc. in this specification mean that the described embodiment may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Further, when combining specific features, structures or characteristics with an embodiment, it has been shown that it is within the knowledge of those skilled in the art to combine such features, structures or characteristics with other embodiments, whether or not explicitly described.

[0042] In addition, in the specification and the subsequent claims, certain terms are used to refer to specific components or parts. Those with ordinary knowledge in the relevant field should understand that manufacturers may use different nouns or terms to refer to the same component or part. The specification and the subsequent claims do not use the difference in names as a way to distinguish components or parts, but use the difference in the functions of components or parts as the criterion for distinction. The terms "comprising" and "including" mentioned throughout the specification and the subsequent claims are open-ended terms and should be interpreted as "including but not limited to". In addition, the term "connected" herein includes any direct and indirect connection means. Indirect connection means include connection through other devices.

[0043] As Figure 1 shown, the environment reconstruction system based on the simultaneous localization and mapping (SLAM) method of the present invention includes a map database 100 and a processor 200. Among them, the map database 100 is used to store data, which includes a plurality of data sources, such as but not limited to inertial measurement unit data 11, wheel encoder data 12, 2D point cloud data 13, depth map data 14, and IR map data 15. The processor 200 is configured to perform: constructing a 3D semantic map based on the simultaneous localization and mapping method using the plurality of data sources and storing it in the map database 100; and updating the 3D semantic map according to depth measurement and semantic information.

[0044] More specifically, the processor 200 may include, for example, a fusion unit 21 and an update unit 22. Among them, the fusion unit 21 can be used to fuse the prior information and the measurement information to obtain the fused positioning information. The prior information is the prior information of the state quantity of the mobile robot at the current moment obtained by establishing a motion equation according to the motion constraints of a mobile robot, and the state quantity may include at least one of position, attitude, linear velocity, angular velocity, and acceleration. The measurement information is the measurement information of a plurality of sensors obtained by establishing an observation equation according to the characteristics of the plurality of sensors on the mobile robot. The sensors may include, for example, but are not limited to, lidar, depth camera, inertial measurement unit (IMU), and wheel encoder, etc. Among them, the lidar may be a single-line lidar, and 2D point cloud data 13 can be obtained through it. The depth camera may be, for example, a visual image acquisition device such as structured light, TOF, or binocular vision that can sense the depth of the environment, such as a 3D depth camera, and depth map data 14, IR data 15, etc. can be obtained through it. The IMU may include, for example, a gyroscope and an accelerometer, and IMU data 11, such as angular velocity, acceleration, and other data, can be obtained through it. Wheel encoder data 12, such as wheel speed data, can be obtained through the wheel encoder. Preferably, the fusion unit 21 uses the Bayesian recursive estimation algorithm to fuse the prior information and the measurement information. Moreover, when the fusion unit 21 performs fusion, the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder is first used as the initial value of the slow-changing data measured by the lidar and the depth camera, and then Bayesian inference is performed together with a registration result.

[0045] The update unit 22 can be used to project the depth map data 14 collected by the depth camera to obtain a 3D point cloud, and update the probability information in the map voxel of the 3D semantic map in combination with the fused positioning information.

[0046] In the present invention, the processor 200 may further include a confidence histogram processing unit 23, which can be used to analyze and remove the low-confidence point cloud noise in the 3D point cloud by using the IR map data 15 collected by the depth camera through the confidence histogram method.

[0047] In the present invention, the processor 200 may further include a downsampling processing unit 24, which can be used to perform downsampling processing on the 3D point cloud.

[0048] In the present invention, the processor 200 may further include a semantic segmentation unit 25 and an edge extraction unit 26. Among them, the semantic segmentation unit 25 may be used to perform semantic segmentation on an object by using the IR map data 15 collected by the depth camera to obtain the semantic information of the object. The edge extraction unit 26 may be used to perform edge extraction on an object by using the IR map data 15 collected by the depth camera to obtain the edge information of the object. Among them, the processor updates the 3D semantic map by combining the semantic information and the edge information. And the object may be a light-transmitting object, a light-reflecting object, and / or a light-absorbing object.

[0049] In the present invention, the processor 200 may further include a Bayesian inference unit 27 and a map point reprojection unit 28. The Bayesian inference unit 27 may be used to perform Bayesian inference based on the semantic information and the edge information to obtain an inference result. The map point reprojection unit 29 is used to reproject the map points of the 3D point cloud according to the inference result to obtain the reprojected 3D point cloud.

[0050] In the present invention, the processor 200 may further include a pose optimization unit 29, which may be used to perform pose optimization on the pose during fusion.

[0051] As Figure 2 shown, the environment reconstruction method based on the simultaneous localization and mapping method of the present invention includes:

[0052] Step S1, using multiple data sources to perform fusion based on the simultaneous localization and mapping method to construct a 3D semantic map and store it in a map database. Among them, the multiple data sources may include, for example, but are not limited to, inertial measurement unit data, wheel encoder data, 2D point cloud data, depth map data, and IR map data.

[0053] Step S2, updating the 3D semantic map according to depth measurement and semantic information.

[0054] In the present invention, the fusion may include, for example: establishing a motion equation according to the motion constraints of a mobile robot, and obtaining prior information on the state quantity of the mobile robot at the current moment, where the state quantity of the motion equation may include at least one of position, attitude, linear velocity, angular velocity, and acceleration. Establishing an observation equation according to the characteristics of multiple sensors on the mobile robot, and obtaining measurement information of the multiple sensors. And, fusing the prior information and the measurement information to obtain fused positioning information, and for example, a Bayesian recursive estimation algorithm may be used for fusion. Preferably, when performing fusion, the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder may be first used as the initial value of the slow-changing data measured by the lidar and the depth camera, and then Bayesian inference may be performed together with a registration result. And, when performing fusion, pose optimization may also be performed on the attitude.

[0055] In the present invention, the update may include, for example: projecting the depth map data collected by the depth camera to obtain 3D point clouds, and updating the probability information in the map voxels of the 3D semantic map in combination with the fused positioning information.

[0056] In the present invention, the environment reconstruction method may further include: using the IR map data collected by the depth camera, and analyzing and removing the point cloud noise in the 3D point clouds by a confidence histogram method.

[0057] In the present invention, the environment reconstruction method may further include: performing downsampling processing on the 3D point clouds.

[0058] In the present invention, the environment reconstruction method may further include: using the IR map data collected by the depth camera, performing semantic segmentation and edge extraction on an object to obtain semantic information and edge information of the object, and updating the 3D semantic map in combination with the semantic information and the edge information. For example, Bayesian inference may be performed according to the semantic information and the edge information to obtain an inference result; and according to the inference result, the map points of the 3D point clouds may be reprojected to obtain reprojected 3D point clouds.

[0059] As Figure 3 shown, the environment reconstruction system of the simultaneous localization and mapping method according to the present invention may be applied to a mobile robot 300, and the mobile robot 300 may be, for example, a floor sweeper.

[0060] The positioning data obtained by the single-line lidar can be regarded as absolute positioning data, and the positioning data generated by the wheel speedometer and gyroscope, etc., can be regarded as relative positioning data. If the 3D depth camera only relies on the relative positioning data generated by the wheel speedometer and gyroscope during the movement, map stitching will occur due to the cumulative error of the relative positioning data. And the present invention combines the characteristics of the single-line lidar with a long range and high accuracy and the large amount of data and rich information of the 3D depth camera, and can simultaneously obtain depth, contour, IR and other image information of the environment and obstacles, so as to realize high-precision environmental map modeling. The present invention effectively fuses multiple data sources such as IMU data, wheel encoder data, 2D point cloud data, depth map data and IR map data, and can obtain high-precision positioning information, so as to more accurately construct a 3D semantic map of the environment. And, the present invention can realize the reconstruction of the high-precision environmental map by updating the 3D semantic map according to depth measurement and semantic information. In this way, the present invention can enable the robot to truly see the home environment, thus completely solving the problems that the existing sweeping robots cannot see low obstacles and cannot avoid pet feces, slippers, socks, cables, etc.

[0061] In addition, since there are generally light-transmitting, light-reflecting, and light-absorbing objects in the home environment that are difficult to measure by optical sensors, the depth camera "cannot measure accurately and cannot measure", which reduces the positioning accuracy and the consistency of the 3D semantic map. Therefore, the present invention uses the IR map data collected by the depth camera, analyzes and eliminates the low-confidence point cloud noise through the confidence histogram method. At the same time, for the light-transmitting objects, light-reflecting objects, and light-absorbing objects that cannot be measured, the 3D semantic map can be updated by combining the semantic information and edge information of the objects, so as to realize the reconstruction of the high-precision environmental map with global consistency.

[0062] The present invention discloses A1, an environmental reconstruction method based on a simultaneous localization and mapping method, including:

[0063] Fusing multiple data sources based on the simultaneous localization and mapping method to construct a 3D semantic map and storing it in a map database; and

[0064] Updating the 3D semantic map according to depth measurement and semantic information.

[0065] A2. The environmental reconstruction method according to A1, wherein the multiple data sources include inertial measurement unit data, wheel encoder data, 2D point cloud data, depth map data, and IR map data.

[0066] A3. The environmental reconstruction method according to A2, wherein the fusion includes:

[0067] Establishing a motion equation according to the motion constraints of a mobile robot to obtain the prior information of the state quantity of the mobile robot at the current moment;

[0068] Establish an observation equation based on the characteristics of multiple sensors on the mobile robot, and obtain the measurement information of the multiple sensors; and

[0069] Fuse the prior information and the measurement information to obtain the fused positioning information.

[0070] A4. The environment reconstruction method according to A3, wherein the update includes:

[0071] Use the depth map data collected by the depth camera to project to obtain a 3D point cloud, and update the probability information in the map voxel of the 3D semantic map in combination with the fused positioning information.

[0072] A5. The environment reconstruction method according to A4, wherein the environment reconstruction method further includes:

[0073] Use the IR map data collected by the depth camera, and analyze and remove the point cloud noise in the 3D point cloud by the confidence histogram method.

[0074] A6. The environment reconstruction method according to A5, wherein the environment reconstruction method further includes:

[0075] Perform downsampling processing on the 3D point cloud.

[0076] A7. The environment reconstruction method according to A6, wherein the environment reconstruction method further includes:

[0077] Use the IR map data collected by the depth camera to perform semantic segmentation and edge extraction on the object to obtain the semantic information and edge information of the object, and update the 3D semantic map in combination with the semantic information and the edge information.

[0078] A8. The environment reconstruction method according to A7, wherein updating the 3D semantic map in combination with the semantic information and the edge information includes:

[0079] Perform Bayesian inference according to the semantic information and the edge information to obtain an inference result; and

[0080] Reproject the map points of the 3D point cloud according to the inference result to obtain the reprojected 3D point cloud.

[0081] A9. The environment reconstruction method according to A8, wherein the object is a reflective object and / or an absorbent object.

[0082] A10. The environment reconstruction method according to any one of A3 to A9, wherein the state quantity of the motion equation includes at least one of position, attitude, linear velocity, angular velocity, and acceleration.

[0083] A11. The environmental reconstruction method according to A10, wherein the sensors include a lidar, a depth camera, an inertial measurement unit, and a wheel encoder.

[0084] A12. The environmental reconstruction method according to A11, wherein the fusion is to fuse the prior information and the measurement information by using a Bayesian recursive estimation algorithm.

[0085] A13. The environmental reconstruction method according to A12, wherein when performing the fusion, the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder is used as the initial value of the slow-changing data measured by the lidar and the depth camera, and then Bayesian inference is performed together with a registration result.

[0086] A14. The environmental reconstruction method according to A13, wherein when performing the fusion, it further includes:

[0087] Performing pose optimization on the pose.

[0088] A15. An environmental reconstruction system based on a simultaneous localization and mapping method, comprising:

[0089] A map database for storing data, including a plurality of data sources;

[0090] A processor configured to perform:

[0091] Constructing a 3D semantic map based on the plurality of data sources by using a simultaneous localization and mapping method and storing it in the map database; and

[0092] Updating the 3D semantic map according to depth measurement and semantic information.

[0093] A16. The environmental reconstruction system according to A15, wherein the plurality of data sources include inertial measurement unit data, wheel encoder data, 2D point cloud data, depth map data, and IR map data.

[0094] A17. The environmental reconstruction system according to A16, wherein the processor includes:

[0095] A fusion unit for fusing prior information and measurement information to obtain fused localization information, wherein the prior information is the prior information of the state quantity of the mobile robot at the current moment obtained by establishing a motion equation according to the motion constraints of a mobile robot, and the measurement information is the measurement information of the plurality of sensors obtained by establishing an observation equation according to the characteristics of the plurality of sensors on the mobile robot; and

[0096] An update unit, configured to project depth map data collected by a depth camera to obtain 3D point clouds, and update probability information within map voxels of the 3D semantic map in combination with the fused positioning information.

[0097] A18. The environment reconstruction system according to A17, wherein the processor further includes:

[0098] A confidence histogram processing unit, configured to analyze and remove point cloud noise in the 3D point clouds by using confidence histogram method with IR map data collected by the depth camera.

[0099] A19. The environment reconstruction system according to A18, wherein the processor further includes:

[0100] A downsampling processing unit, configured to perform downsampling processing on the 3D point clouds.

[0101] A20. The environment reconstruction system according to A18, wherein the processor further includes:

[0102] A semantic segmentation unit, configured to perform semantic segmentation on an object by using IR map data collected by the depth camera to obtain semantic information of the object;

[0103] An edge extraction unit, configured to perform edge extraction on an object by using IR map data collected by the depth camera to obtain edge information of the object;

[0104] Wherein, the processor updates the 3D semantic map in combination with the semantic information and the edge information.

[0105] A21. The environment reconstruction system according to A20, wherein the processor further includes:

[0106] A Bayesian inference unit, configured to perform Bayesian inference according to the semantic information and the edge information to obtain an inference result; and

[0107] A map point reprojection unit, configured to reproject map points of the 3D point clouds according to the inference result to obtain reprojected 3D point clouds.

[0108] A22. The environment reconstruction method according to A21, wherein the object is a reflective object and / or an absorptive object.

[0109] A23. The environment reconstruction system according to any one of A17 to A22, wherein the state quantity of the motion equation includes at least one of position, attitude, linear velocity, angular velocity, and acceleration.

[0110] A24. The environmental reconstruction system according to A23, wherein the sensors include lidar, depth camera, inertial measurement unit, and wheel encoder.

[0111] A25. The environmental reconstruction system according to A24, wherein the fusion unit fuses the prior information and the measurement information by using the Bayesian recursive estimation algorithm.

[0112] A26. The environmental reconstruction system according to A25, wherein when the fusion unit performs fusion, the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder is used as the initial value of the slow-changing data measured by the lidar and the depth camera, and then Bayesian inference is performed together with a registration result.

[0113] A27. The environmental reconstruction system according to A26, wherein the processor further includes:

[0114] A pose optimization unit for optimizing the pose during fusion.

[0115] A28. A mobile robot, comprising:

[0116] An environmental reconstruction system based on the simultaneous localization and mapping method according to any one of A15 to A27.

[0117] A29. The mobile robot according to A28, wherein the mobile robot is a floor sweeper.

[0118] In the specification provided herein, a large number of specific details are set forth. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0119] Similarly, it should be understood that, in order to streamline this disclosure and assist in understanding one or more of the various inventive aspects, in the foregoing description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting the intention that the claimed invention has more features than are expressly recited in each claim. Rather, as reflected by the following claims, the inventive aspects lie in less than all of the features of the preceding single embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate embodiment of the present invention.

[0120] Those skilled in the art can understand that the modules, units, or groups in the devices in the embodiments can be adaptively changed and disposed in one or more devices different from those of the embodiments. The modules, units, or components in the embodiments can be combined into one module, unit, or component, and can also be divided into multiple sub-modules, sub-units, or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be adopted to combine all the features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise clearly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) can be replaced by an alternative feature that provides the same, equivalent, or similar purpose.

[0121] Of course, the present invention can also have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention. However, these corresponding changes and deformations should all fall within the protection scope of the appended claims of the present invention.

Claims

1. An environmental reconstruction method based on a simultaneous localization and mapping method, characterized in that, Including: Fusing multiple data sources based on a simultaneous localization and mapping method to construct a 3D semantic map and storing it in a map database; And Updating the 3D semantic map according to depth measurement and semantic information; The multiple data sources include inertial measurement unit data, wheel encoder data, 2D point cloud data, depth map data, and IR map data; The fusion includes: Establishing a motion equation based on the motion constraints of a mobile robot to obtain prior information on the state quantity of the mobile robot at the current moment; Establishing an observation equation based on the characteristics of multiple sensors on the mobile robot to obtain measurement information of the multiple sensors; and Fusing the prior information and the measurement information to obtain fused localization information; The state quantity of the motion equation includes at least one of position, attitude, linear velocity, angular velocity, and acceleration; The sensors include a lidar, a depth camera, an inertial measurement unit, and a wheel encoder; The fusion is to fuse the prior information and measurement information using a Bayesian recursive estimation algorithm; When performing fusion, the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder is used as the initial value of the slow-changing data measured by the lidar and the depth camera, and then Bayesian inference is performed together with a registration result.

2. The environmental reconstruction method according to claim 1, characterized in that The update includes: Projecting the depth map data collected by the depth camera to obtain a 3D point cloud, and updating the probability information in the map voxels of the 3D semantic map in combination with the fused localization information.

3. The environmental reconstruction method according to claim 2, wherein The environment reconstruction method further includes: Using the IR map data collected by the depth camera, analyzing and removing the point cloud noise in the 3D point cloud through a confidence histogram method.

4. The environmental reconstruction method according to claim 3, wherein The environment reconstruction method further includes: Performing downsampling processing on the 3D point cloud.

5. The environmental reconstruction method according to claim 4, characterized in that The environment reconstruction method further includes: Using the IR map data collected by the depth camera, performing semantic segmentation and edge extraction on an object to obtain the semantic information and edge information of the object, and updating the 3D semantic map in combination with the semantic information and the edge information.

6. The environmental reconstruction method according to claim 5, characterized in that Updating the 3D semantic map in combination with the semantic information and the edge information includes: Performing Bayesian inference based on the semantic information and the edge information to obtain an inference result; and Reprojecting the map points of the 3D point cloud according to the inference result to obtain a reprojected 3D point cloud.

7. The environmental reconstruction method according to claim 6, characterized in that The object is a reflective object and / or an absorptive object.

8. The environmental reconstruction method according to claim 1, characterized in that When performing fusion, it further includes: Performing pose optimization on the attitude.

9. An environmental reconstruction system based on a simultaneous localization and mapping method, characterized in that, Including: A map database for storing data, including multiple data sources; A processor configured to execute: Constructing a 3D semantic map based on the multiple data sources using a simultaneous localization and mapping method and storing it in the map database; and Updating the 3D semantic map according to depth measurement and semantic information; The multiple data sources include inertial measurement unit data, wheel encoder data, 2D point cloud data, depth map data, and IR map data; The processor includes: A fusion unit, configured to fuse prior information and measurement information to obtain fused positioning information, where the prior information is prior information of the state quantity of the mobile robot at the current moment obtained by establishing a motion equation based on the motion constraints of a mobile robot, and the measurement information is measurement information of a plurality of sensors obtained by establishing an observation equation based on the characteristics of the plurality of sensors on the mobile robot; The state quantity of the motion equation includes at least one of position, attitude, linear velocity, angular velocity, and acceleration; The sensors include a lidar, a depth camera, an inertial measurement unit, and a wheel encoder; The fusion unit fuses the prior information and the measurement information by using a Bayesian recursive estimation algorithm; When performing fusion, the fusion unit uses the fusion result of the fast-changing data measured by the inertial measurement unit and the wheel encoder as the initial value of the slow-changing data measured by the lidar and the depth camera, and then performs Bayesian inference together with a registration result.

10. The environmental reconstruction system according to claim 9, characterized in that, The processor further includes: An update unit, configured to project the depth map data collected by the depth camera to obtain 3D point clouds, and update the probability information in the map voxels of the 3D semantic map in combination with the fused positioning information.

11. The environmental reconstruction system according to claim 10, wherein The processor further includes: A confidence histogram processing unit, configured to analyze and remove the point cloud noise in the 3D point clouds by using the confidence histogram method with the IR map data collected by the depth camera.

12. The environmental reconstruction system according to claim 11, characterized in that, The processor further includes: A downsampling processing unit, configured to perform downsampling processing on the 3D point clouds.

13. The environmental reconstruction system according to claim 11, wherein, The processor further includes: A semantic segmentation unit, configured to perform semantic segmentation on an object by using the IR map data collected by the depth camera to obtain semantic information of the object; An edge extraction unit, configured to perform edge extraction on an object by using the IR map data collected by the depth camera to obtain edge information of the object; Wherein, the processor updates the 3D semantic map in combination with the semantic information and the edge information.

14. The environmental reconstruction system according to claim 13, wherein The processor further includes: A Bayesian inference unit, configured to perform Bayesian inference according to the semantic information and the edge information to obtain an inference result; and A map point reprojection unit, configured to reproject the map points of the 3D point clouds according to the inference result to obtain reprojected 3D point clouds.

15. The environmental reconstruction system according to claim 14, wherein The object is a reflective object and / or an absorptive object.

16. The environmental reconstruction system according to claim 9, characterized in that, The processor further includes: An attitude optimization unit, configured to optimize the attitude during fusion.

17. A mobile robot, characterized in that, Including: An environment reconstruction system based on the simultaneous localization and mapping method according to any one of claims 9 to 16.

18. The mobile robot according to claim 17, wherein The mobile robot is a floor sweeper.

Images