Robot three-dimensional map pose display method, device, equipment and storage medium

Abstract

The present disclosure provides a robot three-dimensional map pose display method and device, equipment and storage medium, relating to the technical field of data processing. The method comprises: obtaining a three-dimensional map of a space where the robot is located; obtaining a two-dimensional map constructed by the robot; matching the three-dimensional map with the two-dimensional map constructed by the robot to obtain the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot; obtaining the pose of the robot on the two-dimensional map constructed by the robot; and displaying the pose of the robot in the three-dimensional map according to the pose of the robot on the two-dimensional map constructed by the robot and the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot. The method realizes the acquisition of height information of obstacles in the area where the robot is located.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a robot three-dimensional map pose display method and device, equipment and a readable storage medium. BACKGROUND

[0002] With the development of computer technology and artificial intelligence technology, various robots with intelligent systems have appeared, such as sweeping robots, mopping robots, vacuum cleaners, weeding machines and the like. These robots can automatically travel and perform cleaning or removal operations in a certain area without user operation. A laser distance sensor (LDS) is usually installed in the robot, and in the working process, the robot measures the distance between the robot and various obstacles in the working area through the LDS, thereby drawing an instant map of the area, and feeding back the drawn map to the user, so that the user can master the map information of the area where the robot is located.

[0003] In the related art, the instant map provided by the robot to the user is a two-dimensional (2-Dimension, 2D) map, and the user can only know the planar state of the area where the robot is located through such a map, cannot obtain the height information of the obstacles in the area where the robot is located, and cannot intuitively feel the working environment of the robot.

[0004] As described above, how to provide a method of obtaining the height information of the obstacles in the area where the robot is located becomes a problem to be solved.

[0005] The above information disclosed in the background section is only intended to strengthen the understanding of the background of the present disclosure, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present disclosure provides a robot three-dimensional map pose display method, device, equipment and readable storage medium, which at least partially solves the technical problem that the related art cannot obtain the height information of the obstacles in the area where the robot is located.

[0007] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0008] According to an aspect of the present disclosure, a robot three-dimensional map pose display method is provided, including: obtaining a three-dimensional map of a space where the robot is located; obtaining a two-dimensional map constructed by the robot; matching the three-dimensional map with the two-dimensional map constructed by the robot to obtain a correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot; obtaining a pose of the robot on the two-dimensional map constructed by the robot; and displaying the pose of the robot in the three-dimensional map according to the pose of the robot on the two-dimensional map constructed by the robot and the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot.

[0009] According to an embodiment of the present disclosure, the matching the three-dimensional map with the two-dimensional map constructed by the robot to obtain the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot includes: obtaining an effective part of the three-dimensional map; projecting the effective part of the three-dimensional map to a horizontal plane to obtain a two-dimensional projection map; and matching the two-dimensional projection map with the two-dimensional map constructed by the robot to obtain the correspondence relationship between the two-dimensional projection map and the two-dimensional map constructed by the robot.

[0010] According to an embodiment of the present disclosure, the obstacle data obtained in the process of constructing the two-dimensional map by the robot is three-dimensional data; and the obtaining the effective part of the three-dimensional map includes: determining a scanning range of the robot according to the three-dimensional data; and determining the three-dimensional map located in the scanning range of the robot as the effective part of the three-dimensional map.

[0011] According to an embodiment of the present disclosure, the matching the two-dimensional projection map with the two-dimensional map constructed by the robot to obtain the correspondence relationship between the two-dimensional projection map and the two-dimensional map constructed by the robot includes: matching the two-dimensional projection map with the two-dimensional map constructed by the robot by using a method of maximizing an overlapping area; and obtaining the correspondence relationship between the two-dimensional projection map and the two-dimensional map constructed by the robot when the overlapping area between the two-dimensional projection map and the two-dimensional map constructed by the robot is maximized.

[0012] According to an embodiment of the present disclosure, the matching the three-dimensional map with the two-dimensional map constructed by the robot to obtain the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot includes: obtaining a mark of a specified obstacle in the three-dimensional map; obtaining a mark of the specified obstacle in the two-dimensional map constructed by the robot; and matching the mark of the specified obstacle in the three-dimensional map and the mark of the specified obstacle in the two-dimensional map constructed by the robot to obtain the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot.

[0013] According to an embodiment of the present disclosure, the specified obstacles include a plurality of obstacles, and the plurality of specified obstacles are not located on a straight line.

[0014] According to an embodiment of the present disclosure, the specified obstacles include a charging pile and a wall.

[0015] According to an embodiment of the present disclosure, the method further includes modifying the three-dimensional map according to a two-dimensional map constructed by the robot when the robot constructs the two-dimensional map.

[0016] According to an embodiment of the present disclosure, the method further includes displaying the three-dimensional model of the robot and the three-dimensional map in a same scale.

[0017] According to another aspect of the present disclosure, a robot three-dimensional map pose display device is provided, including: a three-dimensional map acquisition module configured to obtain a three-dimensional map of a space in which a robot is located; a constructed map acquisition module configured to obtain a two-dimensional map constructed by the robot; a map matching module configured to match the three-dimensional map and the two-dimensional map constructed by the robot, and obtain a correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot; a pose acquisition module configured to obtain a pose of the robot on the two-dimensional map constructed by the robot; and a three-dimensional display module configured to display the pose of the robot in the three-dimensional map according to the pose of the robot on the two-dimensional map constructed by the robot and the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot.

[0018] According to an embodiment of the present disclosure, the map matching module includes: a map selection module configured to obtain an effective part of the three-dimensional map; a two-dimensional projection module configured to project the effective part of the three-dimensional map to a horizontal plane to obtain a two-dimensional projection map; and a two-dimensional map matching module configured to match the two-dimensional projection map and the two-dimensional map constructed by the robot, and obtain a correspondence relationship between the two-dimensional projection map and the two-dimensional map constructed by the robot.

[0019] According to an embodiment of the present disclosure, obstacle data obtained during the process in which the robot constructs the two-dimensional map is three-dimensional data; and the map selection module is further configured to determine a scanning range of the robot according to the three-dimensional data, and determine the three-dimensional map located in the scanning range of the robot as the effective part of the three-dimensional map.

[0020] According to an embodiment of the present disclosure, the two-dimensional map matching module is further configured to match the two-dimensional projection map and the two-dimensional map constructed by the robot by using a method of maximizing an overlapping area, and obtain the correspondence relationship between the two-dimensional projection map and the two-dimensional map constructed by the robot when the overlapping area between the two-dimensional projection map and the two-dimensional map constructed by the robot is maximized.

[0021] According to an embodiment of the present disclosure, the map matching module further comprises: a first obstacle marker obtaining module, configured to obtain a marker of a specified obstacle in the three-dimensional map; a second obstacle marker obtaining module, configured to obtain a marker of the specified obstacle in the two-dimensional map constructed by the robot; and a marker matching module, configured to match the marker of the specified obstacle in the three-dimensional map and the marker of the specified obstacle in the two-dimensional map constructed by the robot, to obtain a correspondence between the three-dimensional map and the two-dimensional map constructed by the robot.

[0022] According to an embodiment of the present disclosure, the specified obstacles comprise a plurality of obstacles, and the plurality of specified obstacles are not located on a straight line.

[0023] According to an embodiment of the present disclosure, the specified obstacles comprise charging piles and walls.

[0024] According to an embodiment of the present disclosure, the device further comprises a three-dimensional map modifying module, configured to modify the three-dimensional map according to the two-dimensional map constructed by the robot when the robot constructs the two-dimensional map.

[0025] According to an embodiment of the present disclosure, the three-dimensional display module is further configured to display the three-dimensional model of the robot and the three-dimensional map in a same scale.

[0026] According to another aspect of the present disclosure, there is provided a device comprising a memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor executes the executable instructions to implement any of the above methods.

[0027] According to another aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer executable instructions that, when executed by a processor, implement any of the above methods.

[0028] The robot three-dimensional map and pose display method provided by the embodiments of the present disclosure can obtain a three-dimensional map of a space where a robot is located and a two-dimensional map constructed by the robot, match the three-dimensional map and the two-dimensional map constructed by the robot to obtain a correspondence between the three-dimensional map and the two-dimensional map constructed by the robot, and then display a pose of the robot in the three-dimensional map according to a pose of the robot on the two-dimensional map constructed by the robot and the correspondence between the three-dimensional map and the two-dimensional map constructed by the robot, so that height information of obstacles in a region where the robot is located can be obtained.

[0029] It should be understood that the general description above and the detailed description below are only examples and are not limiting of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0030] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram showing a system structure in an embodiment of the present disclosure.

[0032] Figure 2 A flowchart showing a method for displaying a three-dimensional map of a robot in an embodiment of the present disclosure.

[0033] Figure 3A A schematic diagram showing a three-dimensional map of a space where a robot is located in an embodiment of the present disclosure.

[0034] Figure 3B A schematic diagram showing another three-dimensional map of a space where a robot is located in an embodiment of the present disclosure.

[0035] Figure 3C A schematic diagram showing still another three-dimensional map of a space where a robot is located in an embodiment of the present disclosure.

[0036] Figure 4A A schematic diagram showing an AR device for drawing a three-dimensional map in an embodiment of the present disclosure.

[0037] Figure 4B A schematic diagram showing another AR device for drawing a three-dimensional map in an embodiment of the present disclosure.

[0038] Figure 5 A flowchart showing a method for matching a three-dimensional map and a two-dimensional grid map in an embodiment of the present disclosure.

[0039] Figure 6 A process schematic diagram showing a method for matching a two-dimensional projection map and a two-dimensional grid map in an embodiment of the present disclosure.

[0040] Figure 7 A flowchart showing another method for matching a three-dimensional map and a two-dimensional grid map in an embodiment of the present disclosure. Figure 5 A process schematic diagram showing the steps S502 shown in FIG. 5B in an embodiment.

[0041] Figure 8 A flowchart showing another method for matching a three-dimensional map and a two-dimensional grid map in an embodiment of the present disclosure.

[0042] Figure 9A A flowchart showing a method for robot working in an embodiment of the present disclosure.

[0043] Figure 9B A schematic diagram showing a robot architecture in an embodiment of the present disclosure.

[0044] Figure 10A block diagram of a robot three-dimensional map pose display device is shown.

[0045] Figure 11 A block diagram of another robot three-dimensional map pose display device is shown.

[0046] Figure 12 A block diagram of a robot three-dimensional map pose display system is shown.

[0047] Figure 13 A structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0048] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. The accompanying drawings are included to provide a further understanding of example embodiments of the disclosure and are incorporated into and constitute a part of this specification. The drawings are not intended to be a complete specification of all example embodiments and are not intended to represent the only examples to which the claims can be embodied. Rather, the purpose of the drawings is to explain example embodiments of the disclosure.

[0049] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail in order to avoid obscuring aspects of the disclosure.

[0050] In the description of the disclosure, the meaning of "a plurality of" is at least two, such as two, three, and the like, unless otherwise explicitly specified and limited. The symbol " / " generally represents that the front and rear associated objects are in an "or" relationship.

[0051] In the present disclosure, unless otherwise explicitly specified and limited, the term "connection" and the like should be understood in a broad sense, for example, can be electrically connected or can communicate with each other; can be directly connected, or can be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above-mentioned terms in the present disclosure can be understood according to the specific circumstances.

[0052] As described above, the instant map provided by the robot to the user in the related art is a two-dimensional map, and the user can only know the planar state of the area where the robot is located through such a map, and cannot obtain the height information of the obstacles in the area where the robot is located. Therefore, the present disclosure provides a robot three-dimensional map pose display method, which obtains a three-dimensional map of the space where the robot is located and a two-dimensional grid map constructed by the robot, matches the three-dimensional map with the two-dimensional grid map to obtain a corresponding relationship, and then displays the pose of the robot in the three-dimensional map according to the pose of the robot on the two-dimensional grid map and the corresponding relationship between the three-dimensional map and the two-dimensional grid map, so that the height information of the obstacles in the area where the robot is located can be obtained. In order to facilitate understanding, the following first explains several terms related to the present disclosure.

[0053] An intelligent robot is a comprehensive system integrating environment perception, dynamic decision and planning, behavior control and execution, and other functions, which integrates research results of sensor technology, information processing, electronic engineering, computer engineering, automation control engineering, artificial intelligence and other disciplines, and represents the highest achievement of mechatronics, and is one of the most active fields of scientific and technological development. Intelligent robots can be divided into fixed robots and mobile robots according to their movement modes. Fixed robots such as robot arms are widely used in industry. Mobile robots can be divided into wheel-type mobile robots, walking mobile robots, tracked mobile robots, crawling robots, peristaltic robots and swimming robots according to their movement modes; indoor mobile robots and outdoor mobile robots according to their working environments; functional (horizontal) structure robots, behavior (vertical) structure robots and hybrid robots according to their control architecture; medical robots, military robots, disabled assistance robots, cleaning robots and other robots according to their functions and uses. With the continuous improvement of robot performance, the application range of mobile robots has been greatly expanded, not only in industries such as industry, agriculture, medical care and service, but also in fields such as city safety, national defense and space exploration.

[0054] A mobile robot is a robot system composed of sensors, remote controllers and automatically controlled mobile carriers. Mobile robots have a mobile function, and have greater mobility and flexibility than fixed robots in replacing people to work in dangerous and harsh environments (such as radiation, toxic substances, etc.) and environments that people cannot reach (such as space, underwater, etc.).

[0055] Augmented Reality (AR) is a new technology that integrates real world information and virtual world information "seamlessly", and is a technology that simulates and superimposes virtual information onto real world through computer and other scientific technologies, so that the virtual information is perceived by human senses, and the real environment and virtual objects are simultaneously superimposed in the same picture or space in real time, achieving a sensory experience beyond reality.

[0056] Figure 1 An exemplary system architecture 10 to which the robot three-dimensional map pose display method or the robot three-dimensional map pose display device of the present disclosure can be applied is shown.

[0057] As shown in Figure 1 , the system architecture 10 can include a terminal device 102, a network 104, a server 106, and a database 108. The terminal device 102 can be various electronic devices with a display screen and supporting input and output, including but not limited to a smartphone, a tablet computer, a laptop computer, a desktop computer, an AR head-mounted device, a mobile robot (such as a cleaning robot, a guide robot), and the like. The network 104 is a medium to provide a communication link between the terminal device 102 and the server 106. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, and the like. The server 106 can be a server or a cluster of servers that provide various services, such as a background processing server that provides support for map modeling of data sensed by the cleaning robot 102 to work in an environment. The database 108 can be a warehouse that organizes, stores, and manages data according to a data structure, including but not limited to a relational database, a cloud database, and the like, such as a database that stores map data of a working area of a robot.

[0058] The user can use the terminal device 102 to interact with the server 106 and the database 108 through the network 104 to receive or send data, and the like. The server 106 can also receive data from or send data to the database 108 through the network 104. For example, after the server 106 obtains the 3D map of the working area of the cleaning robot 102 from the database 108, it can plan a working route for the cleaning robot, and send the information of the planned working route to the AR head-mounted device 102 through the network 104, so that the user can view the simulated working route of the cleaning robot in the AR map through the AR head-mounted device.

[0059] It should be understood that Figure 1 the number of terminal devices, networks, and servers in may be merely illustrative. Depending on the implementation needs, there can be any number of terminal devices, networks, and servers.

[0060] Figure 2 is a flow chart of a robot three-dimensional map pose display method according to an exemplary embodiment. As shown in the method can be applied to the server side of the system described above, and can also be applied to the terminal device of the system described above. Figure 2

[0061] Referring to Figure 2 , the method 20 provided by the embodiment of the disclosure can include the following steps.

[0062] In step S202, a three-dimensional map of the space where the robot is located is obtained. A 3D mapping device can be arranged in the space where the robot working area is located, and a 3D mapping device can be used to draw the map using an augmented reality application (such as an AR application based on ARkit, ARcore, etc.) on the platform. First, the dense point cloud of the space is obtained, and then the 3D mesh model of the space is generated based on the point cloud. Then, based on the generated mesh model, the model mapping is completed, that is, the texture is mapped to the 3D mesh model of the space through the coordinates, and the 3D map is drawn. As shown in Figures 3A-3C Figures 3A-3C three three-dimensional map schematic diagrams of the space where the robot is located are shown, Figure 3A a panoramic angle schematic of a three-dimensional map of a space where a robot is located, Figure 3B a local enlarged view schematic of a three-dimensional map of a space where a robot is located, Figure 3C a top view schematic of a three-dimensional map of a space where a robot is located.

[0063] ARkit is an AR application framework. By combining the camera, motion sensor and graphics processor of a mobile device with algorithms such as depth sensing and artificial light rendering, developers can implement AR applications (such as 3D mapping applications) on mobile devices. ARcore can achieve similar functions to ARkit.

[0064] The 3D map can be drawn by a device with 3D mapping function. In some embodiments, for example, an AR device including an AR drawing device can be used for 3D mapping, and the AR drawing device can be a mobile device with 3D mapping function, including but not limited to iPAD Pro, CANVAS iPAD and other devices with depth imaging capability, as shown in Figures 4A-4B Figure 4A is a schematic diagram of an AR drawing device CANVAS iPAD, Figure 4B ​​​A schematic diagram of an AR mapping device, iPAD Pro. The AR device maps the completed 3D map, which can be sent to a 3D display device, which can be an AR display device, which can be included in the AR device. The AR display device can be a mobile device with the capability of displaying 3D maps and preset models, including but not limited to iPAD, iPhone, and other devices with display capabilities. The AR mapping device and the AR display device can be the same device.

[0065] In step S204, a two-dimensional map constructed by the robot is obtained. The robot can construct a two-dimensional map in real time during movement, for example, a sweeping robot measures the distance between itself and various obstacles in the working area during cleaning work through the installed LDS, thereby drawing an instant map of the area. Various lidar (Lidar) simultaneous localization and mapping (SLAM) methods can be used to draw the instant map, for example, the HectorSLAM algorithm based on optimization (solving the least squares problem), the Gmapping algorithm based on particle filtering, Cartographer, etc., where Cartographer is a 2D and 3D SLAM library supported by Google's open source robot operating system (ROS).

[0066] In the method provided in the embodiments of the present disclosure, the three-dimensional map of the space where the robot is located involved in step S202 can be preformed and stored in a server or a terminal device, etc. In addition, this step can also be performed simultaneously with the two-dimensional map constructed by the robot involved in step S204. The embodiments of the present disclosure do not limit this.

[0067] In step S206, the three-dimensional map is matched with the two-dimensional map constructed by the robot to obtain the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot. Through the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot, the objects in the two-dimensional map can be correspondingly displayed in the three-dimensional map.

[0068] In some embodiments, for example, the three-dimensional map can be projected in the direction of the ground plane to generate a two-dimensional projection map, obtain the correspondence relationship between the two-dimensional projection map and the three-dimensional map, and then the two-dimensional projection map is matched with the two-dimensional map constructed by the robot, for example, through an optimization algorithm to obtain the correspondence relationship between the two maps when the overlapping area of the two maps is maximum, and then the correspondence relationship between the three-dimensional map and the two-dimensional map is obtained. The specific implementation can refer to Figure 5 This will not be described in detail here.

[0069] In some embodiments, the three-dimensional map and the two-dimensional map can be matched according to the positions of the markers in the three-dimensional map and the corresponding markers in the two-dimensional map. For details, reference can be made to the specific embodiments Figure 8 which are not described in detail herein.

[0070] In step S208, the pose of the robot on the two-dimensional map constructed by the robot is obtained. The pose can be estimated by fusing various internal sensors (e.g., odometer, compass, accelerometer, etc.) set by the robot itself through a SLAM method, and simultaneously using external sensors (e.g., laser range finder, vision device, etc.) to perceive the environment, and correcting the pose of the robot by comparing the features of the environment, so as to obtain the pose of the robot on the two-dimensional map.

[0071] In the method provided by the embodiments of the present disclosure, the pose of the robot on the two-dimensional map obtained in step S208 can be obtained while the two-dimensional map is constructed by the robot, or can be obtained after the construction of the two-dimensional map is completed, and the embodiments of the present disclosure do not limit this.

[0072] In step S210, the pose of the robot in the three-dimensional map is displayed according to the pose of the robot on the two-dimensional map constructed by the robot and the correspondence between the three-dimensional map and the two-dimensional map. After the correspondence between the three-dimensional map and the two-dimensional map is obtained, the pose of the robot on the two-dimensional map can be corresponded to the three-dimensional map, and the three-dimensional model of the robot and the three-dimensional map can be combined and displayed in the same scale when displayed, so that the real-time pose of the robot in the three-dimensional map can be intuitively observed on the AR display device.

[0073] According to the method for displaying the pose of the robot in the three-dimensional map provided by the embodiments of the present disclosure, the three-dimensional map of the space where the robot is located and the two-dimensional map constructed by the robot are obtained, the three-dimensional map and the two-dimensional map are matched to obtain the correspondence, and then the pose of the robot in the three-dimensional map is displayed according to the pose of the robot on the two-dimensional map and the correspondence between the three-dimensional map and the two-dimensional map, so that the height information of the obstacles in the area where the robot is located can be obtained.

[0074] Figure 5 is a flowchart of a method for matching a three-dimensional map and a two-dimensional map according to an example embodiment. As shown in the method Figure 5 may be applied to the server side of the system described above, or can be applied to the terminal device of the system described above.

[0075] Referring to Figure 5 the method 50 provided by the embodiments of the present disclosure can include the following steps.

[0076] In step S502, an effective part of the three-dimensional map is obtained. The scheme provided in the embodiments of the present disclosure is to install an LDS on a robot to realize measurement of obstacles around the robot and drawing of a two-dimensional map. The LDS arranged on the robot can have a certain field of view angle in the vertical direction, and therefore the range scanned by the LDS can also be a three-dimensional region. Generally, due to problems such as that the robot itself is relatively low and the field of view angle of the LDS in the vertical direction is limited, the LDS cannot scan a relatively high position in the environment where the robot is located, and therefore the two-dimensional map generated based on the LDS only includes a part close to the ground in the environment where the robot is located.

[0077] In some embodiments, for example, the part of the three-dimensional map at a corresponding height can be selected from the three-dimensional map according to the height of the LDS and the field of view angle of the LDS in the vertical direction, and the corresponding part of the three-dimensional map and the scanning range of the LDS of the robot is obtained.

[0078] In another embodiment, for example, the point cloud of the three-dimensional map can be filtered to remove the point cloud outside the scanning range of the robot, and then the correspondence between the point cloud of the three-dimensional map in the scanning range of the robot and the three-dimensional map is obtained as the part of the three-dimensional map. For details, refer to Figure 7 , which will not be described in detail here.

[0079] In step S504, the effective part of the three-dimensional map is projected to a horizontal plane to obtain a two-dimensional projection map. The point cloud of the part of the three-dimensional map can be projected in the horizontal plane direction according to the correspondence with the three-dimensional map to obtain the two-dimensional projection map.

[0080] In step S506, the two-dimensional projection map is matched with the two-dimensional map constructed by the robot to obtain the correspondence between the two-dimensional projection map and the two-dimensional map constructed by the robot. In the embodiments of the present disclosure, the two-dimensional projection map can be matched with the two-dimensional grid map by using a method of maximizing the overlapping area, that is, the two-dimensional projection map can be represented in the coordinate system of the two-dimensional grid map (or the two-dimensional grid map can be represented in the coordinate system of the two-dimensional projection map), and the overlapping area of the two maps is calculated while rotating, translating and the like, and iteration is continuously performed to obtain the correspondence between the two-dimensional projection map and the two-dimensional grid map when the overlapping area of the two-dimensional projection map and the two-dimensional grid map is maximized. Figure 6 A process of matching the two-dimensional projection map with the two-dimensional map constructed by the robot is shown in a schematic diagram as shown in Figure 6 From top to bottom, the overlapping area of the two-dimensional projection map 602 and the two-dimensional map 604 gradually increases until the process approaches coincidence, and the two-dimensional projection map 602 and the two-dimensional map 604 coincide to complete the matching, and the rotation and translation parameters can be obtained.

[0081] In step S508, the correspondence between the three-dimensional map and the two-dimensional map constructed by the robot is determined according to the correspondence between the two-dimensional projection map and the two-dimensional map. Since the two-dimensional projection map is obtained from the three-dimensional map, the correspondence between the two-dimensional projection map and the three-dimensional map can be obtained, and then the correspondence between the three-dimensional map and the two-dimensional map can be determined according to the correspondence between the two-dimensional projection map and the two-dimensional map constructed by the robot.

[0082] According to the method provided by the embodiment of the present disclosure, the part of the three-dimensional map within the scanning range of the robot is projected in the horizontal plane direction to obtain a two-dimensional projection map, then the two-dimensional projection map is matched with the two-dimensional map constructed by the robot, and then the correspondence between the three-dimensional map and the two-dimensional map is determined according to the correspondence between the two-dimensional projection map and the two-dimensional map constructed by the robot, which can effectively prevent the situation that the matching effect is poor due to the projection of the image of the object outside the scanning range of the robot in the three-dimensional map to the two-dimensional projection map.

[0083] Figure 7 The method shown in FIG. 5 is used to match the three-dimensional map and the two-dimensional map. Figure 5 The process diagram of step S502 in an embodiment is shown. The three-dimensional map includes a three-dimensional point cloud of the space where the robot is located, and the part of the three-dimensional map includes a three-dimensional point cloud of the space within the LDS scanning range of the robot. As shown in FIG. 6, in the embodiment of the present disclosure, step S502 can further include the following steps. Figure 7

[0084] In step S5022, the three-dimensional point cloud not greater than the LDS scanning range of the robot is selected from the three-dimensional point cloud. The three-dimensional point cloud not greater than the LDS scanning range of the robot can be selected based on the coordinates of the three-dimensional point cloud in the coordinate axis perpendicular to the ground in the map coordinate system (such as the earth coordinate system). For example, the LDS of the floor cleaning robot can have a scanning height of 15 cm, 20 cm or 25 cm, etc.

[0085] In step S5024, the part of the three-dimensional map is obtained based on the three-dimensional point cloud not greater than the LDS scanning range of the robot.

[0086] According to the method provided by the embodiment of the present disclosure, in the 3D-2D conversion process of the three-dimensional map, only the point cloud within the scanning range of the robot is projected to the ground plane by the 3D map to generate a 2D projection map, which can improve the accuracy of matching the three-dimensional map with the two-dimensional map constructed by the robot.

[0087] Figure 8 FIG. 7 is a flow chart of another method for matching a three-dimensional map and a two-dimensional map according to an example embodiment. As shown in FIG. 7, the method can be applied to the server side of the system, or can be applied to the terminal device of the system. Figure 8

[0088] Reference is made to​​Figure 8 The method 80 provided by the embodiments of the present disclosure can include steps S802 to S806.

[0089] In step S802, the marker of the specified obstacle in the three-dimensional map is obtained. The specified obstacle can be automatically identified by the AR scanning device when shooting, and the marker of the specified obstacle in the three-dimensional map is obtained. The specified obstacle can be, for example, a charging pile, a table, a chair, a wall, and the like. The wall plane can also be identified. After the marker is shot by the AR scanning device, the marker can be identified by the object recognition algorithm on the server through networking, or the marker photos in the cleaning environment can be pre-stored locally, and the object recognition algorithm on the local device can be used for matching identification.

[0090] In step S804, the marker of the specified obstacle in the two-dimensional map constructed by the robot is obtained. A shooting device can be arranged on the robot, and after the corresponding obstacle is identified by networking or a local algorithm, the specified obstacle is marked in the 2D map during the 2D map construction process.

[0091] In step S806, the marker of the specified obstacle in the three-dimensional map and the marker of the specified obstacle in the two-dimensional map constructed by the robot are matched to obtain the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot. If the markers of the specified obstacles are matched, at least three specified obstacles not on the same straight line (two markers of the specified obstacles are connected as a marker line), or a line of specified obstacles (such as the projection of the vertical plane of the wall on the horizontal plane, etc.) and a specified obstacle are needed. The rotation parameter is calculated by the marker line, and the translation parameter is calculated by correlating the feature points of the markers, so as to obtain the rotation parameter and the translation parameter for matching the three-dimensional map and the two-dimensional map.

[0092] According to the method provided by the embodiments of the present disclosure, the three-dimensional map and the two-dimensional map are matched by the specified obstacles identified during the mapping process, which improves the accuracy of map matching to a certain extent.

[0093] Figure 9A is a flowchart of a robot working method according to an exemplary embodiment. As Figure 9A The method shown in the figure can be applied to the server side of the system described above, or can be applied to the terminal device of the system described above.

[0094] Reference Figure 9A The method 90 provided by the embodiments of the present disclosure can include the following steps.

[0095] In step S902, a three-dimensional map of a space where the robot is located is obtained. After the AR scanning device draws the three-dimensional map of the space where the robot is located, the three-dimensional map can be shared with the AR display device.

[0096] In step S904, real-time scanning results of the robot are obtained. The robot can scan the surrounding environment during the working process based on a SLAM method to obtain information of objects such as obstacles. Figure 9B A robot is shown, which is provided with front and side ToF (Time of Flight) sensor modules for perceiving the environment. The ToF sensor obtains the distance of a target object by continuously sending light pulses to the target and receiving the light returned from the object by the sensor, and obtains the distance of the target object by detecting the flight (round trip) time of the light pulses.

[0097] In step S904, the three-dimensional map is modified according to the real-time scanning results of the robot. The robot can send the real-time scanning results to the AR display device, and the AR display device supplements or corrects the three-dimensional map according to the real-time scanning results of the robot.

[0098] According to the method provided by the embodiments of the present disclosure, the 3D map generated by the AR scanning device can be supplemented or corrected according to the real-time scanning results of the robot, and the accuracy of the displayed 3D map is improved.

[0099] Figure 10 is a block diagram of a robot three-dimensional map pose display device according to an exemplary embodiment. As Figure 10 The device shown can be applied to the server side of the system described above, or can be applied to the terminal device of the system described above.

[0100] Referring to Figure 10 , the device 100 provided by the embodiments of the present disclosure can include a three-dimensional map acquisition module 1002, a constructed map acquisition module 1004, a map matching module 1006, a pose acquisition module 1008, and a three-dimensional display module 1010.

[0101] The three-dimensional map acquisition module 1002 can be used to obtain a three-dimensional map of a space where the robot is located.

[0102] The constructed map acquisition module 1004 can be used to obtain a two-dimensional map constructed by the robot.

[0103] The map matching module 1006 can be used to match the three-dimensional map with the two-dimensional map constructed by the robot to obtain a corresponding relationship between the three-dimensional map and the two-dimensional map constructed by the robot.

[0104] The pose acquisition module 1008 can be used to obtain a pose of the robot on the two-dimensional map constructed by the robot.

[0105] The three-dimensional display module 1010 can be used to display the pose of the robot in the three-dimensional map according to the pose of the robot on the two-dimensional map constructed by the robot and the corresponding relationship between the three-dimensional map and the two-dimensional map constructed by the robot.

[0106] Figure 11 is a block diagram of a robot three-dimensional map pose display device according to an exemplary embodiment. As shown in the figure, the device can be applied to the server end of the system described above, or to the terminal device of the system described above. Figure 11

[0107] With reference to Figure 11 , the device 110 provided by the embodiments of the present disclosure can include a three-dimensional map acquisition module 1102, a constructed map acquisition module 1104, a map matching module 1106, a pose acquisition module 1108, a three-dimensional display module 1110, and a three-dimensional map correction module 1112; the map matching module 1106 can include a map selection module 11062, a two-dimensional projection module 11064, a two-dimensional map matching module 11066, a three-dimensional map matching module 11068, a first obstacle marker acquisition module 110692, a second obstacle marker acquisition module 110694, and a marker matching module 110696.

[0108] The three-dimensional map acquisition module 1102 can be used to obtain a three-dimensional map of a space in which a robot is located.

[0109] The constructed map acquisition module 1104 can be used to obtain a two-dimensional map constructed by a robot. The obstacle data obtained during the process of constructing a two-dimensional map by the robot is three-dimensional data.

[0110] The map matching module 1106 can be used to match the three-dimensional map with the two-dimensional map constructed by the robot, to obtain a corresponding relationship between the three-dimensional map and the two-dimensional map constructed by the robot.

[0111] The map selection module 11062 can be used to obtain an effective part of the three-dimensional map.

[0112] The map selection module 11062 can also be used to: determine a scanning range of the robot according to the three-dimensional data; and determine that the three-dimensional map located in the scanning range of the robot is the effective part of the three-dimensional map.

[0113] The two-dimensional projection module 11064 can be used to project the effective part of the three-dimensional map to a horizontal plane, to obtain a two-dimensional projection map.

[0114] The two-dimensional map matching module 11066 can be used to match the two-dimensional projection map with the two-dimensional map constructed by the robot, to obtain a corresponding relationship between the two-dimensional projection map and the two-dimensional map constructed by the robot.

[0115] The two-dimensional map matching module 11066 can also be used to match the two-dimensional projection map with the two-dimensional map constructed by the robot by using a method of maximizing an overlapping area; and obtain the corresponding relationship between the two-dimensional projection map and the two-dimensional map constructed by the robot when the overlapping area of the two-dimensional projection map and the two-dimensional map constructed by the robot is maximized. ​

[0116] The three-dimensional map matching module 11068 can be configured to determine the correspondence between the three-dimensional map and the two-dimensional grid map according to the correspondence between the two-dimensional projection map and the two-dimensional grid map.

[0117] The first obstacle marker obtaining module 110692 can be configured to obtain the marker of the specified obstacle in the three-dimensional map. The specified obstacles include a plurality of obstacles, and the plurality of specified obstacles are not located on a straight line. The specified obstacles include charging piles and walls.

[0118] The second obstacle marker obtaining module 110694 can be configured to obtain the marker of the specified obstacle in the two-dimensional map constructed by the robot.

[0119] The marker matching module 110696 can be configured to match the marker of the specified obstacle in the three-dimensional map and the marker of the specified obstacle in the two-dimensional map constructed by the robot, to obtain the correspondence between the three-dimensional map and the two-dimensional map constructed by the robot.

[0120] The pose obtaining module 1108 can be configured to obtain the pose of the robot on the two-dimensional map constructed by the robot.

[0121] The three-dimensional display module 1110 can be configured to display the pose of the robot in the three-dimensional map according to the pose of the robot on the two-dimensional map constructed by the robot and the correspondence between the three-dimensional map and the two-dimensional map constructed by the robot.

[0122] The three-dimensional display module 1110 can also be configured to display the three-dimensional model of the robot in the three-dimensional map in a scale.

[0123] The three-dimensional map correction module 1112 can be configured to modify the three-dimensional map according to the two-dimensional map constructed by the robot when the robot constructs the two-dimensional map.

[0124] Figure 12 is a block diagram of a robot three-dimensional map pose display system according to an example embodiment. As shown in FIG. 10, the robot three-dimensional map pose display system includes a two-dimensional map constructing module 1002, a three-dimensional map constructing module 1004, a two-dimensional map matching module 1006, a three-dimensional map matching module 11068, a first obstacle marker obtaining module 110692, a second obstacle marker obtaining module 110694, a marker matching module 110696, a pose obtaining module 1108, and a three-dimensional display module 1110. Figure 12As shown, the sweeping robot 1202 provided with the depth sensor 12021 can be connected with the augmented reality scanning device 1204, and the sweeping robot 1202 can acquire the 3D map of the environment in real time through the augmented reality scanning device 1204; the sweeping robot 1202 can also obtain the 3D map drawn by the augmented reality scanning device 1204 and save it when sweeping for the first time or when reset by the user. The sweeping robot 1202 can be connected with the augmented reality display device 1206, and the 2D grid map and the pose generated based on the depth sensor 12021 can be sent to the augmented reality display device 1206 in real time, and the point cloud of the observed obstacles can also be uploaded to the augmented reality display device 1206, so that the augmented reality display device 1206 updates the 3D map; the augmented reality scanning device 1204 can also be connected with the augmented reality display device 1206, and the augmented reality scanning device 1204 can share the 3D map drawn to the augmented reality display device 1206, so that the augmented reality display device 1206 matches the 2D grid map generated by the sweeping robot 1202 with the 3D map, obtains the corresponding relationship between the 2D grid map and the 3D map and saves it. When the sweeping robot 1202 sweeps the area again, the sweeping robot 1202 uploads the pose information to the augmented reality display device 1206 in real time, and the augmented reality display device 1206 displays the pose of the sweeping robot 1202 in the 3D map in real time according to the saved corresponding relationship between the 2D grid map and the 3D map.

[0125] Figure 13 A structural schematic diagram of an electronic device in an embodiment of the present disclosure is shown. It should be noted that, Figure 13 The device shown is only an example of a computer system, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0126] As Figure 13 shown, the device 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1302 or programs loaded from a storage portion 1308 to a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 are also stored. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other through a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0127] The following components are connected to the I / O interface 1305: an input part 1306 including a keyboard, a mouse, etc.; an output part 1307 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1308 including a hard disk, etc.; and a communication part 1309 including a network interface card such as a LAN card, a modem, etc. The communication part 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as necessary. A removable media 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1310 as necessary, so that a computer program read out therefrom is installed in the storage part 1308 as necessary.

[0128] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 1309, and / or installed from the removable media 1311. When the computer program is executed by the central processing unit (CPU) 1301, the above-described functions defined in the system of the present disclosure are executed.

[0129] It should be noted that the computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, etc., or any suitable combination of the above.

[0130] The flow diagrams and block diagrams in the drawings are illustrations of possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0131] The modules described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. The described modules can also be arranged in a processor, for example, a processor can be described as including a three-dimensional map acquisition module, a constructed map acquisition module, a map matching module, a pose acquisition module, and a three-dimensional display module. Among them, the names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the three-dimensional map acquisition module can also be described as "a module that acquires a three-dimensional map to the connected AR rendering device".

[0132] As another aspect, the present disclosure also provides a computer readable medium, which can be included in the device described in the above embodiments; or can exist independently without being assembled into the device. The above computer readable medium carries one or more programs, when the one or more programs are executed by the device, the device includes: obtaining a three-dimensional map of a space where a robot is located; obtaining a two-dimensional map constructed by the robot; matching the three-dimensional map with the two-dimensional map constructed by the robot to obtain a correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot; obtaining a pose of the robot on the two-dimensional map constructed by the robot; and displaying the pose of the robot in the three-dimensional map according to the pose of the robot on the two-dimensional map constructed by the robot and the correspondence relationship between the three-dimensional map and the two-dimensional map constructed by the robot.

[0133] The exemplary embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A method for displaying the pose of a robot on a 3D map, characterized in that, include: A three-dimensional map of the space where the robot is located is obtained. The three-dimensional map is drawn by acquiring spatial data and drawing it based on a mobile device with three-dimensional map drawing function set up in the space where the robot is located. Obtain the two-dimensional map constructed by the robot; The 3D map is matched with the 2D map constructed by the robot to obtain the correspondence between the 3D map and the 2D map constructed by the robot; Obtain the pose of the robot on the two-dimensional map constructed by the robot; The robot's pose is displayed in the 3D map based on the robot's pose on the 2D map constructed by the robot and the correspondence between the 3D map and the 2D map constructed by the robot.

2. The method according to claim 1, characterized in that, Matching the 3D map with the 2D map constructed by the robot to obtain the correspondence between the 3D map and the 2D map constructed by the robot includes: Obtain the effective portion of the 3D map; The effective portion of the three-dimensional map is projected onto a horizontal plane to obtain a two-dimensional projected map. Match the two-dimensional projected map with the two-dimensional map constructed by the robot to obtain the correspondence between the two-dimensional projected map and the two-dimensional map constructed by the robot.

3. The method according to claim 2, characterized in that, The obstacle data acquired by the robot during the process of constructing a two-dimensional map is three-dimensional data; The effective portion of obtaining the 3D map includes: The scanning range of the robot is determined based on the three-dimensional data; The 3D map within the scanning range of the robot is determined to be a valid part of the 3D map.

4. The method according to claim 2, characterized in that, Matching the two-dimensional projected map with the two-dimensional map constructed by the robot to obtain the correspondence between the two-dimensional projected map and the two-dimensional map constructed by the robot includes: The two-dimensional projected map is matched with the two-dimensional map constructed by the robot by maximizing the overlap area. Obtain the correspondence between the two-dimensional projected map and the two-dimensional map constructed by the robot when the overlap area between the two-dimensional projected map and the two-dimensional map constructed by the robot is maximized.

5. The method according to claim 1, characterized in that, Matching the 3D map with the 2D map constructed by the robot to obtain the correspondence between the 3D map and the 2D map constructed by the robot includes: Obtain the markers of specified obstacles in the 3D map; Obtain the markers of the specified obstacles in the two-dimensional map constructed by the robot; The correspondence between the three-dimensional map and the two-dimensional map constructed by the robot is obtained by matching the markers of the specified obstacle in the three-dimensional map with the markers of the specified obstacle in the two-dimensional map constructed by the robot.

6. The method according to claim 5, characterized in that, The designated obstacles include multiple obstacles, and the multiple designated obstacles are not located in a straight line.

7. The method according to claim 6, characterized in that, The designated obstacles include charging piles and walls.

8. The method according to any one of claims 1 to 7, characterized in that, Also includes: When the robot is building a two-dimensional map, the three-dimensional map is modified based on the two-dimensional map built by the robot.

9. The method according to any one of claims 1 to 7, characterized in that, Also includes: The robot's 3D model is displayed proportionally to the 3D map.

10. A robot 3D map pose display device, characterized in that, include: A 3D map acquisition module is used to obtain a 3D map of the space where the robot is located. The 3D map is obtained by a mobile device with 3D map drawing function set in the space where the robot is located, which acquires spatial data and draws the map accordingly. A map acquisition module is constructed to obtain a two-dimensional map constructed by the robot; The map matching module is used to match the three-dimensional map with the two-dimensional map constructed by the robot to obtain the correspondence between the three-dimensional map and the two-dimensional map constructed by the robot. The pose acquisition module is used to obtain the pose of the robot on the two-dimensional map constructed by the robot; A 3D display module is used to display the robot's pose in a 3D map based on the robot's pose on a 2D map constructed by the robot and the correspondence between the 3D map and the 2D map constructed by the robot.

11. An apparatus comprising: A memory, a processor, and executable instructions stored in the memory and executable in the processor, characterized in that the processor, when executing the executable instructions, implements the method as described in any one of claims 1-9.

12. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement the method as described in any one of claims 1-9.

Images