Three-dimensional map construction method and robot

Abstract

The invention relates to a three-dimensional map construction method and a robot, and the method comprises the steps: controlling a sensor of the robot to collect data when the robot is in a lifting collection mode; in the lifting collection mode, the robot is in a lifted state; and performing three-dimensional reconstruction based on the acquired data of the sensor to obtain a three-dimensional map. According to the method, when the robot is in the lifting collection mode, the sensor of the robot is controlled to collect data, and in the lifting collection mode, the robot is in the lifted state, so that the visual angle of the robot is lifted, the sensing range of the sensor of the robot is increased under the lifted visual angle, and the collected environment data is enriched; therefore, three-dimensional reconstruction is carried out by using the data acquired by the robot sensor at a relatively high visual angle, and a high-quality three-dimensional map can be obtained.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a three-dimensional map construction method and a robot. Background Art

[0002] With the development of robot technology, robots have greatly brought convenience to people's lives. Especially with the development of cleaning robots and AI technology, the development of the cleaning robot industry has ushered in a new booming period. Cleaning robots are becoming an important component of homes and other entertainment venues, playing an increasingly important role in cleaning services.

[0003] For robots, 3D reconstruction is mainly used to build scene information and various maps, so that the robot can know its position in the world environment and can perceive the location information of surrounding obstacles. Through path planning, the robot can achieve positioning, navigation, obstacle avoidance and path planning. Therefore, accurate 3D reconstruction is very important for improving the performance of robots. Traditional 3D reconstruction methods have the problem of poor reconstruction effect. Summary of the invention

[0004] Based on this, it is necessary to provide a three-dimensional map construction method, device, robot, computer-readable storage medium and computer program product that can improve the construction quality in response to the above technical problems.

[0005] In a first aspect, the present application provides a three-dimensional map construction method, the method comprising:

[0006] When the robot is in a lifting collection mode, the sensor of the robot is controlled to collect data; in the lifting collection mode, the robot is in a lifted state;

[0007] Three-dimensional reconstruction is performed based on the collected data of the sensor to obtain a three-dimensional map.

[0008] In a second aspect, the present application provides a three-dimensional map construction device, the device comprising:

[0009] A collection module, used to control the robot's sensors to collect data when the robot is in a lifting collection mode; in the lifting collection mode, the robot is in a lifted state;

[0010] The construction module is used to perform three-dimensional reconstruction based on the collected data of the sensor to obtain a three-dimensional map.

[0011] In a third aspect, the present application provides a robot comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the above embodiments of the claims when executing the computer program.

[0012] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the methods of the above-mentioned embodiments.

[0013] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the methods of the above embodiments when executed by a processor.

[0014] The above-mentioned three-dimensional map construction method, device, robot, computer-readable storage medium and computer program product control the robot's sensors to collect data when the robot is in the lifting collection mode. In the lifting collection mode, the robot is in an elevated state, so that the robot's viewing angle is raised. At the elevated viewing angle, the perception range of the robot's sensors is increased, enriching the collected environmental data, thereby using the data collected by the robot's sensors at a higher viewing angle to perform three-dimensional reconstruction, and obtain a high-quality three-dimensional map. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of the structure of a robot in one embodiment;

[0016] Figure 2 A schematic diagram of signal flow between a sensing part and a control part of a robot in one embodiment;

[0017] Figure 3 A schematic diagram of a process of constructing a three-dimensional map in an embodiment;

[0018] Figure 4 A schematic flow chart of the steps of marking the area where the identified object of interest is located in the three-dimensional map in one embodiment;

[0019] Figure 5 This is a schematic diagram of an interface for confirming a prompt for focusing on an area where an item is located in one embodiment;

[0020] Figure 6 A schematic diagram of a flow chart of a three-dimensional map construction method in another embodiment;

[0021] Figure 7 It is a structural block diagram of a three-dimensional map construction device in one embodiment;

[0022] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The present application embodiment provides a robot, such as Figure 1 As shown, the robot comprises a robot body 101 and an image module 102. The image module 102 can be installed on the top or front side of the robot body 101.

[0025] Specifically, the image module 102 may include a digital image acquisition device and a depth image acquisition device.

[0026] like Figure 2 As shown, the sensing and control part 20 of the robot includes:

[0027] Controller 201, driver 202, multiple types of sensors 203 and communication module 204. Multiple types of sensors may include image modules, IMU (Inertial Measurement Unit), laser radar, infrared sensors, collision sensors, ultrasonic sensors, wall sensors, cliff sensors and anti-fall sensors. Among them, each sensor in the multiple types of sensors 203 and the driver 202 are connected to the controller 201 by signal. The controller 201 processes the sensor data collected by the multiple types of sensors, makes decisions, generates control instructions based on the decision results, and sends control instructions to the driver 202. The driver 202 may include a driver for the robot driving wheel, a driver for the module adjustment device, etc. The decision may include navigation, obstacle avoidance and control of various components of the robot. Based on the communication module 204, it can communicate with the server 10, and the server 10 can be connected to the user terminal 30 for communication. The control instructions can be sent to the robot through the user terminal 30. For example, robot operation instructions, robot charging instructions, robot map building instructions, etc. It should be understood that the user terminal 30 is installed with robot-related applications to achieve management of the robot.

[0028] The user terminal 30 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0029] The robot refers to a self-moving device equipped with a sensor device that can move autonomously to perform tasks. In one embodiment, the robot includes but is not limited to a cleaning robot, a food delivery robot, and a logistics robot. The cleaning robot includes robots used for cleaning home and non-home scenes, such as sweepers, mops, automatic floor scrubbers, entertainment venues (such as swimming pool cleaners), and other cleaning robots.

[0030] In one embodiment, Figure 3 As shown, a three-dimensional map construction method is provided. Figure 2 The method is executed in the controller, user terminal or server shown in the figure, and includes the following steps:

[0031] Step 302, when the robot is in a lifting collection mode, control the robot's sensors to collect data; in the lifting collection mode, the robot is in a lifted state.

[0032] The lifting collection mode refers to a mode in which the robot collects data about the surrounding environment in a lifted state. That is, in the lifting collection mode, the robot is in a lifted state. In some embodiments, the robot can be actively lifted by an internal drive, actively lifted and in a lifted state, or lifted, raised or held by an external force and in a lifted state.

[0033] The robot is usually equipped with a drive wheel assembly. In the traditional data collection mode, the drive wheel assembly is controlled to move within the working range and collect data. The range of the robot's data collection is limited by the robot's own height and the sensor installation height. Therefore, the robot is generally on the same horizontal plane and can only see the world at a low angle, so the range of the robot's ability to perceive the surrounding environment is limited. Taking the cleaning robot as an example, a household cleaning robot is usually only about 10CM. The machine can only see objects at a low angle, and the viewing range and height are severely limited. As a result, it is likely that the machine will not be able to see all nearby or large objects, which can easily lead to low semantic recognition accuracy, thereby affecting the robot's navigation and obstacle avoidance capabilities.

[0034] In this embodiment, in the lift acquisition mode, the robot is in a lift state, which raises the height of the robot. In the lift acquisition mode, the viewing angle range and height that the machine can see are raised, increasing the probability of seeing the whole object. For example, in the sensing acquisition mode, the height of the robot is only about 10CM, but in the lift acquisition mode, the height of the robot can reach 1 meter. In comparison, the lift acquisition mode can increase the robot's visual range.

[0035] In the lift collection mode, the robot sensor is controlled to collect data. In the lift collection mode, the robot sensor collects data in the same manner as the robot sensor collects data in the traditional collection mode. For example, the sensor collection frequency can be the same.

[0036] Take the robot being picked up and entering the lifting collection mode as an example. In the lifting collection mode, the robot passively follows the user's movement to collect data, and the walking speed is controlled by the user's walking speed. In some embodiments, the way the robot sensor collects data may be different from the way the robot sensor collects data in the traditional collection mode. For example, considering that the person's walking speed may be faster, the sensor's collection frequency can be increased. Alternatively, the sensor's collection frequency can be determined based on the user's walking speed. For example, when the user's walking speed is too fast, the sensor's collection frequency is increased, or when the user's walking speed is slow, the sensor's collection frequency is restored to the normal collection frequency.

[0037] In one embodiment, the robot's sensors include an image module, a lidar, and an inertial navigation module.

[0038] Among them, the image module refers to a sensor device and combination for acquiring image data, which is used to collect images. Image data may include digital images and depth images. In one embodiment, the image module includes a digital image acquisition device and a depth image acquisition device. The image acquisition device is not limited to conventional cameras and cameras, and includes various sensors for semantic recognition / navigation and obstacle avoidance after acquiring data. The depth image acquisition device is not limited to various sensors such as TOF / 3DTOF / binocular cameras that can directly or indirectly acquire depth images. The image module can also include an RGBD camera. Functionally, the RGBD camera adds a depth measurement function to the function of an ordinary RGB image acquisition device, and can collect digital images and depth images.

[0039] LiDAR emits lasers to sense the surrounding environment and obtain ranging information.

[0040] The inertial navigation module may be an MU (Inertial Measuring Unit), which is used to obtain position and posture information.

[0041] Step 304: Perform three-dimensional reconstruction based on the data collected by the sensor to obtain a three-dimensional map.

[0042] In one embodiment, the data collected by the sensor in the elevated acquisition mode can be used for three-dimensional reconstruction to obtain a three-dimensional map. In one embodiment, the data collected by the sensor in the elevated acquisition mode and the underground acquisition mode can be combined for three-dimensional reconstruction to obtain a three-dimensional map.

[0043] The three-dimensional point cloud can be obtained based on the digital image and the depth image. Specifically, the three-dimensional point cloud information is obtained by aligning the digital image and the depth image and fusing the two.

[0044] Specifically, alignment refers to determining the correspondence between pixels in a digital image and pixels in a depth image. The specific alignment method may use a classical algorithm or a trained model to achieve alignment of image and depth data.

[0045] One alignment method can be: obtain the intrinsic matrix of the image acquisition device, the intrinsic matrix of the depth acquisition device, and the extrinsic rotation matrix and offset matrix of the image acquisition device and the depth acquisition device, traverse the coordinates of each pixel point in the digital image, and calculate the coordinates of the pixel points in the depth image corresponding to the pixel points in the digital image through the alignment formula, so as to obtain the correspondence between the pixel points in the digital image and the pixel points in the depth image. On this basis, based on the correspondence between the pixel points in the digital image and the pixel points in the depth image, the digital image and the depth image are fused to obtain a three-dimensional point cloud.

[0046] Correspondingly, according to the correspondence between the pixels in the digital image and the pixels in the depth image, the depth value of each pixel in the digital image (i.e., the distance from a certain point in the real scene corresponding to the pixel to the camera) can be obtained, and then each pixel in the digital image is converted to the world coordinate system to obtain the three-dimensional point cloud information corresponding to the frame of digital image and the depth image. In one embodiment, the digital image can be an RGB image, so that each frame of three-dimensional point cloud information includes three-dimensional space coordinates (relative to the module center) and three-dimensional space point cloud information of color information (RGB), as well as corresponding timestamp information.

[0047] After obtaining the three-dimensional point cloud information, three-dimensional reconstruction is performed based on multiple frames of the three-dimensional point cloud and the corresponding robot posture information to obtain a three-dimensional map.

[0048] The above-mentioned three-dimensional map construction method controls the robot's sensors to collect data when the robot is in the lifting collection mode. In the lifting collection mode, the robot is in an elevated state, so that the robot's viewing angle is raised. At the elevated viewing angle, the perception range of the robot's sensors is increased, enriching the collected environmental data, thereby using the data collected by the robot's sensors at a higher viewing angle for three-dimensional reconstruction, and a high-quality three-dimensional map can be obtained.

[0049] In one embodiment, performing three-dimensional reconstruction based on the collected data of the sensor to obtain a three-dimensional map may include: obtaining a first three-dimensional point cloud in a lift collection mode and first robot posture information corresponding to each frame of the first three-dimensional point cloud based on the data collected by the sensor; obtaining a second three-dimensional point cloud in a ground collection mode and second robot posture information corresponding to each frame of the second three-dimensional point cloud; performing three-dimensional reconstruction based on multiple frames of the first three-dimensional point cloud, the posture information of the first robot, the second three-dimensional point cloud and the second robot posture information to obtain a three-dimensional map.

[0050] This implementation combines the data collected by sensors in the elevation collection mode and the underground collection mode for three-dimensional reconstruction to obtain a three-dimensional map. This method can utilize the environmental information of a higher perspective collected in the elevation collection mode and the environmental information of a lower perspective collected in the underground collection mode, enriching the collected environmental data and improving the construction quality of the three-dimensional map.

[0051] Specifically, the real-time posture of the robot is the real-time posture corresponding to each frame of the three-dimensional point cloud of the robot. The posture information of the robot is the basis for building a three-dimensional map. The posture data of the robot can be obtained based on digital images and depth images, or based on an inertial navigation module.

[0052] The method of obtaining the robot posture data based on the digital image and the depth image can be obtained by geometric means. Specifically, by obtaining the feature points of the previous and next frame data, the feature points are matched, and then the posture transformation of each frame relative to the previous frame or relative to the initial position can be calculated by multi-view geometry, and then the posture when each frame data is obtained can be calculated.

[0053] The robot pose data can be obtained based on digital images and depth images, or it can be obtained using a trained model. The corresponding network model is trained in advance by deep learning (supervised learning), and then the digital image and depth image are input into the network to generate the estimated pose for use as the pose of each frame of data.

[0054] In another way, the calculated pose and the measured pose of the inertial navigation module can be fused to obtain the real-time pose. Using the three-dimensional point cloud and pose, a three-dimensional map can be constructed.

[0055] In another embodiment, the three-dimensional map construction method further includes: performing identification based on the collected data to obtain an object recognition result; obtaining an object of interest in the object recognition result; and for the object of interest in the object recognition result, marking an area where the identified object of interest is located in the three-dimensional map icon.

[0056] In this application, by adding a lifting acquisition mode, the accuracy and range of the three-dimensional map are improved, and the accuracy of semantic recognition can be further improved. Specifically, in the three-dimensional reconstruction process, it is also possible to identify based on the three-dimensional point cloud data to obtain object recognition results in the environment. The recognition method includes: clustering and segmenting the point cloud to obtain a segmented point cloud, inputting the segmented point cloud into the image recognition model, and obtaining an object recognition result. The object recognition result includes the object category and confidence level of each point cloud. The object recognition result can provide a basis for robot navigation, operation, and escape.

[0057] Currently, there are products on the market that display 3D maps. They are based on 2D maps and find the corresponding semantically recognized 3D models from the template library for display. There is no RGB information. Therefore, the display effect is very different from the actual situation, and the user's senses and experience are not very good. After 3D reconstruction and semantic recognition, this application directly displays a 3D map similar to the real scene, and a 3D semantic box indicates the result of semantic recognition.

[0058] On this basis, in this embodiment, the object of interest can be further determined based on the object recognition result. The object of interest is an object that the user actively pays attention to and is worthy of the user's attention. The object of interest can be determined based on the value of the object, the fragility of the object, and the degree of danger of the object. For example, the objects of interest can include valuables, dangerous items, and fragile items. In one embodiment, the category of the object of interest can also be pre-set by the user. After the object of interest is identified, the area where the identified object of interest is located is marked in the three-dimensional map based on the location information of the object of interest. By marking the area where the object of interest is located, the object of interest can be highlighted in the three-dimensional map to serve as a prompt or warning.

[0059] In another embodiment, Figure 4 As shown, the marking of the area where the identified object of interest is located in the three-dimensional map icon includes:

[0060] Step 402: Perform recognition based on the collected data to obtain an object recognition result.

[0061] Specifically, the point cloud is clustered and segmented to obtain a segmented point cloud, and the segmented point cloud is input into the image recognition model to obtain an object recognition result. The object recognition result includes the object category, confidence, object location information, etc. of each point cloud. The object recognition result can provide a basis for robot navigation, operation and escape. Among them, the image recognition model can be obtained by supervised learning training. Specifically, the segmented point cloud in the sample set is input into the image recognition model to be trained to obtain a prediction result, and the cross entropy loss is obtained according to the difference between the prediction result and the annotation result of the segmented point cloud. The parameters of the image recognition model are adjusted according to the cross entropy loss until the model is stable to obtain a trained image recognition model.

[0062] Step 404: Obtain the object of interest in the object recognition result.

[0063] The category of the object of interest can be pre-set, and the object category is searched in the pre-set category of the object of interest. If the object can be found, the object belongs to the object of interest. For example, it is pre-set that the objects of interest include porcelain. If the object recognition result includes porcelain, it can be determined that the object of interest is included in the object recognition result. The initial category of the object of interest can be set by the developer based on experience. During use, the user can add or delete the category of the object of interest through the user terminal according to the items in the working environment. Among them, the user terminal is installed with an application (APP), and the robot can be managed through the APP. Another implementation method can also be to feedback the object recognition result through the application, and the user selects to determine the object of interest.

[0064] Step 406: Mark the object of interest in the three-dimensional map in different marking ways according to the category of the object of interest.

[0065] Different categories of objects of concern may be marked in different ways. For example, for dangerous or fragile items, the area where the relevant objects are located may be displayed in red. For another example, for valuable items, the area where the relevant objects are located may be displayed in gold. The above marking methods are only examples. In other implementations, other marking methods may be used, as long as the areas where the objects of concern of different categories are located can be distinguished.

[0066] In this embodiment, by marking the area where the object of interest is located in the three-dimensional map in different marking methods according to the categories of the object of interest, it is convenient to distinguish different object categories and can play a role in prompting or warning the object of interest.

[0067] In another embodiment, the area where the object of interest is located is identified in the three-dimensional map in different identification methods according to the category of the object of interest, including: determining a target area where the object of interest is located; outputting a confirmation prompt for the target area, the confirmation prompt including the location of the target area and the corresponding category of the object of interest; determining a target identification method for the target area according to the confirmation result of the target area and the category of the object of interest; the identification methods for target areas where objects of interest of different categories are located are different; and identifying the target area in the three-dimensional map according to the target identification method.

[0068] Specifically, according to the recognition result of the object of interest, the area where the object of interest is located can be further sent to the user for confirmation. Based on the user's confirmation, the robot can know what type of object of interest exists in the area, and then different processing methods can be adopted for areas where different objects of interest are located.

[0069] Specifically, after the object of interest is determined, the object of interest and the area where it is located are output to the application for the user to confirm. For example, the user is prompted to confirm through an APP box selection or pop-up window. The user can confirm whether the category of the object of interest is accurate and whether the area where the object of interest is located is accurate. The user can also modify the category of the object of interest based on the actual situation in the environment. For example, if fragile items are identified in a certain area, but after confirmation by the user, the item is a valuable item, the category of the object of interest can be modified, and then the area can be determined as the area where the valuable item is located.

[0070] According to the confirmation result of the target area confirmed by the user and the category of the object of interest, the target identification method of the target area is determined. The identification methods of the target areas where the objects of interest of different categories are located are different.

[0071] The marking methods of the areas where the objects of interest of different categories are located are preset. For example, the area where fragile items are located can be displayed with a red background and appropriate transparency, and the area where valuable items are located can be displayed with a gold background and appropriate transparency.

[0072] By marking different areas where items of concern are located, different treatment methods can be used for different areas. For example, for areas where fragile items are located, the robot can use a safer speed and safe distance when cleaning the area, or set a restricted area for the area and not allow the robot to enter. For example, for areas where valuable items are located, a restricted area can be set for the area to avoid possible damage to the valuable items due to work reasons.

[0073] In some embodiments, multiple types of items of interest may be placed in the same area or adjacent areas, such as fragile items and valuable items placed in one area, and the areas of interest may overlap. However, considering the personal safety and property safety of users, overlapping areas where items of interest are located is acceptable.

[0074] A confirmation prompt of an implementation manner is as follows Figure 5 As shown, the category of the object of interest, the item image and the target area image are included, and a category confirmation button is provided. When the user triggers the corresponding category confirmation button, the object of interest and the category of the object of interest confirmed by the user can be obtained.

[0075] In this embodiment, the user's confirmation of the area where the object of interest is located is obtained by interacting with the user. The interactive method can improve the accuracy of identifying the area where the object of interest is located.

[0076] In another embodiment, the three-dimensional map construction method further includes: determining a restricted area based on the area where the object of interest is located; and marking the restricted area in the three-dimensional map according to the restricted area.

[0077] The restricted area refers to the area where the robot is prohibited from passing. Based on the restricted area marked in the three-dimensional map, the robot will not enter the area during operation.

[0078] In one embodiment, a restricted area is determined based on the area where the object of concern is located and an appropriate safety distance. In one embodiment, the area where the object of concern with a high degree of concern is located can also be set as a restricted area. For example, for dangerous and fragile items, a restricted area can be set after adding an appropriate safety distance on the basis of the area. In one embodiment, in response to a restricted area setting operation, the area where the object of concern indicated by the restricted area setting operation is located is determined as a restricted area area, and this method can be selectively set as a restricted area by the user. For example, the area where valuables are located can be selectively set as a restricted area by the user.

[0079] like Figure 5 As shown, when the user confirms the target area where the object of interest is located, a button for setting a restricted area is also provided. After the user confirms that the area where the object of interest is located is set as a restricted area, the restricted area is set after adding an appropriate safety distance based on the area.

[0080] In some embodiments, the confirmation prompt may further indicate the hazards of not setting the restricted area. For example, if the user does not set the restricted area, the machine may cause damage to the item due to work reasons, and the user is interactively reminded to know.

[0081] For restricted areas, you can mark them in the 3D map, for example, by setting boundaries or marking them in black. The robot will not enter the restricted areas during operation, thus preventing the robot from touching valuables, dangerous goods or fragile items.

[0082] In this embodiment, by determining the restricted passage area based on the area where the object of interest is located, the robot can be prevented from touching valuables, dangerous items or fragile items, thereby ensuring the safety of the robot and the items.

[0083] In another embodiment, the three-dimensional map construction method further includes: in the lift acquisition mode, displaying the three-dimensional map constructed at the current moment in real time.

[0084] When the robot is in the lifting acquisition mode, the robot is in an idle state. At this time, it can perform 3D reconstruction while acquiring. The 3D reconstructed 3D map is displayed in real time for users to view. The advantage of this is that users can determine the position to be reconstructed based on the difference between the 3D map reconstructed at the current moment and the actual working scene, so as to facilitate the active adjustment of the robot's orientation position, so as to collect data at the corresponding position and improve the 3D map.

[0085] In another embodiment, the three-dimensional map construction method further includes: determining a target area to be collected based on the three-dimensional map reconstructed at the current moment; and outputting a collection path prompt based on the target area to be collected.

[0086] Specifically, the three-dimensional map can be detected to determine the area to be improved, and then the collection path from the current position to the target area to be collected can be output on the application according to the area to be improved. The collection path can be in the form of an arrow, for example, the application displays the collection path in the form of an arrow, indicating that the robot moves in this direction in the lifting state to collect the complete three-dimensional point cloud data. This method can actively give users collection prompts, which improves convenience.

[0087] In another embodiment, the three-dimensional map construction method further includes: acquiring a moving speed in the lift acquisition mode; and outputting a moving speed prompt when the moving speed does not meet acquisition requirements.

[0088] Specifically, in the underground collection mode, the robot collects data by moving itself. In the elevated collection mode, the robot passively follows the movement of people to collect data. However, the walking speed of people is unstable. Too fast movement speed affects the collection effect, and too slow movement speed affects the collection efficiency. Based on this, the robot monitors the movement speed in real time in the elevated collection mode. When the movement speed does not meet the collection requirements, the movement speed prompt is output.

[0089] The acquisition speed requirement is a preset speed range. If the moving speed in the lift acquisition mode is not within the range, a moving speed prompt is output to prompt the user to increase or slow down the moving speed.

[0090] The moving speed prompt may be a robot voice announcement, for example, when the moving speed is too fast, the robot may voice announce "please slow down", for example, when the moving speed is too slow, the robot may voice announce "please speed up". The moving speed prompt may also be through an application prompt.

[0091] In this embodiment, by monitoring the moving speed in the lifting acquisition mode and outputting a moving speed prompt when the moving speed does not meet the requirements, the user can be prompted to control the moving speed within the acquisition range in the lifting acquisition mode, thereby taking into account both acquisition quality and acquisition efficiency.

[0092] In another embodiment, the three-dimensional reconstruction method also includes: when at least one of the robot's recognition results and task processing results meets the trigger condition of the lifting acquisition mode, outputting a lifting reminder; when a lifting operation is detected, controlling the robot to enter the lifting acquisition mode.

[0093] The triggering condition of the acquisition mode may include: the recognition result of the robot includes the robot recognizing a specific object. The specific object may be preset by the system or by the user. For example, when the recognition result includes items such as table legs, since it is uncertain whether there are fragile items on the desktop, the triggering condition of the lifting acquisition mode may be triggered to perform three-dimensional reconstruction after lifting to determine whether there are fragile items on the desktop.

[0094] The triggering condition of the acquisition mode may include: the robot cannot recognize the object. When the robot cannot recognize the object, the triggering condition of the lifting acquisition mode may be triggered to perform three-dimensional reconstruction after lifting and accurately recognize the object.

[0095] The triggering condition of the acquisition mode may include: the robot cannot recognize the object. When the robot cannot recognize the object, the triggering condition of the lifting acquisition mode may be triggered to perform three-dimensional reconstruction after lifting, and perceive the surrounding environment based on the three-dimensional reconstruction result.

[0096] When the robot recognizes a specific object, the robot cannot recognize the object, or the robot cannot recognize the object, it triggers the lifting acquisition mode to have a high viewing angle, which can improve the accuracy of recognition. The specific object can be dangerous objects, valuable objects and fragile objects.

[0097] When the robot's recognition result meets the triggering conditions of the lift acquisition mode, a lift reminder is output. The lift reminder is used to remind the user to pick up the robot, or to instruct the robot to lift up actively. The lift reminder can be a robot voice broadcast, or it can be sent to the application, or the lift reminder is sent to the lift drive device for lifting.

[0098] This method can trigger the lift acquisition mode when the recognition effect is poor, and obtain a larger viewing angle, more accurate and more detailed three-dimensional reconstruction result in the world coordinate system, thereby improving the quality of image recognition.

[0099] In one embodiment, the triggering condition of the acquisition mode may include: when the processing result of the robot performing the task based on the two-dimensional map is a task failure, the target task may include an obstacle avoidance task, a navigation task, or a path planning task.

[0100] Therefore, when any of the processing results of the robot's obstacle avoidance task, path planning task, and navigation task based on the two-dimensional map fails, it is determined that the lifting acquisition trigger condition is met. At the same time, the robot monitors the lifting operation in real time. When the lifting operation is detected, the robot is controlled to enter the lifting acquisition mode. The lifting operation can be determined based on the robot's posture.

[0101] This method can actively trigger the lifting acquisition mode when the robot fails to perform a task, and with the help of external force, obtain a larger viewing angle, more accurate and more detailed three-dimensional reconstruction result in the world coordinate system, thereby providing an accurate high-definition map for the robot to perform tasks and improving the success rate of the robot's tasks.

[0102] In another embodiment, the three-dimensional map construction method further includes: obtaining a lifting acquisition mode start instruction; in response to the lifting acquisition mode start instruction, monitoring the lifting operation; when the lifting operation is monitored, controlling the robot to enter the lifting acquisition mode.

[0103] Specifically, when the layout of the working range changes, or the user increases the placement of items of interest in the working range, or the user believes that the robot's working quality is not high, the lifting and collection mode start command can be actively triggered.

[0104] Specifically, the user actively sends a lift collection mode start instruction to the robot through the application program according to the needs. After receiving the instruction, the robot monitors the lifting operation in real time. When the lifting operation is detected, the robot is controlled to enter the lift collection mode. In one embodiment, in response to the lift collection mode start instruction, the lifting operation is monitored; when the lifting operation is detected, the robot is controlled to enter the lift collection mode. Lifting in one embodiment means picking up, and picking up refers to picking up the robot under the action of an external force so that the robot is lifted and placed in the lift collection mode. For example, the lift collection instruction is manually started, and the user actively triggers the lift collection based on the needs.

[0105] When the user uses the lifting and collecting function for the first time, he or she may not know how to perform the lifting and collecting function. Therefore, when the lifting and collecting function is detected, a voice broadcast or an APP output of the lifting and collecting function prompt can be provided. The lifting and collecting function prompt can include the moving speed, moving height, moving direction and other precautions, so as to provide users with information to improve processing efficiency.

[0106] A three-dimensional map construction method, such as Figure 6 As shown, it includes two stages:

[0107] Phase 1: Three-dimensional map construction phase.

[0108] In the three-dimensional map construction stage, when the robot is in the lifting collection mode, the robot's sensors are controlled to collect data; in the lifting collection mode, the robot is in a lifted state; three-dimensional reconstruction is performed based on the collected data of the sensors to obtain a three-dimensional map.

[0109] Specifically, the robot's sensors include an RGB camera, a depth sensor, and an inertial navigation module.

[0110] The RGB camera is used to collect RGB images, the depth sensor is used to collect depth images, and the inertial navigation module is used to collect posture data.

[0111] The construction of the three-dimensional map specifically includes: aligning the RGB image and the depth image to obtain a three-dimensional point cloud, obtaining the position and pose of each frame of the three-dimensional point cloud, and performing three-dimensional reconstruction based on the position and pose and the three-dimensional point cloud to obtain a three-dimensional map.

[0112] The robot's posture data can be obtained based on digital images and depth images, based on an inertial navigation module, or by fusing the calculated posture with the measured posture of the inertial navigation module.

[0113] One way to enter the lifting collection mode is to output a lifting reminder when at least one of the robot's recognition results and task processing results meets the triggering conditions of the lifting collection mode; when a lifting operation is detected, the robot is controlled to enter the lifting collection mode.

[0114] Another way to enter the lifting and collection mode is to obtain a lifting and collection mode start instruction; in response to the lifting and collection mode start instruction, monitor the lifting operation; when the lifting operation is monitored, control the robot to enter the lifting and collection mode.

[0115] In order to improve the data collection quality in the lift collection mode, the moving speed in the lift collection mode is obtained; when the moving speed does not meet the collection requirements, a moving speed prompt is output to prompt the user that the moving speed is within the collection requirements, thereby ensuring the collection quality and efficiency.

[0116] In order to facilitate the user to understand the collection requirements, the three-dimensional map constructed at the current moment can be displayed in real time in the lifting collection mode. The user can actively adjust the position of the robot based on the feedback of the three-dimensional map constructed at the current moment to better collect environmental data.

[0117] The second stage: three-dimensional map display processing.

[0118] Specifically, the area where the identified object of interest is located can be marked in the three-dimensional map icon. By marking the area where the object of interest is located, the object of interest can be highlighted in the three-dimensional map to play a role of prompting or warning.

[0119] Specifically, identification is performed based on the collected data to obtain an object identification result; an object of interest in the object identification result is obtained; and according to the category of the object of interest, an area where the object of interest is located is identified in the three-dimensional map in different identification methods.

[0120] In one embodiment, the user's confirmation of the area where the object of interest is located can be obtained by interacting with the user. The interactive method can improve the accuracy of identifying the area where the object of interest is located. Specifically, determine the target area where the object of interest is located; output a confirmation prompt for the target area, the confirmation prompt includes the location of the target area and the corresponding category of the object of interest; determine the target identification method of the target area based on the confirmation result of the target area and the category of the object of interest; the identification methods of the target areas where objects of interest of different categories are located are different; and identify the target area in the three-dimensional map based on the target identification method.

[0121] In one embodiment, a restricted area is determined based on the area where the object of interest is located; and the restricted area is marked in the three-dimensional map according to the restricted area. In this embodiment, by determining the restricted area based on the area where the object of interest is located, the robot can be prevented from touching valuables, dangerous items, or fragile items, thereby ensuring the safety of the robot and the items.

[0122] The three-dimensional map construction method of the present application aims to solve the problems of limited viewing angle and height range, low recognition confidence, and incomplete map information of the robot in underground collection mode. By lifting the robot's viewing angle, the perception range of the robot's sensor increases under the raised viewing angle, enriching the collected environmental data, and thus using the data collected by the robot's sensor under a higher viewing angle for three-dimensional reconstruction to obtain a high-quality three-dimensional map. Furthermore, on the high-quality three-dimensional map, the area where the object of interest is located and the setting of the restricted area are marked, which can facilitate users to understand the map information in an interactive way, and facilitate the robot to subsequently adopt matching processing methods for different areas, thereby improving the quality of robot operations.

[0123] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0124] Based on the same inventive concept, the embodiment of the present application also provides a 3D map construction device for implementing the 3D map construction method involved above. The implementation solutions provided by each device to solve the problem are similar to the implementation solutions recorded in the above method, so the specific limitations in the embodiments of each device provided below can refer to the limitations of the method above, and will not be repeated here.

[0125] In one embodiment, a three-dimensional map construction device, such as Figure 7 As shown, including:

[0126] The collection module 702 is used to control the robot's sensors to collect data when the robot is in a lifting collection mode; in the lifting collection mode, the robot is in a lifted state.

[0127] The construction module 704 is used to perform three-dimensional reconstruction based on the collected data of the sensor to obtain a three-dimensional map.

[0128] The above-mentioned three-dimensional map construction device controls the robot's sensors to collect data when the robot is in the lifting collection mode. In the lifting collection mode, the robot is in a lifted state, so that the robot's viewing angle is raised. At the lifted perspective, the perception range of the robot's sensors is increased, enriching the collected environmental data, thereby using the data collected by the robot's sensors at a higher perspective to perform three-dimensional reconstruction, and obtain a high-quality three-dimensional map.

[0129] In another embodiment, the three-dimensional map construction device further includes:

[0130] The recognition module is used to perform recognition based on the collected data to obtain an object recognition result.

[0131] A focus module is used to obtain the focus object in the object recognition result.

[0132] The identification module is used to identify the area where the identified object of interest is located in the three-dimensional map icon.

[0133] In another embodiment, the identification module is used to identify the area where the object of interest is located in the three-dimensional map in different identification ways according to the category of the object of interest.

[0134] In another embodiment, an identification processing module is used to determine a target area where the object of interest is located; output a confirmation prompt for the target area, the confirmation prompt including the location of the target area and the corresponding category of the object of interest; determine a target identification method for the target area based on the confirmation result of the target area and the category of the object of interest; the identification methods of the target areas where objects of interest of different categories are located are different; and identify the target area in the three-dimensional map according to the target identification method.

[0135] In another embodiment, the three-dimensional construction device further comprises:

[0136] A restricted area marking module is used to determine a restricted area based on the area where the object of interest is located; and mark the restricted area in the three-dimensional map according to the restricted area.

[0137] In another embodiment, the restricted area identification module is used to determine the restricted area according to the area where the object of interest is located and the safety distance, and / or, in response to the restricted area setting operation, determine the area where the object of interest is located indicated by the restricted area setting operation as the restricted area.

[0138] In another embodiment, the three-dimensional construction device further comprises:

[0139] The map display module is used to display the three-dimensional map constructed at the current moment in real time in the lift acquisition mode.

[0140] In another embodiment, the three-dimensional construction device further comprises:

[0141] The prompt module is used to obtain the moving speed in the lifting acquisition mode; when the moving speed does not meet the acquisition requirements, output a moving speed prompt.

[0142] In another embodiment, the three-dimensional construction device further comprises:

[0143] The trigger module is used to output a lifting reminder when at least one of the robot's recognition results and task processing results meets the trigger condition of the lifting collection mode; when a lifting operation is detected, the robot is controlled to enter the lifting collection mode.

[0144] In another embodiment, the trigger module is further used to obtain a lifting and collection mode start instruction; in response to the lifting and collection mode start instruction, monitor the lifting operation; when the lifting operation is monitored, control the robot to enter the lifting and collection mode.

[0145] Each module in the above three-dimensional map construction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0146] In one embodiment, a robot is provided, whose internal structure diagram can be shown as follows: Figure 8 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a three-dimensional map construction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell.

[0147] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] In one embodiment, a robot is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the methods of the above embodiments when executing the computer program.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods of the above embodiments are implemented.

[0150] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the methods of the above embodiments when executed by a processor.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional map construction method, It is characterized in that The method comprises: When the robot is in a lifting collection mode, the sensor of the robot is controlled to collect data; in the lifting collection mode, the robot is in a lifting state; Three-dimensional reconstruction is performed based on the collected data of the sensor to obtain a three-dimensional map.

2. The method according to claim 1, It is characterized in that The method further comprises: Performing identification based on the collected data to obtain an object recognition result; Obtaining an object of interest in the object recognition result; The area where the identified object of interest is located is marked in the three-dimensional map icon.

3. The method according to claim 2, It is characterized in that The marking in the three-dimensional map icon of the area where the identified object of interest is located includes: marking the area where the object of interest is located in the three-dimensional map in different marking ways according to the category of the object of interest.

4. The method according to claim 3, It is characterized in that The marking of the area where the object of interest is located in the three-dimensional map in different marking ways according to the category of the object of interest includes: Determine a target area where the object of interest is located; Outputting a confirmation prompt for the target area, the confirmation prompt including the location of the target area and the category of the corresponding object of interest; Determine a target identification method of the target area according to the confirmation result of the target area and the category of the object of interest; the identification methods of the target area where the objects of interest of different categories are located are different; The target area is identified in the three-dimensional map according to the target identification method.

5. The method according to any one of claims 2 to 4, It is characterized in that The method further comprises: Determine a restricted area based on the area where the object of interest is located; A restricted traffic zone is marked in the three-dimensional map according to the restricted traffic zone area.

6. The method according to claim 5, It is characterized in that The determining of the restricted area based on the area where the object of interest is located includes at least one of the following methods: The first method is to determine the restricted area based on the area where the object of interest is located and the safety distance; The second method is to respond to a restricted zone setting operation and determine the area where the object of interest indicated by the restricted zone setting operation is located as the restricted zone area.

7. The method according to claim 1, It is characterized in that The method further comprises: Acquire the moving speed in the lifting acquisition mode; When the moving speed does not meet the collection requirements, a moving speed prompt is output.

8. The method according to claim 1, It is characterized in that The method further comprises: When at least one of the robot's recognition result and task processing result meets the trigger condition of the lift collection mode, a lift reminder is output; When a lifting operation is detected, the robot is controlled to enter the lifting collection mode.

9. The method according to claim 1, It is characterized in that The method further comprises: Get the command to start the lift acquisition mode; In response to the lift acquisition mode start instruction, monitoring the lift operation; When a lifting operation is detected, the robot is controlled to enter the lifting collection mode.

10. A robot, include: A controller, a cleaning member, a driver, and a sensor; the driver and the sensor are both connected to the controller; the controller comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 9 when executing the computer program.

Images