Robot mapping method and related device, robot and storage medium

Abstract

The invention discloses a robot mapping method, a related device, a robot and a storage medium, and the robot mapping method comprises the steps: obtaining a shot image and radar data of the robot at a current point location; wherein the radar data comprises a plurality of measuring point data; aligning the shot image with the radar data, and identifying an image area of each object in the shot image; determining a target grid occupied by each object in the grid map based on measuring point data aligned with the image area of each object in the radar data; and in response to the existence of the shielding object in each object, determining a to-be-explored area shielded by the shielding object at least based on the current point location and a target grid occupied by the shielding object in the grid map, and determining an exploration point location to which the robot is going based on the to-be-explored area. According to the scheme, the robot mapping precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robotic mapping method and related devices, a robot, and a storage medium. Background Art

[0002] With the rapid development of electronic technology, robots have been increasingly used in daily life, industrial production and other aspects. For example, sweeping robots, companion robots, etc.

[0003] Before a robot can start working, it usually needs to build a map in order to execute formal instructions such as cleaning the floor. However, existing robots usually have large mapping errors. In view of this, how to improve the accuracy of robot mapping has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a robot mapping method and related devices, robots and storage media, which can improve the robot mapping accuracy.

[0005] In order to solve the above-mentioned technical problems, the first aspect of the present application provides a robot mapping method, including: obtaining a captured image and radar data of the robot at a current point; wherein the radar data includes a number of measurement point data; aligning the captured image with the radar data, and identifying the image area of ​​each object in the captured image; based on the measurement point data in the radar data that are respectively aligned with the image area of ​​each object, determining the target grids occupied by each object in the grid map; in response to the presence of obstructions in each object, determining the area to be explored blocked by the obstruction based on at least the current point and the target grid occupied by the obstruction in the grid map, and determining the exploration point to be reached by the robot based on the area to be explored.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a robot mapping device, including: a data acquisition module, an alignment and recognition module, a grid determination module and a point determination module, the data acquisition module is used to acquire the captured image and radar data of the robot at the current point; wherein the radar data includes a plurality of measuring point data; the alignment and recognition module is used to align the captured image with the radar data, and identify the image area of ​​each object in the captured image; the grid determination module is used to determine the target grids occupied by each object in the grid map based on the measuring point data in the radar data that are respectively aligned with the image area of ​​each object; the point determination module is used to determine the area to be explored blocked by the occlusion in response to the existence of an occlusion in each object, at least based on the current point and the target grid occupied by the occlusion in the grid map, and determine the exploration point to be reached by the robot based on the area to be explored.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the robot mapping method in the above first aspect.

[0008] In order to solve the above-mentioned technical problems, the fourth aspect of the present application provides a robot, which at least includes a camera device, a radar device, a motion device and a control device, and the camera device, the radar device and the motion device are electrically connected to the control device respectively, and the control device is the electronic device in the above-mentioned third aspect.

[0009] In order to solve the above technical problems, the fifth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the robot mapping method of the first aspect.

[0010] The above scheme obtains the captured image and radar data of the robot at the current point, and the radar data includes several measuring point data, and then aligns the captured image and the radar data, and identifies the image area of ​​each object in the captured image, so as to determine the target grid occupied by each object in the grid map based on the measuring point data in the radar data that are respectively aligned with the image area of ​​each object, and then in response to the existence of occluders in each object, at least based on the current point and the target grid occupied by the occluder in the grid map, determine the area to be explored blocked by the occluder, and based on the area to be explored, determine the exploration point to be reached by the robot, so on the one hand, it can combine the image and radar data for mapping, which helps to improve the mapping accuracy to a certain extent, and on the other hand, it also detects whether there are occluders in each detected object during the mapping process, and when there are occluders, further determines the area to be explored blocked by the occluder in combination with the current detection results, so as to determine the exploration point to be reached by the robot, which helps to supplement the mapping of the blocked area and can avoid the lack of mapping as much as possible. Therefore, the mapping accuracy of the robot can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of an embodiment of the robot mapping method of the present application; Figure 2a is a schematic diagram of an embodiment of determining an area to be explored; Figure 2b is a schematic diagram of an embodiment of determining exploration points when the category of the obstruction is a furniture object; Figure 3 It is a schematic diagram of the framework of an embodiment of the robot mapping device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5It is a schematic diagram of the framework of an embodiment of the robot of the present application; Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0013] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0014] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0015] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the robot mapping method of the present application. Specifically, it may include the following steps: Step S11: Obtain the captured image and radar data of the robot at the current position.

[0016] In the disclosed embodiment, radar data includes several measuring point data. It should be noted that the radar ray is reflected on the surface of the object and received by the radar, and the measuring point data can be formed. The measuring point data contains the relative angle, relative distance and other position information between the reflection point on the surface of the object and the radar, and the spatial position of the reflection point on the surface of the object can be located accordingly. Of course, the above description is only a principle description of the measuring point data. The specific process of obtaining the measuring point data can refer to the radar technical details, which will not be repeated here. In addition, as a possible example, during the mapping process, the robot can use a preset strategy to determine the measurement point for mapping. When the robot performs environmental detection at a certain measurement point, the measurement point is used as the current point. For example, a measurement point can be determined every time a certain distance is traveled, or it can also try to move close to the edge of the space to be mapped, and then determine a measurement point after traveling a certain distance along the edge each time. Of course, the above examples are only several possible examples of preset strategies for robots to determine measurement points, and other possible ways of preset strategies are not limited thereto. The specific ways of preset strategies are not given examples one by one here.

[0017] Step S12: Align the captured image with the radar data, and identify the image area of ​​each object in the captured image.

[0018] In an implementation scenario, the captured image and radar data may be dedistorted before alignment and recognition. It should be noted that the dedistortion of the captured image can be achieved in combination with parameters such as camera distortion coefficients, and the technical details of camera distortion correction can be referred to for details. The dedistortion of radar data can be achieved in combination with parameters such as posture information provided by the laser radar odometer, and the technical details of radar distortion correction can be referred to for details, which will not be repeated here.

[0019] In one implementation scenario, the camera and radar on the robot can be calibrated in advance to obtain external parameters between the camera and the radar (such as coordinate system conversion parameters, etc.), and obtain the internal parameters of the camera and the radar. On this basis, after obtaining the captured image and radar data, the captured image and the radar data can be aligned in combination with the internal parameters and the external parameters, and then the correspondence between the pixel points in the captured image and the measured point data can be established, that is, a certain pixel point in the captured image and the measured point data are both sensed from a certain three-dimensional point in the physical space, and the above-mentioned correspondence exists between the two. It should be noted that the specific process of alignment between the captured image and the radar data can refer to the technical details of data alignment, which will not be repeated here.

[0020] In one implementation scenario, in order to identify the image area of ​​each object in the captured image, a target detection algorithm such as FasterRCNN can be used to perform target detection on the captured image to obtain the image area (such as a rectangular box bbox) of each object in the captured image. It should be noted that each object may include but is not limited to: furniture objects, building components and other categories, which are not limited here. In addition, during the target detection process, not only the image area of ​​each object can be output, but also the category to which each object belongs, such as furniture objects such as beds and tables, or building components such as walls and partitions.

[0021] In another implementation scenario, different from the aforementioned implementation, in order to identify the image area of ​​each object in the captured image, an image segmentation algorithm such as U-Net can be used to segment the captured image to obtain the image area (such as contour mask) of each object in the captured image. For the specific categories to which each object belongs, please refer to the aforementioned related description, which will not be repeated here. In addition, during the image segmentation process, not only the image area of ​​each object can be output, but also the category to which each object belongs, such as furniture objects such as beds and tables, or building components such as walls and partitions.

[0022] In another implementation scenario, different from the aforementioned implementation, in order to identify the image area of ​​each object in the captured image, the captured image can also be subjected to target detection and image segmentation, based on which the detection result of the target detection and the segmentation result of the image segmentation can be combined to obtain the image area of ​​each object. Exemplarily, if the detection result of the target detection does not detect the image area of ​​any object at a certain position, but the segmentation result of the image segmentation segments the image area of ​​a certain object at the same position, it can be determined that the image area of ​​a certain object is recognized at the position. In addition, as mentioned above, in the process of target detection and image segmentation, the categories to which each object belongs can be output respectively, and the object category obtained by the target detection at the same position and the image category obtained by the image segmentation can be combined to determine the final category of the object at the position. Exemplarily, if the category of the object obtained by target detection at a certain position is furniture, and the category of the object obtained by image segmentation is unknown, it can be determined that the final category of the object at the position is furniture. Of course, the above examples are only a few possible examples of how to combine the segmentation results of image segmentation and the detection results of target detection, and do not limit other possible combinations of the segmentation results of image segmentation and the detection results of target detection, which will not be given one by one here.

[0023] Step S13: based on the measurement point data in the radar data respectively aligned with the image areas of the respective objects, determining the target grids respectively occupied by the respective objects in the grid map.

[0024] Specifically, the measuring point data in the radar data that are aligned with the image area of ​​each object can be obtained as the target data, and the target data can include at least one of the relative angle and relative distance with the current point. Exemplarily, taking the image area of ​​a certain object containing N pixels as an example, the measuring point data corresponding to the N pixels can be obtained according to the corresponding relationship established after the aforementioned data alignment, and then all these measuring point data can be used as the target data of the object. On this basis, the target grids occupied by each object in the grid map can be determined based on the target data and the current point corresponding to the target position in the grid map. Exemplarily, taking the target data containing relative angles and relative distances as an example, for any target data of a certain object, the target position in the grid map can be taken as the reference point, the straight line passing through the reference point and in the same direction as the robot at the current point can be taken as the reference line, the straight line passing through the reference point and deviating from the reference line at the relative angle in the target data can be taken as the deviation line, and the reference point on the deviation line is taken as the starting point to deviate from the relative distance in the target data in the detection direction of the robot at the current point, then the grid where the position is located is the target grid occupied by the object in the grid map. In the above method, the measurement point data respectively aligned with the image area of ​​each object in the radar data are obtained as the target data, and the target data includes at least one of the relative angle and relative distance with the current point position, and then based on the target data and the current point position corresponding to the target position in the grid map, the target grids occupied by each object in the grid map are determined, which can improve the accuracy of determining the target grids occupied by each object in the grid map.

[0025] Step S14: In response to the existence of obstructions in each object, determine the area to be explored blocked by the obstruction based at least on the current point and the target grid occupied by the obstruction in the grid map, and determine the exploration point to be reached by the robot based on the area to be explored.

[0026] In an implementation scenario, for any object, it can be determined whether the object is an obstruction in combination with the image area of ​​the object in the captured image. Exemplarily, it can be determined whether it is an obstruction based on factors such as the area occupied by the image area in the captured image, the proportion of the length of the image area in the captured image, and the proportion of the width of the image area in the captured image. For example, through the aforementioned image recognition, it can be identified that the image area of ​​the furniture object "bed" in the captured image has a length that occupies 90% of the length of the captured image, or an area that occupies 85% of the captured image, or a width that occupies 80% of the width of the captured image, then it can be determined that the furniture object "bed" in the captured image is an obstruction. It should be noted that the above example is only a possible example in the actual application process, and does not limit the discrimination threshold of factors such as area, length, width, etc. when determining whether it is an obstruction, nor does it limit the specific type of obstruction (such as it may be a furniture object or a building component). Other possible situations are not given examples one by one here.

[0027] In another implementation scenario, for any object, radar data can be combined to determine whether the object is an obstruction. Exemplarily, the measurement point data can also include arrival time (characterizing the time difference from the transmitted signal to the received signal), and whether it is an obstruction can be determined based on the arrival time in each measurement point data. For example, if the arrival time in the measurement point data of a large proportion of rays starting from a certain ray suddenly decreases, and these measurement point data all belong to the same object, then the object can be considered to be an obstruction.

[0028] In another implementation scenario, for any object, the captured image and radar data can be combined to jointly determine whether the object is an occluder, so that the determination of whether the object is an occluder based on the captured image and the determination of whether the object is an occluder based on the radar data complement each other. It should be noted that the specific method of complementing each other can refer to the specific method of combining target detection and image segmentation to determine the image area, which will not be repeated here.

[0029] In an implementation scenario, if there are no obstructions in each object, the next point can be moved to continue to execute the aforementioned step S11 to continue to build and improve the grid map, and so on until the map is built. It should be noted that the method for determining the next point can refer to the aforementioned description of the preset strategy, which will not be repeated here.

[0030] In an implementation scenario, when there are occluders in each object, the area to be explored that is blocked by the occluders can be determined first. Figure 2a , Figure 2a is a schematic diagram of an embodiment of determining an area to be explored. Figure 2aAs shown, specifically, the target position (such as A1) corresponding to the current point in the grid map can be obtained first, and the connected domain formed by the target grid (such as the grid filled with oblique shadows) occupied by the occluder in the grid map can be obtained as the target area, and then the target position is used as the starting point to connect the ends of the target area to form a target ray (such as ray A1C and ray A1F), and finally, the area to be explored (such as trapezoid BCDE) can be determined based on at least the target ray, the first edge line of the target area (such as BE) and the second edge line of the grid map (such as CD and DE). Of course, Figure 2a The area to be explored shown is only a possible example in the actual application process, and does not limit the specific shape and specific position of the area to be explored, and no examples are given here one by one. In the above method, the current point position is obtained to correspond to the target position in the grid map, and the connected domain formed by the target grid occupied by the occluder in the grid map is obtained as the target area, and then the end of the target area is connected with the target position as the starting point to form a target ray, so as to determine the area to be explored based on at least the target ray, the first edge line of the target area and the area enclosed by the second edge line of the grid map. The area to be explored can be determined only by point-line operations, which helps to reduce the complexity of determining the area to be explored as much as possible while ensuring the accuracy of the area to be explored as much as possible.

[0031] In a specific implementation scenario, when the robot is building a map, the real-time position of the robot in the grid map can be determined based on parameters such as the robot's posture information. For specific details, please refer to technical details such as SLAM (Simultaneous Localization And Mapping), which will not be repeated here.

[0032] In a specific implementation scenario, as a possible example, the area enclosed by the target ray, the first edge line, and the second edge line can be directly used as the area to be explored; or, as another possible example, the area enclosed by the target ray, the first edge line, and the second edge line can be first obtained as a candidate area, and then the area to be explored of the robot is obtained based on the connected domain formed by the unoccupied grids in the candidate area, that is, the connected domain formed by the unoccupied grids can be specifically selected in the candidate area as the area to be explored. It should be noted that if the grid is not occupied, it means that the grid position is highly likely not to have been explored, or although it has been explored, no actual connection with the object has been established. In the above method, after obtaining the area enclosed by the target ray, the first edge line, and the second edge line as the candidate area, the connected domain formed by the unoccupied grids in the candidate area is used as the area to be explored, which helps to further improve the accuracy of the area to be explored.

[0033] In an implementation scenario, please continue to refer to Figure 2aAfter determining the area to be explored, a third edge line (such as BC) other than the first edge line and the second edge line in the area to be explored can be obtained, and the exploration point (such as A2) can be determined on the side of the third edge line away from the area to be explored. In other words, the third edge line does not overlap with any of the map edge of the grid map and the object edge of the obstruction. In the above manner, the exploration position is determined on the side of the third edge line other than the first edge line and the second edge line away from the detection area, which helps the robot to cover the area to be explored as comprehensively as possible at the exploration point after reaching the exploration point, thereby being able to improve the area to be explored in the grid map as efficiently as possible.

[0034] In a specific implementation scenario, as a possible implementation method, please continue to refer to Figure 2a Specifically, a point that is a preset distance from the third edge line and outside the area to be explored can be selected on the perpendicular bisector GA2 of the third edge line (such as BC) as an exploration point (such as A2). It should be noted that the preset distance can be determined according to the camera parameters of the robot (such as the field of view, etc.). Taking the determination according to the field of view as an example, the smaller the field of view, the larger the preset distance can be set so that the robot's field of view covers the area to be explored as comprehensively as possible. Conversely, the larger the field of view, in order to improve the imaging quality of the area to be explored, the preset distance can be set slightly smaller so that the robot's field of view covers the area to be explored as comprehensively as possible, and the area to be explored can be imaged at close range as much as possible to improve the imaging quality.

[0035] In a specific implementation scenario, as another possible implementation method, after determining the aforementioned perpendicular bisector, the shortest path from the robot's target position to the perpendicular bisector can be determined (for example, when there is no obstruction, the shortest path is the vertical line from the target position to the perpendicular bisector). The robot can move along the shortest path to the perpendicular bisector and determine whether the robot's field of view at this position can cover the area to be explored. If so, this position can be directly used as an exploration point. Otherwise, the robot can continue to move along the perpendicular bisector away from the area to be explored, and continue to determine whether the robot's field of view can cover the area to be explored, and then determine that the current position is an exploration point.

[0036] In one implementation scenario, as a possible implementation method, after determining the exploration point, the robot can move to the exploration point and use the exploration point as the new current point. In response to the category of the obstruction being a building component (such as a wall, etc.), the robot returns to the step of obtaining the captured image and radar data of the robot at the current point, until the step of determining the target grid occupied by each object in the grid map based on the measurement point data that is aligned with the image area of ​​each object in the radar data. That is to say, in the case where the category of the obstruction is a building component such as a wall, the robot can use the captured image and radar data at the exploration point to complete the area to be explored in the grid map according to the above-mentioned related description, and then continue to move to the next point according to the preset strategy. In the above manner, in the case where the category of the obstruction is a building component, the aforementioned mapping process is executed again according to the captured image and radar data at the exploration point to complete the area to be explored in the grid map, which can improve the integrity of the robot's mapping.

[0037] In another implementation scenario, as another possible implementation, after determining the exploration point, the robot can move to the exploration point and use the exploration point as the new current point. Different from the above implementation, in response to the category of the occluding object being a furniture object, the robot returns to the step of obtaining the robot's captured image and radar data at the current point and iterates until all perspectives of the furniture object to which the occluding object belongs are detected by the robot at all the exploration points that have been traveled. Please refer to Figure 2a and Figure 2b , Figure 2b FIG. 1 is a schematic diagram of an embodiment of determining exploration points when the category of the obstruction is furniture. Figure 2a For example, if the occluder is a furniture object, the robot moves to the exploration point A2 and takes the exploration point A2 as the new current point. In response to the category of the occluder being a furniture object, the robot can return to the aforementioned step S11 and iterate, that is, the grid map can be improved (such as Figure 2b After that, a new area to be explored is determined according to the above method (such as Figure 2b and determine a new exploration point according to the above method (such as Figure 2b A3 in the figure), and then move to the exploration point A3, and use the exploration point A3 as the new current point, and in response to the category of the occluding object being furniture, return to the aforementioned step S11 and iterate, that is, the grid map can be improved again (such as Figure 2b A grid filled with grid shadows, and its corresponding object belongs to the category of furniture objects). At this time, since the furniture objects ( Figure 2b ) shown as shaded grids in the figure Figure 2bThe lower middle view, right view and upper view (the left view is excluded because it is blocked) are all detected by the robot at each exploration point it has reached (such as Figure 2b The exploration points A1, A2, and A3 are explored, so the iteration can be terminated. For example, at this time, the next point can be moved to according to the above-mentioned preset strategy. It should be noted that Figure 2b For the specific meanings of various styles of lines, please refer to Figure 2a For example, the unbold dashed line starting from the exploration point represents the target ray, the bold dashed line represents the first edge line, the bold solid line represents the second edge line, and the black dot represents the exploration point. Figure 2b The above-mentioned method is only a possible example of multi-view mapping when the category of the obstruction is furniture. Other possible situations will not be given one by one here. The above-mentioned method is required. When the category of the obstruction is furniture, exploration points are continuously selected around the furniture object to iteratively execute the above-mentioned mapping process to improve the furniture objects in the to-be-explored area in the grid map from multiple perspectives. On the one hand, the accuracy and completeness of the robot's mapping of furniture objects can be improved, and on the other hand, the mapping time is not significantly increased. Therefore, the accuracy, completeness and efficiency of the robot's mapping can be improved at the same time.

[0038] In an implementation scenario, after the map is built in the above manner, the work instruction can also be executed based on the grid map in response to receiving the work instruction. For example, the robot can be a cleaning robot, and the work instruction can be a cleaning instruction, so it can accurately clean around and under furniture objects such as tables and beds based on the grid map; or, the robot can also be a companion robot, and the work instruction can be a companion instruction, so it can accurately accompany the target person based on the grid map; or, the robot can also be a pet robot, and the work instruction can be a patrol instruction, so it can accurately patrol the desired indoor location (such as a balcony, etc.) based on the grid map. Of course, the above examples are only several possible examples of the robot executing work instructions in actual applications, and other possible situations are not given one by one here. In the above method, after the grid map is built, in response to receiving the work instruction, the robot can execute the work instruction based on the accurate and complete grid map built above, which helps to improve the work efficiency of the robot.

[0039] The above scheme obtains the captured image and radar data of the robot at the current point, and the radar data includes several measuring point data, and then aligns the captured image and the radar data, and identifies the image area of ​​each object in the captured image, so as to determine the target grid occupied by each object in the grid map based on the measuring point data in the radar data that are respectively aligned with the image area of ​​each object, and then in response to the existence of occluders in each object, at least based on the current point and the target grid occupied by the occluder in the grid map, determine the area to be explored blocked by the occluder, and based on the area to be explored, determine the exploration point to be reached by the robot, so on the one hand, it can combine the image and radar data for mapping, which helps to improve the mapping accuracy to a certain extent, and on the other hand, it also detects whether there are occluders in each detected object during the mapping process, and when there are occluders, further determines the area to be explored blocked by the occluder in combination with the current detection results, so as to determine the exploration point to be reached by the robot, which helps to supplement the mapping of the blocked area and can avoid the lack of mapping as much as possible. Therefore, the mapping accuracy of the robot can be improved.

[0040] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of the robot mapping device of the present application. The robot mapping device 30 includes: a data acquisition module 31, an alignment recognition module 32, and grid determination modules 33 and 34. The data acquisition module 31 is used to acquire the captured image and radar data of the robot at the current point; wherein the radar data includes a number of measurement point data; the alignment recognition module 32 is used to align the captured image with the radar data and identify the image area of ​​each object in the captured image; the grid determination module 33 is used to determine the target grids occupied by each object in the grid map based on the measurement point data in the radar data that are respectively aligned with the image area of ​​each object; the point determination module 34 is used to determine the area to be explored blocked by the occlusion based on at least the current point and the target grid occupied by the occlusion in the grid map in response to the existence of occlusions in each object, and determine the exploration point to be reached by the robot based on the area to be explored.

[0041] In the above scheme, the robot mapping device 30 obtains the captured image and radar data of the robot at the current point, and the radar data includes a number of measuring point data, and then aligns the captured image with the radar data, and identifies the image area of ​​each object in the captured image, thereby determining the target grids occupied by each object in the grid map based on the measuring point data in the radar data that are respectively aligned with the image area of ​​each object, and then responding to the existence of occluders in each object, at least based on the current point and the target grid occupied by the occluders in the grid map, determine the area to be explored that is blocked by the occluders, and based on the area to be explored, determine the exploration point to be reached by the robot, so on the one hand, it can combine the image and radar data for mapping, which helps to improve the mapping accuracy to a certain extent, and on the other hand, in the mapping process, it also detects whether there are occluders in each detected object, and when there are occluders, further determines the area to be explored that is blocked by the occluders in combination with the current detection results, so as to determine the exploration point to be reached by the robot, which helps to supplement the mapping of the blocked area and can avoid the lack of mapping as much as possible. Therefore, the mapping accuracy of the robot can be improved.

[0042] In some disclosed embodiments, the point determination module 34 includes an acquisition submodule for acquiring the target position in the grid map corresponding to the current point, and acquiring the connected domain formed by the target grid occupied by the obstruction in the grid map as the target area; the point determination module 34 includes a connection submodule for connecting the ends of the target area with the target position as the starting point to form a target ray; the point determination module 34 includes a determination submodule for determining the area to be explored based on at least the area enclosed by the target ray, the first edge line of the target area and the second edge line of the grid map.

[0043] In some disclosed embodiments, the determination submodule includes a candidate area acquisition unit, which is used to acquire the area enclosed by the target ray, the first edge line and the second edge line as a candidate area; the determination submodule includes an exploration area determination unit, which is used to obtain the robot's area to be explored based on the connected domain formed by unoccupied grids in the candidate area.

[0044] In some disclosed embodiments, the point determination module 34 includes an edge sub-module for obtaining a third edge line other than the first edge line and the second edge line in the area to be explored; the point determination module 34 includes a point sub-module for determining the exploration point on the side of the third edge line away from the area to be explored.

[0045] In some disclosed embodiments, the point submodule is specifically used to select a point on the perpendicular bisector of the third edge line that is a preset distance away from the third edge line and is outside the area to be explored as an exploration point.

[0046] In some disclosed embodiments, the robot mapping device 30 also includes an exploration travel module, which is used to travel to an exploration point and use the exploration point as a new current point; the robot mapping device 30 also includes a first loop module, which is used to respond to the category of the obstruction being a furniture object, return to the step of obtaining the robot's captured image and radar data at the current point and iterate until all perspectives of the furniture object to which the obstruction belongs are detected by the robot at all the exploration points that have been traveled to.

[0047] In some disclosed embodiments, the robot mapping device 30 also includes an exploration travel module, which is used to travel to an exploration point and use the exploration point as a new current point; the robot mapping device 30 also includes a second loop module, which is used to return to the step of acquiring the captured image and radar data of the robot at the current point in response to the category of the obstruction being a building component, until the step of determining the target grids occupied by each object in the grid map based on the measurement point data in the radar data that are respectively aligned with the image areas of each object.

[0048] In some disclosed embodiments, the grid determination module 33 includes a measurement point acquisition submodule, which is used to obtain measurement point data in the radar data that are respectively aligned with the image area of ​​each object as target data; wherein the target data includes at least one of a relative angle and a relative distance with the current point position; the grid determination module 33 includes a grid determination submodule, which is used to determine the target grids occupied by each object in the grid map based on the target data and the current point position corresponding to the target position in the grid map.

[0049] In some disclosed embodiments, the robot mapping device 30 further includes an instruction execution module for executing work instructions based on the grid map in response to receiving work instructions after the grid map is constructed.

[0050] See also Figure 4 , Figure 4 4 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned robot mapping method embodiments. The robot mapping method can be specifically referred to in the aforementioned disclosed embodiments, which will not be repeated here.

[0051] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned robot mapping method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0052] In the above scheme, the electronic device 40 obtains the captured image and radar data of the robot at the current point, and the radar data includes a number of measurement point data, and then aligns the captured image with the radar data, and identifies the image area of ​​each object in the captured image, thereby determining the target grid occupied by each object in the grid map based on the measurement point data in the radar data that are respectively aligned with the image area of ​​each object, and then responding to the existence of occluders in each object, at least based on the current point and the target grid occupied by the occluders in the grid map, determine the area to be explored that is blocked by the occluders, and based on the area to be explored, determine the exploration point to be reached by the robot, so on the one hand, it can combine the image and radar data for mapping, which helps to improve the mapping accuracy to a certain extent, and on the other hand, in the mapping process, it also detects whether there are occluders in each detected object, and when there are occluders, further determines the area to be explored that is blocked by the occluders in combination with the current detection results, so as to determine the exploration point to be reached by the robot, which helps to supplement the mapping of the blocked area and can avoid the lack of mapping as much as possible. Therefore, the mapping accuracy of the robot can be improved.

[0053] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of the robot of the present application. The robot 50 includes at least a camera device 51, a radar device 52, a motion device 53 and a control device 54. The camera device 51, the radar device 52 and the motion device 53 are electrically connected to the control device 54 respectively. The control device 54 is an electronic device in the above-mentioned electronic device embodiment. For details, please refer to the aforementioned disclosed embodiment, which will not be repeated here. It should be noted that the camera device 51 is used to capture images, the radar device 52 is used to collect radar data, and the motion device 53 is used to drive the robot 50 to move as a whole. In addition, the robot 50 may specifically include but is not limited to: cleaning robots such as sweeping robots, mopping robots, and mopping robots, and may also include but is not limited to companion robots, pet robots, etc., and the specific type of the robot 50 is not limited here.

[0054] In the above scheme, the robot 50 at least includes a camera device 51, a radar device 52, a motion device 53 and a control device 54. The camera device 51, the radar device 52 and the motion device 53 are electrically connected to the control device 54 respectively. The control device 54 is the electronic device in the above electronic device embodiment. On the one hand, it can combine the image and radar data to build a map, which helps to improve the accuracy of the map to a certain extent. On the other hand, during the mapping process, it also detects whether there are obstructions in each detected object, and when there are obstructions, it further determines the area to be explored that is blocked by the obstruction in combination with the current detection results, so as to determine the exploration point to be reached by the robot, which helps to supplement the map of the blocked area and avoid map missing as much as possible. Therefore, the accuracy of the robot's mapping can be improved.

[0055] See also Figure 6 , Figure 6 60 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 60 of the present application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor, and the program instructions 61 are used to implement the steps in any of the above-mentioned robot mapping method embodiments.

[0056] In the above scheme, the computer-readable storage medium 60 obtains the captured image and radar data of the robot at the current point, and the radar data includes a number of measuring point data, and then aligns the captured image with the radar data, and identifies the image area of ​​each object in the captured image, thereby determining the target grids occupied by each object in the grid map based on the measuring point data in the radar data that are respectively aligned with the image area of ​​each object, and then responding to the existence of occluders in each object, at least based on the current point and the target grid occupied by the occluders in the grid map, determine the area to be explored that is blocked by the occluders, and based on the area to be explored, determine the exploration point to be reached by the robot, so on the one hand, it can combine the image and radar data for mapping, which helps to improve the mapping accuracy to a certain extent, and on the other hand, in the mapping process, it also detects whether there are occluders in each detected object, and when there are occluders, further determines the area to be explored that is blocked by the occluders in combination with the current detection results, so as to determine the exploration point to be reached by the robot, which helps to supplement the mapping of the blocked area and can avoid the lack of mapping as much as possible. Therefore, the mapping accuracy of the robot can be improved.

[0057] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0058] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0059] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0060] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0061] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0062] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0063] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A robot mapping method, characterized in that: include: Obtaining the captured image and radar data of the robot at the current position; wherein the radar data includes data of several measuring points; aligning the captured image with the radar data, and identifying image regions of respective objects in the captured image; Determine the target grids occupied by the respective objects in the grid map based on the measurement point data in the radar data that are respectively aligned with the image areas of the respective objects; In response to the existence of obstructions in the various objects, an area to be explored that is blocked by the obstructions is determined based at least on the current point and the target grid occupied by the obstructions in the grid map, and based on the area to be explored, an exploration point to be reached by the robot is determined.

2. The method according to claim 1, characterized in that The determining, based at least on the current point and the target grid occupied by the obstruction in the grid map, an area to be explored that is obstructed by the obstruction includes: Acquire the target position in the grid map corresponding to the current point, and acquire the connected domain formed by the target grid occupied by the occluder in the grid map as the target area; Connecting the ends of the target area with the target position as the starting point to form a target ray; The area to be explored is determined based at least on an area enclosed by the target ray, a first edge line of the target area, and a second edge line of the grid map.

3. The method according to claim 2, characterized in that The determining the area to be explored based at least on the area enclosed by the target ray, the first edge line of the target area, and the second edge line of the grid map comprises: Acquire an area enclosed by the target ray, the first edge line, and the second edge line as a candidate area; Based on the connected domain formed by the unoccupied grids in the candidate area, the area to be explored by the robot is obtained.

4. The method according to claim 1, characterized in that: The step of determining the exploration point to be reached by the robot based on the area to be explored includes: Acquire a third edge line in the to-be-explored area; wherein the third edge line does not overlap with any of the map edge of the grid map and the object edge of the occluder; The exploration point is determined on a side of the third edge line away from the to-be-explored area.

5. The method according to claim 4, characterized in that Determining the exploration point on a side of the third edge line away from the to-be-explored area includes: A point which is a preset distance from the third edge line and is outside the to-be-explored area is selected on the perpendicular midline of the third edge line as the exploration point.

6. The method according to claim 1, characterized in that After determining the exploration point to be reached by the robot based on the area to be explored, the method further includes: Travel to the exploration point and use the exploration point as a new current point; In response to the category to which the obstruction belongs being a furniture object, the step of obtaining the captured image and radar data of the robot at the current position is returned and iterated until all perspectives of the furniture object to which the obstruction belongs have been detected by the robot at all the exploration points to which it has traveled.

7. The method according to claim 1, characterized in that After determining the exploration point to be reached by the robot based on the area to be explored, the method further includes: Travel to the exploration point and use the exploration point as a new current point; In response to the category of the obstruction being a building component, the step of obtaining the captured image and radar data of the robot at the current point is returned to, until the step of determining the target grids occupied by each object in the grid map based on the measurement point data in the radar data that are respectively aligned with the image areas of each object is completed.

8. The method according to claim 1, characterized in that The determining, based on the measurement point data in the radar data respectively aligned with the image areas of the respective objects, the target grids respectively occupied by the respective objects in the grid map comprises: Acquire the measurement point data respectively aligned with the image areas of the respective objects in the radar data as target data; wherein the target data includes at least one of a relative angle and a relative distance to the current point position; Based on the target data and the current point position corresponding to the target position in the grid map, the target grids occupied by the various objects in the grid map are determined.

9. The method according to any one of claims 1 to 8, characterized in that: After the grid map is constructed, the method further includes: In response to receiving a work order, the work order is executed based on the grid map.

10. A robot mapping device, characterized in that: include: A data acquisition module is used to acquire the captured image and radar data of the robot at the current point; wherein the radar data includes data of several measuring points; an alignment and recognition module, used to align the captured image with the radar data and recognize image areas of various objects in the captured image; A grid determination module, for determining target grids occupied by each object in a grid map based on the measurement point data in the radar data that are respectively aligned with the image areas of each object; A point determination module is used to determine, in response to the existence of obstructions in the various objects, an area to be explored that is blocked by the obstructions based at least on the current point and the target grid occupied by the obstructions in the grid map, and to determine an exploration point to be reached by the robot based on the area to be explored.

11. An electronic device, characterized in that: The robot mapping method comprises at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the robot mapping method according to any one of claims 1 to 9.

12. A robot, characterized in that: The electronic device at least comprises a camera device, a radar device, a motion device and a control device, wherein the camera device, the radar device and the motion device are electrically connected to the control device respectively, and the control device is the electronic device according to claim 11.

13. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the robot mapping method according to any one of claims 1 to 9.

Images