Embodied Robot and Its Obstacle Avoidance Control Method, Device and Program Product

Abstract

This application relates to the field of robot navigation, and particularly to an embodied robot and its obstacle avoidance control method, device, and program product. The method includes: obtaining a scene image through a monocular vision sensor; determining the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image; determining the distance between the obstacle and the monocular vision sensor; and determining the two-dimensional point cloud information of the obstacle in the grid map; updating the obstacle probability change value of the grid in the grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence; updating the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and performing path planning according to the updated grid map. This method can effectively reduce the influence of noise on the obstacle detection result, improve the detection accuracy of obstacles, and improve the accuracy of task path planning.

Description

Technical Field

[0001] The present application relates to the field of robot navigation, and particularly to an embodied robot, an obstacle avoidance control method, device and program product thereof. Background Art

[0002] During the process of a robot performing tasks, in order to prevent the robot from colliding with obstacles and reduce the number of unplanned downtimes, it is necessary for the robot to detect obstacles in the moving scene to optimize the path of the robot performing tasks. Generally, the robot usually uses a lidar or a binocular camera to detect obstacles in the scene where the robot is located, and simply marks the detected obstacles on the map. The robot performs path planning based on the obstacles marked on the map to avoid obstacles in the scene.

[0003] However, due to cost and other factors, some robots are only equipped with a wired laser or a monocular vision sensor (monocular camera). When detecting obstacles based on the wired laser or the monocular camera, if there is noise, the detection result of the obstacle may be distorted, misdetecting a non-obstacle as an obstacle, or detecting an obstacle as a non-obstacle, affecting the accuracy of the task path planned by the robot and the efficiency of task execution. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide an obstacle avoidance control method, device and program product for an embodied robot to solve the problem that when using a monocular camera or a wired laser to identify obstacles in the prior art, false detection is likely to occur, affecting the accuracy of the planned task path.

[0005] The first aspect of the embodiments of the present application provides an obstacle avoidance control method for an embodied robot, where the embodied robot includes a monocular vision sensor, and the method includes:

[0006] Obtaining a scene image through the monocular vision sensor;

[0007] If there are obstacles in the scene image, determining the obstacle category, obstacle confidence and obstacle detection frame included in the scene image;

[0008] According to the position of the obstacle detection frame in the scene image, determining the distance between the obstacle and the monocular vision sensor by the ground constraint method;

[0009] According to the corresponding relationship between the preset obstacle category and the size range and the size of the obstacle detection frame, determining the preset size of the obstacle, and improving the obstacle range according to the preset size of the obstacle, so as to determine the two-dimensional point cloud information of the obstacle in the grid map;

[0010] Update the obstacle probability change value of the grid in the grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence;

[0011] Update the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and perform path planning according to the updated grid map.

[0012] Combined with the first aspect, in the first possible implementation manner of the first aspect, after obtaining the scene image through the monocular vision sensor, the method further includes:

[0013] Determine the first key point information of the obstacle in the scene image;

[0014] Match the first key point information with the second key point information of the obstacle stored in the map before the current time;

[0015] In response to the matching of the first key point information and the second key point information, increase the obstacle confidence and improve the obstacle contour information in the grid map according to the contour information of the obstacle in the scene image under the current perspective.

[0016] Combined with the first aspect, in the second possible implementation manner of the first aspect, after obtaining the scene image through the monocular vision sensor, it further includes:

[0017] If there is no obstacle in the scene image, construct a virtual point cloud at the bottom edge of the field of view in the field of view range of the current monocular vision sensor;

[0018] Taking the position of the monocular vision sensor as the starting point and each point of the virtual point cloud as the connection point, determine the ray path emitted to the field of view range;

[0019] According to the probability of each grid passed by the ray path, determine the original obstacle detection result of the grid in the grid map;

[0020] Update the obstacle probability of the grid according to the original obstacle detection result of the grid.

[0021] Combined with the second possible implementation manner of the first aspect, in the third possible implementation manner of the first aspect, updating the obstacle probability of the grid according to the original obstacle detection result of the grid includes:

[0022] When the original obstacle detection result of the grid is that there is an obstacle in the grid, reduce the obstacle probability of the grid;

[0023] When the original obstacle detection result of the grid indicates that there is no obstacle in the grid, the obstacle probability of the grid remains unchanged.

[0024] Combined with the third possible implementation manner of the first aspect, in the fourth possible implementation manner of the first aspect, reducing the obstacle probability of the grid includes:

[0025] Determine the probability reduction amount according to the original obstacle category in the grid map and the distance between the original obstacle and the monocular vision sensor;

[0026] Reduce the obstacle probability of the grid according to the probability reduction amount.

[0027] Combined with any one of the first aspect to the fifth possible implementation manner of the first aspect, in the fifth possible implementation manner of the first aspect, updating the grid map according to the change value of the obstacle probability of the grid and the two-dimensional point cloud information of the obstacle in the grid map includes:

[0028] Determine the obstacle probability of the grid according to the change value of the obstacle probability of the grid and the two-dimensional point cloud information of the obstacle in the grid map;

[0029] When the obstacle probability of the grid is greater than or equal to a predetermined probability threshold, update that there is an obstacle in the grid;

[0030] When the obstacle probability of the grid is less than the predetermined probability threshold, keep the state that there is no obstacle in the grid.

[0031] Combined with any one of the first aspect to the fifth possible implementation manner of the first aspect, in the sixth possible implementation manner of the first aspect, determining the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image includes:

[0032] Input the scene image into a pre-trained obstacle detection model, and output the obstacle category, obstacle confidence, and obstacle detection frame through the obstacle detection model.

[0033] A second aspect of the embodiments of the present application provides an obstacle avoidance control device for an embodied robot. The embodied robot includes a monocular vision sensor, and the device includes:

[0034] A scene image acquisition unit, configured to acquire a scene image through a monocular vision sensor;

[0035] An obstacle information determination unit, configured to determine the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image if there is an obstacle in the scene image;

[0036] A distance determination unit for determining the distance between the obstacle and the monocular vision sensor by means of a ground constraint method according to the position of the obstacle detection frame in the scene image;

[0037] A two-dimensional information determination unit for determining a preset size of the obstacle according to the correspondence between the preset obstacle category and the size range and the size of the obstacle detection frame, and improving the obstacle range according to the preset size of the obstacle to determine the two-dimensional point cloud information of the obstacle in the grid map;

[0038] An obstacle probability update unit for updating the obstacle probability change value of the grid in the grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence;

[0039] A grid map update unit for updating the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and performing path planning according to the updated grid map.

[0040] Combined with the second aspect, in the first possible implementation manner of the second aspect, the device further includes:

[0041] A key point information determination unit for determining first key point information of the obstacle in the scene image;

[0042] A key point information matching unit for matching the first key point information with second key point information of the obstacle stored in the map before the current time;

[0043] An obstacle contour information improvement unit for increasing the obstacle confidence and improving the obstacle contour information in the grid map according to the contour information of the obstacle in the scene image at the current perspective in response to the matching of the first key point information and the second key point information.

[0044] Combined with the second aspect, in the second possible implementation manner of the second aspect, the device further includes:

[0045] A virtual point cloud construction unit for constructing a virtual point cloud at the bottom edge of the field of view in the field of view range of the current monocular vision sensor if there is no obstacle in the scene image;

[0046] A ray determination unit for determining a ray path emitted into the field of view range starting from the position of the embodied robot and using each point of the virtual point cloud as a connection point;

[0047] A grid obstacle detection unit, configured to determine the original obstacle detection result of the grids in the grid map according to the probability of each grid through which the ray path passes;

[0048] A probability update unit, configured to update the obstacle probability of the grid according to the original obstacle detection result of the grid.

[0049] Combined with the second possible implementation manner of the second aspect, in the third possible implementation manner of the second aspect, the probability update unit includes:

[0050] A first probability update subunit, configured to reduce the obstacle probability of the grid when the original obstacle detection result of the grid is that there is an obstacle in the grid;

[0051] A second probability update subunit, configured to maintain the obstacle probability of the grid unchanged when the original obstacle detection result of the grid is that there is no obstacle in the grid.

[0052] Combined with the third possible implementation manner of the second aspect, in the fourth possible implementation manner of the second aspect, the first probability update subunit includes:

[0053] A probability decrement determination module, configured to determine a probability decrement according to the original obstacle category in the grid map and the distance between the original obstacle and the monocular vision sensor;

[0054] A probability reduction module, configured to reduce the obstacle probability of the grid according to the probability decrement.

[0055] Combined with any one of the second aspect to the fourth possible implementation manner of the second aspect, in the fifth possible implementation manner of the second aspect, the grid map update unit includes:

[0056] An obstacle probability determination subunit, configured to determine the obstacle probability of the grid according to the change value of the obstacle probability of the grid and the two-dimensional point cloud information of the obstacle in the grid map;

[0057] A first grid update subunit, configured to update that there is an obstacle in the grid when the obstacle probability of the grid is greater than or equal to a predetermined probability threshold;

[0058] A second grid update subunit, configured to keep the state that there is no obstacle in the grid when the obstacle probability of the grid is less than the predetermined probability threshold.

[0059] Combined with any one of the second aspect to the fourth possible implementation manner of the second aspect, in the sixth possible implementation manner of the second aspect, the obstacle information determination unit is configured to:

[0060] Input the scene image into a pre-trained obstacle detection model, and output the obstacle category, obstacle confidence, and obstacle detection box through the obstacle detection model.

[0061] In a third aspect of the embodiments of the present application, a embodied robot is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the embodied robot implements the method according to any one of the first aspects.

[0062] In a fourth aspect of the embodiments of the present application, a computer program product is provided. When it runs on a computer, the computer is enabled to execute the method in the above first aspect or its various implementation manners.

[0063] In a fifth aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of the first aspects are implemented.

[0064] In a sixth aspect of the embodiments of the present application, a chip is provided for implementing the methods in the various implementation manners of the first aspect. Specifically, the above chip includes: a processor for calling and running a computer program from a memory, so that a device installed with the above chip executes the method in the above first aspect or its various implementation manners.

[0065] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: In the embodiments of the present application, a scene image is collected through a monocular vision sensor, and the obstacle category, obstacle confidence, and obstacle detection box included in the scene image are determined by parsing the scene image. Based on the correspondence between the preset size range and the obstacle category, and the size of the obstacle detection box, the preset size of the obstacle is determined. Based on the preset size, the obstacle range is improved, and the two-dimensional information of the obstacle in the scene image is determined, so that more comprehensive obstacle information can be obtained, which is beneficial to more accurate task path planning; Based on the size of the obstacle detection box, the distance between the obstacle and the monocular vision sensor (which can be used to approximately represent the distance between the obstacle and the embodied robot) is determined by the ground constraint method. According to this distance, obstacle confidence, and obstacle category, the obstacle probability of the grid in the grid map corresponding to the scene image is continuously updated, so that the influence of noise on the obstacle detection result can be effectively reduced, the detection accuracy of the obstacle can be improved, and further the accuracy of the task planning path of the embodied robot can be improved, which is beneficial to improving the task execution efficiency of the embodied robot. Description of the Drawings

[0066] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0067] Figure 1 It is a schematic diagram of the implementation process of an obstacle avoidance control method for an embodied robot provided by an embodiment of the present application;

[0068] Figure 2 It is a schematic diagram of the implementation of determining the distance between an obstacle and a monocular vision sensor provided by an embodiment of the present application;

[0069] Figure 3 It is a schematic diagram of the implementation process of a method for improving the contour information of an obstacle provided by an embodiment of the present application;

[0070] Figure 4 It is a schematic diagram of the implementation process of a method for updating the probability of an obstacle provided by an embodiment of the present application;

[0071] Figure 5 It is a schematic diagram of updating the probability of an obstacle in a grid based on a ray provided by an embodiment of the present application;

[0072] Figure 6 It is a schematic diagram of an obstacle avoidance control device for an embodied robot provided by an embodiment of the present application;

[0073] Figure 7 It is a schematic diagram of an embodied robot provided by an embodiment of the present application. Specific Embodiments

[0074] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0075] To illustrate the technical solutions described in the present application, the following will be described through specific embodiments.

[0076] Embodied robots (fully known as Embodied Robots) are a type of intelligent robots with physical entities that can interact with the environment in real time. Embodied robots can achieve a closed-loop of perception, decision-making, and action through the combination of sensors, actuators, and computing units, so as to complete tasks in the real world. Embodied robots may include, but are not limited to, floor-sweeping robots (also known as cleaning robots), object-dragging robots driven by floor-sweeping machines (i.e., floor-sweeping and object-dragging integrated robots), food-delivery robots, self-driven load-carrying robots, escort robots, service robots, etc. It can be understood that the existence forms of the various robots listed above are not limited. For example, they can be wheeled robots, or humanoid robots with two legs, or multi-legged robots, etc.

[0077] During the execution of robot tasks, in order to avoid collisions with obstacles and reduce the number of unplanned downtimes, it is necessary to detect obstacles in the moving scene in real time, so as to optimize the task path. Traditional methods usually rely on lidar or binocular cameras for obstacle detection and mark the detection results on the map, and the robot plans the path accordingly to avoid obstacles.

[0078] However, due to cost considerations, some robots are only equipped with monocular vision sensors (monocular cameras) or line laser sensors. If there is noise, it may lead to distorted obstacle detection, resulting in misjudging non-obstacles as obstacles, or missing real obstacles, thus affecting the accuracy of path planning and the task execution efficiency.

[0079] To solve the above problems, the embodiments of the present application propose an obstacle avoidance control method for embodied robots, as Figure 1 shown in the schematic diagram of the implementation process of this method, which is described in detail as follows:

[0080] In S101, a scene image is obtained through a monocular vision sensor.

[0081] Among them, the monocular vision sensor usually refers to a monocular camera installed on the embodied robot, which is used to capture a two-dimensional image of the environment around the robot. It includes, such as RGB cameras, etc., which can be used to collect color images.

[0082] The scene image is the image data of the environment around the robot, and there may be obstacles in the scene image.

[0083] The monocular vision sensor can collect scene images with a predetermined resolution at a predetermined frequency. For example, it can collect scene images with a resolution of 720p at a frequency of 30 frames per second.

[0084] In S102, if there are obstacles in the scene image, then determine the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image.

[0085] Among them, the obstacle category is the classification corresponding to the obstacles in the scene. For example, the obstacle category can include categories such as shoes, pedestrians, wires, sofas, tables, toys, etc.

[0086] The obstacle confidence is used to represent the probability that the detected obstacle belongs to the obstacle, and is usually a value between 0 and 1. For example, an obstacle confidence of 0.6 means that the probability that the detected object is an obstacle is 60%.

[0087] The obstacle detection box is used to identify the position and size of the detected obstacle in the scene image, and is usually represented by a rectangular box.

[0088] The embodiments of this application can use an object detection model in deep learning, such as the YOLO architecture model, to process the image, output the obstacle category, the obstacle confidence, and the obstacle detection box. Input the scene image into the pre-trained obstacle detection model, and output the obstacle category, the obstacle confidence, and the obstacle detection box through the obstacle detection model. For example, input the image into the trained YOLOv5 model, and the model will output the category of each detected obstacle, the obstacle confidence, and the position coordinates of the obstacle detection box.

[0089] The obstacle detection model for outputting the obstacle category, the obstacle confidence, and the obstacle category can be a single model, or two or three models. For example, three obstacle detection models can be used to parse and calculate the scene image, and output the obstacle category, the obstacle confidence, and the obstacle detection box respectively.

[0090] Obstacles are usually on the same horizontal plane as the embodied robot. Therefore, the width of the obstacle detection box can usually be used to represent the information of one dimension of the area occupied by the obstacle on the horizontal plane, such as the width of the area occupied by the obstacle.

[0091] In S103, according to the position of the obstacle detection box in the scene image, the distance between the obstacle and the monocular vision sensor is determined by the ground constraint method.

[0092] Among them, the ground constraint method is to calculate the distance between the obstacle and the monocular vision sensor (e.g., monocular camera) by using the position of the obstacle in the image and the geometric relationship of the ground. The size of a single pixel can be determined according to the internal parameters of the monocular vision sensor. Through the position of the obstacle detection frame in the scene image, the first pixel point of the ground contact point of the obstacle in the scene image can be determined. According to the number of pixels between the first pixel point and the center point of the scene image, combined with the size of a single pixel, the image distance can be obtained. According to the actually determined height of the monocular vision sensor, the focal length of the monocular vision sensor, and the determined image distance, the distance between the obstacle and the monocular vision sensor can be determined through the back-projection formula.

[0093] For example Figure 2 As shown in the schematic diagram of obstacle imaging, the determined number of pixels (the number of pixels between the first pixel point of the ground contact point of the obstacle in the scene image and the center point of the scene image) is n1, the height of each pixel is h1, the focal length of the monocular vision sensor is f, the distance between the monocular vision sensor and the obstacle is d, and the height of the monocular vision sensor is H. Then the distance between the monocular vision sensor and the obstacle can be expressed as:

[0094] d = H * f / (h1 * n1).

[0095] In S104, according to the corresponding relationship between the preset obstacle category and the size range and the size of the obstacle detection frame, the preset size of the obstacle is determined, and the obstacle range is improved according to the preset size of the obstacle, so as to determine the two-dimensional point cloud information of the obstacle in the grid map.

[0096] Among them, the two-dimensional point cloud information of the obstacle in the grid map is the projection of the discrete point set of the collected obstacle in the grid map, that is, the grid area occupied by the obstacle in the grid map.

[0097] The corresponding relationship between the obstacle category and the size range can be used to estimate the range occupied by obstacles of different categories. For example, the size range of a table is 30cm * 120cm. The embodied robot can obtain the preset size (the preset contour size of the obstacle) of the obstacle by combining the width of the obstacle detection frame in the detected size of the obstacle with the determined obstacle category, obtain the estimated area of the obstacle, collect the contour information of the obstacle according to the estimated area, and then completely determine the area occupied by the obstacle. According to the corresponding relationship between the grid map and the scene map, according to the area occupied by the obstacle in the scene map, the grid occupied by the obstacle in the grid map is determined. According to the grid occupied by the obstacle in the grid map, the two-dimensional point cloud information of the obstacle in the grid map is determined.

[0098] In a possible implementation, the embodied robot can pre-detect the contour information of obstacles in the scene and the identification features of the obstacles, and the identification features can include the distribution features of key points of the obstacles, etc. After acquiring an image including the obstacles, based on the identification features of the obstacles, the current orientation of the obstacles can be determined, and the area currently occupied by the obstacles can be determined, and the two-dimensional point cloud information of the obstacles in the grid map can be determined according to the occupied area.

[0099] In S105, according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence, update the obstacle probability change value of the grid in the grid map corresponding to the scene image.

[0100] When the embodied robot in the embodiment of the present application starts to execute a task, it initializes an obstacle probability for each grid in the grid map corresponding to the scene map. During the execution of the task, according to the detected obstacle information, it determines the obstacle probability change value corresponding to the current detection result, that is, the magnitude of the impact on the original obstacle probability.

[0101] Considering that the closer the obstacle is to the embodied robot, the higher the probability of collision between the embodied robot and it. To improve the running safety of the embodied robot, when the distance between the obstacle and the monocular camera is smaller, the obstacle probability change value is larger. Conversely, when the distance between the obstacle and the monocular camera is larger, the obstacle probability change value is smaller.

[0102] In addition, considering that the risk levels of collisions between different obstacle categories and the robot are different, the impacts of different categories of obstacles on the obstacle probability can be set. For example, when detecting a dangerous obstacle such as a wire that is easy to entangle with the embodied robot, the magnitude of the obstacle probability change value can be increased. When detecting a touchable obstacle such as a sofa, the magnitude of the obstacle probability change value can be decreased.

[0103] In a possible implementation, the obstacle probability can be expressed as:

[0104] delta_p = norm(a * (b * norm(1 / dis) + c * conf)).

[0105] Wherein, delta_p represents the obstacle probability change value. Since this formula is used to update the obstacle probability of the grid where the obstacle exists when the obstacle is detected, therefore, this obstacle probability change value is also the obstacle probability increase value. Norm represents normalization to [0, 1], dis represents the distance between the monocular camera and the obstacle, conf represents the obstacle confidence, a represents the coefficient related to the category, b represents the coefficient related to the distance, and c represents the coefficient related to the obstacle confidence.

[0106] In S106, the grid map is updated according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and path planning is performed according to the updated grid map.

[0107] Determine the obstacle probability of the grid according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map. After determining the two-dimensional point cloud information corresponding to the obstacle in the grid map, the grid where the obstacle probability needs to be increased can be determined. According to the obstacle probability change value determined by the current scene image, combined with the obstacle probability of the grid determined by the previous frame of the scene image of the current scene image (for convenience of description, it can be called the original obstacle probability), the obstacle probability change value is superimposed on the original obstacle probability to obtain the updated obstacle probability of the grid. For example, the original obstacle probability is 0.5, and the obstacle probability change value calculated by distance, obstacle category, and obstacle confidence is 0.07, then the updated obstacle probability is 0.57.

[0108] Based on the updated obstacle probability, it can be compared with a predetermined probability threshold. If the obstacle probability of the grid is greater than or equal to the predetermined probability threshold, it is determined that there is an obstacle in the grid and the grid with the obstacle is updated. If the obstacle probability of the grid is less than the predetermined probability threshold, it is determined that there is no obstacle in the grid, and the original state of the grid without an obstacle is maintained. For example, the obstacle threshold is 0.6, and the updated obstacle probability of the grid is 0.57, which is less than the probability threshold, so it is determined that there is no obstacle in the grid, and thus there is no need to update, and the original state of the grid without an obstacle is maintained.

[0109] After updating the obstacle probabilities of each grid in the obstacle area corresponding to the scene image, an updated grid map can be obtained. Based on whether there are obstacles in the grid map, the task path is planned to reduce the chance of the embodied robot colliding with obstacles and improve the task execution efficiency of the robot.

[0110] In a possible implementation, after obtaining the scene image through a monocular vision sensor, the embodiment of the present application can improve the obstacle contour information through the matching of key point information, so that the embodied robot can perform path planning more accurately, such as Figure 3 shown, the method includes:

[0111] In S301, determine the first key point information of the obstacle in the scene image.

[0112] The first key point information in the embodiments of the present application may include feature points such as corner points, edge points, and texture points of obstacles. The Scale-Invariant Feature Transform (SIFT) method can be used to extract SIFT features in the obstacle detection frame to obtain descriptors of the first key point information.

[0113] In S302, the first key point information is matched with the second key point information of the obstacle stored in the map before the current time.

[0114] During the process of the embodied robot performing tasks, the detected obstacle positions and the key point information of the obstacles, i.e., the second key point information, will be recorded. The second key point information may include partial key point information of the obstacles. The second key point information will gradually accumulate over time during task execution, that is, the longer the execution time, the more data of the second key point information will accumulate.

[0115] When the first key point information is matched with the second key point information, the similarity between the local key point information in the second key point information and the first key point information can be determined. When the similarity meets the predetermined requirements, such as being greater than a predetermined similarity value, it is determined to match the second key point information.

[0116] In S303, in response to the matching of the first key point information and the second key point information, the obstacle confidence is increased and the obstacle contour information in the grid map is improved according to the contour information of the obstacle in the current perspective of the scene image.

[0117] Since the currently adopted scene image may include contour information that was not included before, when the first key point information is matched with the second key point information, the obstacle confidence can be increased to improve the obstacle probability change value of the grid, and the contour information of the obstacle in the map can be improved and supplemented according to the contour information of the current scene image, so as to obtain more accurate and comprehensive contour information of the obstacle, facilitate more accurate update of the obstacle probability, and improve the accuracy and effectiveness of path planning.

[0118] It should be noted that when the obstacle probability of the grid is greater than or equal to a predetermined probability threshold, it is updated that there is an obstacle in the grid. Only at this time will the contour information of the obstacle in the map be improved and supplemented according to the contour information of the current scene image.

[0119] In the embodiments of the present application, after obtaining a scene image through a monocular vision sensor, if no obstacle is detected in the scene image, that is, there is no obstacle currently within the field of view corresponding to the scene image. At this time, the obstacle probability can be updated by means of a ray path to improve the efficiency of obstacle probability update. The update process can be as follows: Figure 4 as shown, including:

[0120] In S401, if there is no obstacle in the scene image, a virtual point cloud is constructed at the bottom edge of the field of view in the current field of view of the monocular vision sensor.

[0121] When there is no obstacle in the scene image, that is, no obstacle is detected within the current field of view. At this time, only the obstacle probability of the grid within the field of view can be updated, that is, the obstacle probability of the grid is reduced.

[0122] To improve the update efficiency, the embodiments of the present application can construct a virtual point cloud according to the bottom edge of the field of view. As shown in Figure 5 the figure, the field of view of the monocular vision sensor of the embodied robot is triangular, with three sides being the left field of view boundary, the right field of view boundary, and the bottom edge of the field of view. The bottom edge of the field of view is the side that is the farthest from the monocular vision sensor among the three sides of the triangle. Among them, the obstacles within the field of view are the original obstacles in the grid map, that is, the obstacles determined by the detection results before obtaining the current scene image.

[0123] In S402, starting from the position of the monocular vision sensor and using each point of the virtual point cloud as a connection point, ray paths emitted into the field of view are determined.

[0124] After constructing a virtual point cloud according to the bottom line of the field of view, multiple ray paths located within the field of view can be obtained by connecting the starting point (the position of the embodied robot) and each point of the virtual point cloud.

[0125] In S403, according to the probability of each grid passed by the ray path, the original obstacle detection result of the grid in the grid map is determined.

[0126] After determining the position of each ray, the ray path can be determined according to the ray, and the original obstacle detection result of each grid within the field of view can be obtained one by one. The original obstacle detection result is the obstacle detection result of the grid within the field of view determined before the current scene image, and this detection result can include whether there is an obstacle in the grid and the obstacle probability of the grid.

[0127] In S404, according to the original obstacle detection result of the grid, the obstacle probability of the grid is updated.

[0128] According to different original obstacle detection results, the obstacle probability of the grid can be updated according to the corresponding update method. For example, if the original obstacle detection result of the grid is that there is an obstacle in the grid, the obstacle probability of the grid can be reduced. If the original obstacle detection result of the grid is that there is no obstacle in the grid, since the obstacle probability of the grid is already less than the probability threshold, the obstacle probability of the grid can be maintained unchanged, or the obstacle probability can be further reduced until it is reduced to the minimum value.

[0129] In the embodiment of the present application, when reducing the original obstacle probability, the probability reduction amount can be determined according to the original obstacle category and the distance between the original obstacle and the monocular vision sensor. According to the determined probability reduction amount, the original obstacle probability is reduced. For example, if the original obstacle probability is 0.8, which is greater than the probability threshold of 0.6, the probability reduction amount can be determined to be 0.5 according to the previously determined obstacle category and the current distance between the monocular vision sensor and the original obstacle. The obstacle probability is updated according to this probability reduction amount, and the updated obstacle probability is 0.3, which is less than the probability threshold.

[0130] In summary, the embodiment of the present application can accurately and effectively identify and update obstacles through a monocular vision sensor, improve the accuracy of obstacle avoidance on the basis of using low costs; determine two-dimensional point cloud information through the detected obstacle information, and update the obstacle probability multiple times according to the distance between the monocular vision sensor and the obstacle, the obstacle category, and the obstacle confidence, which can effectively reduce the influence of noise on the detection result and is beneficial to improving the accuracy of the obstacle detection result. Through the matching of key point information, the contour information of the obstacle can be effectively improved, and the accuracy of path planning can be improved. By updating the grid within the field of view in a ray manner, the update efficiency of the obstacle probability can be effectively improved.

[0131] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

[0132] Figure 6 The figure is a schematic diagram of an obstacle avoidance control device for an embodied robot provided by an embodiment of the present application. The embodied robot includes a monocular vision sensor, and the device includes:

[0133] A scene image acquisition unit 601, configured to acquire a scene image through the monocular vision sensor;

[0134] An obstacle information determination unit 602, configured to determine the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image if there is an obstacle in the scene image;

[0135] A distance determination unit 603, configured to determine the distance between the obstacle and the monocular vision sensor by a ground constraint method according to the position of the obstacle detection frame in the scene image;

[0136] A two-dimensional information determination unit 604, configured to determine a preset size of the obstacle according to the correspondence between the preset obstacle category and the size range and the size of the obstacle detection frame, and improve the obstacle range according to the preset size of the obstacle to determine the two-dimensional point cloud information of the obstacle in the grid map;

[0137] An obstacle probability update unit 605, configured to update the obstacle probability change value of the grid in the grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence;

[0138] A grid map update unit 606, configured to update the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and perform path planning according to the updated grid map.

[0139] Figure 6 The obstacle avoidance control device of the embodied robot shown corresponds to Figure 1 The obstacle avoidance control method of the embodied robot shown.

[0140] Figure 7 It is a schematic diagram of an embodied robot provided by an embodiment of the present application. As Figure 7 shown, the embodied robot 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as an obstacle avoidance control program for an embodied robot. When the processor 70 executes the computer program 72, the steps in the above-mentioned embodiments of the obstacle avoidance control method for an embodied robot are implemented. Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0141] Exemplarily, the computer program 72 can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the embodied robot 7.

[0142] The embodied robot may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art can understand that Figure 7This is merely an example of the embodied robot 7 and does not constitute a limitation on the embodied robot 7. It may include more or fewer components than those shown in the figure, or combine certain components, or have different components. For example, the embodied robot may further include input / output devices, network access devices, buses, etc.

[0143] The processor 70 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0144] The memory 71 may be an internal storage unit of the embodied robot 7, such as the hard disk or memory of the embodied robot 7. The memory 71 may also be an external storage device of the embodied robot 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the embodied robot 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the embodied robot 7. The memory 71 is used to store the computer program and other programs and data required by the embodied robot. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0145] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0146] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0148] In the embodiments provided in this application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0149] The unit described as a separated component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0151] If the above integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the processes in the above-described embodiment methods of the present application can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0152] In addition, the embodiments of the present application also provide a computer program product, which when running on a computer, causes the computer to execute the methods in the above various implementation manners.

[0153] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An obstacle avoidance control method for an embodied robot, characterized in that: The embodied robot includes a monocular vision sensor, and the method includes: Acquire scene images through a monocular vision sensor; If there is an obstacle in the scene image, determining the obstacle category, obstacle confidence and obstacle detection frame included in the scene image; According to the position of the obstacle detection frame in the scene image, determining the distance between the obstacle and the monocular vision sensor by a ground constraint method; Determining a preset size of the obstacle according to a preset correspondence between the obstacle category and the size range and the size of the obstacle detection frame, and improving the obstacle range according to the preset size of the obstacle, thereby determining the two-dimensional point cloud information of the obstacle in the grid map; According to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence, updating the obstacle probability change value of the grid in the grid map corresponding to the scene image; The grid map is updated according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and the path planning is performed according to the updated grid map.

2. The method according to claim 1, characterized in that After acquiring the scene image through the monocular vision sensor, the method further includes: Determining first key point information of an obstacle in the scene image; Matching the first key point information with the second key point information of the obstacle stored in the map before the current time; In response to the first key point information and the second key point information matching, the obstacle confidence is increased and the obstacle contour information in the grid map is improved according to the contour information of the obstacle in the scene image at the current viewing angle.

3. The method according to claim 1, characterized in that After acquiring the scene image through the monocular vision sensor, it also includes: If there are no obstacles in the scene image, a virtual point cloud is constructed at the bottom edge of the field of view within the field of view of the current monocular vision sensor; Taking the position of the monocular vision sensor as a starting point and each point of the virtual point cloud as a connection point, determining a ray path emitted into the field of view; Determining original obstacle detection results of the grids in the grid map according to the probability of each grid passed by the ray path; The obstacle probability of the grid is updated according to the original obstacle detection result of the grid.

4. The method according to claim 3, characterized in that Updating the obstacle probability of the grid according to the original obstacle detection result of the grid includes: When the original obstacle detection result of the grid indicates that there is an obstacle in the grid, reducing the obstacle probability of the grid; When the original obstacle detection result of the grid is that there is no obstacle in the grid, the obstacle probability of the grid is maintained unchanged.

5. The method according to claim 4, characterized in that Reducing the obstacle probability of the grid, including: Determining the probability reduction according to the original obstacle category in the grid map and the distance between the original obstacle and the monocular vision sensor; The obstacle probability of the grid is reduced according to the probability decrement.

6. The method according to claim 1, characterized in that Updating the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map includes: Determining the obstacle probability of the grid according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map; When the obstacle probability of the grid is greater than or equal to a predetermined probability threshold, updating the grid to indicate the existence of an obstacle; When the obstacle probability of the grid is less than a predetermined probability threshold, the grid is kept free of obstacles.

7. The method according to claim 1, characterized in that Determining the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image, including: The scene image is input into a pre-trained obstacle detection model, and the obstacle detection model outputs the obstacle category, obstacle confidence and obstacle detection frame.

8. An obstacle avoidance control device for an embodied robot, characterized in that: The embodied robot includes a monocular vision sensor, and the device includes: A scene image acquisition unit, used for acquiring a scene image through a monocular vision sensor; An obstacle information determination unit, configured to determine the obstacle category, obstacle confidence and obstacle detection frame included in the scene image if there is an obstacle in the scene image; a distance determination unit, configured to determine the distance between the obstacle and the monocular vision sensor by a ground constraint method according to the position of the obstacle detection frame in the scene image; A two-dimensional information determination unit, configured to determine a preset size of an obstacle according to a preset correspondence between an obstacle category and a size range and a size of the obstacle detection frame, and to improve the obstacle range according to the preset size of the obstacle, thereby determining two-dimensional point cloud information of the obstacle in a grid map; An obstacle probability updating unit, configured to update an obstacle probability change value of a grid in a grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence; The grid map updating unit is used to update the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and perform path planning according to the updated grid map.

9. An embodied robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the embodied robot implements the method according to any one of claims 1 to 7.

10. A computer program product comprising computer program instructions, characterized in that When the computer program is executed, the method according to any one of claims 1 to 7 is performed.

Images