Body-equipped robot, obstacle avoidance control method and device thereof, and program product

Abstract

The invention relates to the field of robot navigation, in particular to a robot with a body, an obstacle avoidance control method and device thereof and a program product. The method comprises the following steps: acquiring a scene image through a monocular vision sensor; determining an obstacle category, an obstacle confidence coefficient and an obstacle detection frame included in the scene image; determining the distance between the obstacle and the monocular vision sensor; determining two-dimensional point cloud information of the obstacle in the grid map; according to the obstacle category, the distance between the obstacle and the monocular vision sensor and the obstacle confidence, updating an obstacle probability change value of a grid in a grid map corresponding to the scene image; and updating the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and performing path planning according to the updated grid map. The method can effectively reduce the influence of noise on the obstacle detection result, improve the obstacle detection precision, and improve the task path planning accuracy.

Description

Technical Field

[0001] The present application relates to the field of robot navigation, and in particular to an embodied robot and its obstacle avoidance control method, device and program product. Background Art

[0002] In the process of robot execution, in order to prevent the robot from colliding with obstacles and reduce the number of unplanned downtime, the robot needs to detect obstacles in the moving scene to optimize the path of the robot to perform tasks. In general, the robot usually uses laser radar or binocular camera to detect obstacles in the scene where the robot is located, and simply marks the detected obstacles on the map. The robot plans the path according to the obstacles marked in the map to avoid obstacles in the scene.

[0003] However, due to cost and other factors, some robots are only equipped with wired lasers or monocular vision sensors (monocular cameras). When detecting obstacles based on wired lasers or monocular cameras, if there is noise, the obstacle detection results may be distorted, and non-obstacles may be mistakenly detected as obstacles, or obstacles may be detected as non-obstacles, affecting the accuracy of the task path planned by the robot and the efficiency of task execution. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an obstacle avoidance control method, device and program product for an embodied robot to solve the problem in the prior art that when using a monocular camera or a line laser to identify obstacles, false detection is prone to occur, affecting the accuracy of the planned task path.

[0005] A first aspect of an embodiment of the present application provides an obstacle avoidance control method for an embodied robot, the embodied robot comprising a monocular vision sensor, the method comprising: Acquire scene images through a monocular vision sensor; If there is an obstacle in the scene image, determining the obstacle category, obstacle confidence and obstacle detection frame included in the scene image; According to the position of the obstacle detection frame in the scene image, determining the distance between the obstacle and the monocular vision sensor by a ground constraint method; Determining a preset size of the obstacle according to a preset correspondence between the obstacle category and the size range and the size of the obstacle detection frame, and improving the obstacle range according to the preset size of the obstacle, thereby determining two-dimensional point cloud information of the obstacle in the grid map; According to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence, updating the obstacle probability change value of the grid in the grid map corresponding to the scene image; The grid map is updated according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and the path planning is performed according to the updated grid map.

[0006] In combination with the first aspect, in a first possible implementation manner of the first aspect, after acquiring the scene image through the monocular vision sensor, the method further includes: Determining first key point information of an obstacle in the scene image; Matching the first key point information with the second key point information of the obstacle stored in the map before the current time; In response to the first key point information and the second key point information matching, the obstacle confidence is increased and the obstacle contour information in the grid map is improved according to the contour information of the obstacle in the scene image at the current viewing angle.

[0007] In combination with the first aspect, in a second possible implementation manner of the first aspect, after acquiring a scene image through a monocular vision sensor, the method further includes: If there are no obstacles in the scene image, a virtual point cloud is constructed at the bottom edge of the field of view within the field of view of the current monocular vision sensor; Taking the position of the monocular vision sensor as a starting point and each point of the virtual point cloud as a connection point, determining a ray path emitted into the field of view; Determining original obstacle detection results of the grids in the grid map according to the probability of each grid passed by the ray path; The obstacle probability of the grid is updated according to the original obstacle detection result of the grid.

[0008] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, updating the obstacle probability of the grid according to the original obstacle detection result of the grid includes: When the original obstacle detection result of the grid indicates that there is an obstacle in the grid, reducing the obstacle probability of the grid; When the original obstacle detection result of the grid is that there is no obstacle in the grid, the obstacle probability of the grid is maintained unchanged.

[0009] In combination with the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, reducing the obstacle probability of the grid includes: Determine the probability reduction according to the original obstacle category in the grid map and the distance between the original obstacle and the monocular vision sensor; The obstacle probability of the grid is reduced according to the probability decrement.

[0010] In combination with any one of the first aspect to the fifth possible implementation manner of the first aspect, in the fifth possible implementation manner of the first aspect, updating the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map includes: Determining the obstacle probability of the grid according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map; When the obstacle probability of the grid is greater than or equal to a predetermined probability threshold, updating the grid to indicate the existence of an obstacle; When the obstacle probability of the grid is less than a predetermined probability threshold, the grid is kept free of obstacles.

[0011] In combination with any one of the first aspect to the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, determining the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image includes: The scene image is input into a pre-trained obstacle detection model, and the obstacle detection model outputs the obstacle category, obstacle confidence and obstacle detection frame.

[0012] A second aspect of an embodiment of the present application provides an obstacle avoidance control device for an embodied robot, the embodied robot comprising a monocular vision sensor, the device comprising: A scene image acquisition unit, used for acquiring a scene image through a monocular vision sensor; An obstacle information determination unit, configured to determine the obstacle category, obstacle confidence and obstacle detection frame included in the scene image if there is an obstacle in the scene image; a distance determination unit, configured to determine the distance between the obstacle and the monocular vision sensor by a ground constraint method according to the position of the obstacle detection frame in the scene image; A two-dimensional information determination unit, configured to determine a preset size of an obstacle according to a preset correspondence between an obstacle category and a size range and a size of the obstacle detection frame, and to complete the obstacle range according to the preset size of the obstacle so as to determine two-dimensional point cloud information of the obstacle in a grid map; An obstacle probability updating unit, configured to update an obstacle probability change value of a grid in a grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence; The grid map updating unit is used to update the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and perform path planning according to the updated grid map.

[0013] In combination with the second aspect, in a first possible implementation manner of the second aspect, the apparatus further includes: A key point information determining unit, used to determine first key point information of an obstacle in the scene image; A key point information matching unit, used to match the first key point information with the second key point information of the obstacle stored in the map before the current time; The obstacle contour information improvement unit is used to increase the obstacle confidence in response to the first key point information matching the second key point information and improve the obstacle contour information in the grid map according to the contour information of the obstacle in the scene image at the current viewing angle.

[0014] In combination with the second aspect, in a second possible implementation manner of the second aspect, the apparatus further includes: A virtual point cloud construction unit, used for constructing a virtual point cloud at the bottom edge of the field of view within the field of view of the current monocular vision sensor if there are no obstacles in the scene image; A ray determination unit, used to determine a ray path emitted into the field of view, taking the position of the embodied robot as a starting point and each point of the virtual point cloud as a connection point; A grid obstacle detection unit, used to determine the original obstacle detection results of the grids in the grid map according to the probability of each grid passed by the ray path; The probability updating unit is used to update the obstacle probability of the grid according to the original obstacle detection result of the grid.

[0015] In combination with the second possible implementation manner of the second aspect, in a third possible implementation manner of the second aspect, the probability updating unit includes: A first probability updating subunit, configured to reduce the obstacle probability of the grid when the original obstacle detection result of the grid indicates that an obstacle exists in the grid; The second probability updating subunit is used to maintain the obstacle probability of the grid unchanged when the original obstacle detection result of the grid is that there is no obstacle in the grid.

[0016] In combination with the third possible implementation manner of the second aspect, in a fourth possible implementation manner of the second aspect, the first probability updating subunit includes: A probability reduction determination module is used to determine the probability reduction according to the original obstacle category in the grid map and the distance between the original obstacle and the monocular vision sensor; A probability reduction module is used to reduce the obstacle probability of the grid according to the probability reduction.

[0017] In combination with any one of the second aspect to the fourth possible implementation manner of the second aspect, in a fifth possible implementation manner of the second aspect, the grid map updating unit includes: An obstacle probability determination subunit, used to determine the obstacle probability of the grid according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map; A first grid updating subunit, configured to update the grid to indicate that an obstacle exists when the probability of an obstacle in the grid is greater than or equal to a predetermined probability threshold; The second grid updating subunit is used to keep the grid free of obstacles when the obstacle probability of the grid is less than a predetermined probability threshold.

[0018] In combination with any one of the second aspect to the fourth possible implementation manner of the second aspect, in a sixth possible implementation manner of the second aspect, the obstacle information determination unit is used to: The scene image is input into a pre-trained obstacle detection model, and the obstacle detection model outputs the obstacle category, obstacle confidence and obstacle detection frame.

[0019] A third aspect of an embodiment of the present application provides an embodied robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the embodied robot implements the method described in any one of the first aspects.

[0020] A fourth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method in the first aspect or its various implementations.

[0021] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0022] The sixth aspect of the embodiment of the present application provides a chip for implementing the methods in each implementation of the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the method in the first aspect or its implementation.

[0023] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application collect scene images through a monocular vision sensor, determine the obstacle category, obstacle confidence and obstacle detection frame in the scene image by analyzing the scene image, determine the preset size of the obstacle based on the correspondence between the preset size range and the obstacle category and the size of the obstacle detection frame, improve the obstacle range based on the preset size, and determine the two-dimensional information of the obstacle in the scene image, so that more comprehensive obstacle information can be obtained, which is conducive to more accurate task path planning; based on the size of the obstacle detection frame, the distance between the obstacle and the monocular vision sensor is determined by the ground constraint method (which can be used to approximate the distance between the obstacle and the embodied robot), and the obstacle probability of the grid in the grid map corresponding to the scene image is continuously updated according to the distance, obstacle confidence and obstacle category, so that the influence of noise on the obstacle detection result can be effectively reduced, the obstacle detection accuracy is improved, and the accuracy of the task planning path of the embodied robot is further improved, which is conducive to improving the task execution efficiency of the embodied robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 It is a schematic diagram of the implementation flow of an obstacle avoidance control method of an embodied robot provided in an embodiment of the present application; Figure 2 This is a schematic diagram of an implementation of determining the distance between an obstacle and a monocular vision sensor provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the implementation process of a method for improving obstacle profile information provided in an embodiment of the present application; Figure 4 It is a schematic diagram of an implementation flow of a method for updating obstacle probability provided in an embodiment of the present application; Figure 5 is a schematic diagram of an obstacle probability based on ray updating grid provided in an embodiment of the present application; Figure 6 is a schematic diagram of an obstacle avoidance control device of an embodied robot provided in an embodiment of the present application; Figure 7 It is a schematic diagram of an embodied robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0027] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0028] Embodied robots (full name in English: Embodied Robots) are a type of intelligent robots that have physical entities and can interact with the environment in real time. Embodied robots can achieve a closed loop of perception, decision-making and action through the combination of sensors, actuators and computing units, so as to complete tasks in the real world. Embodied robots may include, but are not limited to, sweeping robots (also called cleaning robots), mopping robots driven by sweepers (i.e. sweeping and mopping robots), food delivery robots, autonomously driven carrying robots, accompanying robots, service robots, etc. It can be understood that the existence forms of the various robots listed above are not limited. For example, they can be wheeled robots, humanoid robots with two legs, or multi-legged robots.

[0029] In order to avoid collisions with obstacles and reduce unplanned downtime during robot missions, it is necessary to detect obstacles in the moving scene in real time to optimize the mission path. Traditional methods usually rely on lidar or binocular cameras for obstacle detection and mark the detection results on the map. The robot then plans the path to avoid obstacles.

[0030] However, due to cost considerations, some robots are only equipped with monocular vision sensors (monocular cameras) or line laser sensors. If there is noise, it may cause distortion in obstacle detection, misjudging non-obstacles as obstacles, or missing real obstacles, which will affect the accuracy of path planning and the efficiency of task execution.

[0031] To solve the above problems, the present application proposes an obstacle avoidance control method for an embodied robot, such as Figure 1 The following is a schematic diagram of the implementation process of this method, which is described in detail as follows: In S101, a scene image is acquired through a monocular vision sensor.

[0032] Among them, the monocular vision sensor usually refers to a monocular camera installed on the embodied robot, which is used to capture a two-dimensional image of the robot's surrounding environment. It includes RGB cameras, etc., which can be used to collect color images.

[0033] The scene image is the image data of the robot's surrounding environment, and there may be obstacles in the scene image.

[0034] The monocular vision sensor can capture scene images with a predetermined resolution at a predetermined frequency, for example, capturing scene images with a resolution of 720p at a frequency of 30 frames per second.

[0035] In S102, if there is an obstacle in the scene image, the obstacle category, obstacle confidence and obstacle detection frame included in the scene image are determined.

[0036] The obstacle category is the classification corresponding to the obstacles in the scene. For example, the obstacle category may include shoes, pedestrians, wires, sofas, tables, toys, and the like.

[0037] The obstacle confidence is used to indicate the probability that the detected obstacle is an obstacle, and is usually a value between 0 and 1. For example, an obstacle confidence of 0.6 means that the probability that the detected object is an obstacle is 60%.

[0038] The obstacle detection frame is used to identify the position and size of the obstacles detected in the scene image, and is usually represented by a rectangular frame.

[0039] The embodiment of the present application can use the target detection model in deep learning, such as the YOLO architecture model, to process the image, output the obstacle category, obstacle confidence and obstacle detection frame, input the scene image into the pre-trained obstacle detection model, and output the obstacle category, obstacle confidence and obstacle detection frame through the obstacle detection model. For example, when the image is input into the trained YOLOv5 model, the model will output the location coordinates of each detected obstacle category, obstacle confidence and obstacle detection frame.

[0040] The obstacle detection model used to output obstacle category, obstacle confidence and obstacle category can be a single model or two or three models. For example, three obstacle detection models can be used to parse and calculate the scene image and output obstacle category, obstacle confidence and obstacle detection box respectively.

[0041] The obstacle is usually on the same horizontal plane as the embodied robot. Therefore, the width of the obstacle detection frame can usually be used to represent one-dimensional information of the area occupied by the obstacle on the horizontal plane, for example, it can be used to represent the width of the area occupied by the obstacle.

[0042] In S103, according to the position of the obstacle detection frame in the scene image, the distance between the obstacle and the monocular vision sensor is determined by a ground constraint method.

[0043] Among them, the ground constraint method is to use the geometric relationship between the position of the obstacle in the image and the ground to calculate the distance between the obstacle and the monocular vision sensor (for example: monocular camera). The size of a single pixel can be determined according to the internal parameters of the monocular vision sensor. The position of the obstacle detection frame in the scene image can be used to determine the first pixel point of the obstacle's ground contact point in the scene image. The image distance can be obtained based on the number of pixels between the first pixel point and the center point of the scene image, combined with the size of a single pixel. According to the predetermined actual height of the monocular vision sensor, the focal length of the monocular vision sensor and the determined image distance, the distance between the obstacle and the monocular vision sensor can be determined by the back projection formula.

[0044] for example Figure 2 As shown in the obstacle imaging schematic diagram, the determined number of pixels (the number of pixels between the first pixel point of the obstacle's ground contact point in the scene image and the center point of the scene image) is n1, the height of each pixel is h1, the focal length of the monocular vision sensor is f, the distance between the monocular vision sensor and the obstacle is d, and the height of the monocular vision sensor is H. Then the distance between the monocular vision sensor and the obstacle can be expressed as: d=H*f / (h1*n1).

[0045] In S104, the preset size of the obstacle is determined according to the corresponding relationship between the preset obstacle category and the size range and the size of the obstacle detection frame, and the obstacle range is improved according to the preset size of the obstacle to determine the two-dimensional point cloud information of the obstacle in the grid map.

[0046] The two-dimensional point cloud information of the obstacle in the grid map is the projection of the collected discrete point set of the obstacle in the grid map, that is, the grid area occupied by the obstacle in the grid map.

[0047] The correspondence between obstacle categories and size ranges can be used to estimate the ranges occupied by obstacles of different categories. For example, the size range of a table is 30cm*120cm. The embodied robot can obtain the preset size of the obstacle (the preset outline size of the obstacle) based on the width of the obstacle detection frame in the detected obstacle size, combined with the determined obstacle category, and obtain the estimated area of ​​the obstacle. By collecting the outline information of the obstacle based on the estimated area, the area occupied by the obstacle can be fully determined. According to the correspondence between the grid map and the scene map, the grid occupied by the obstacle in the grid map is determined according to the area occupied by the obstacle in the scene map. According to the grid occupied by the obstacle in the grid map, the two-dimensional point cloud information of the obstacle in the grid map is determined.

[0048] In a possible implementation, the embodied robot can pre-detect the outline information and identification features of obstacles in the scene, and the identification features may include the distribution features of key points of the obstacles, etc. After acquiring an image including the obstacle, the current position of the obstacle and the area currently occupied by the obstacle can be determined based on the identification features of the obstacle, and the two-dimensional point cloud information of the obstacle in the grid map can be determined based on the occupied area.

[0049] In S105 , the obstacle probability change value of the grid in the grid map corresponding to the scene image is updated according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence.

[0050] When the embodied robot in the embodiment of the present application starts to perform a task, it will initialize an obstacle probability for each grid in the grid map corresponding to the scene map. During the execution of the task, it will determine the obstacle probability change value corresponding to the detection result, that is, the magnitude of the impact on the original obstacle probability, based on the detected obstacle information.

[0051] Considering that the closer the obstacle is to the embodied robot, the higher the probability of the embodied robot colliding with it, in order to improve the safety of the embodied robot, the smaller the distance between the obstacle and the monocular camera, the greater the obstacle probability change value. Conversely, the greater the distance between the obstacle and the monocular camera, the smaller the obstacle probability change value.

[0052] In addition, considering the different dangers of collisions between different obstacle types and the robot, the impact of different types of obstacles on the obstacle probability can be set. For example, when a dangerous obstacle such as a wire that is easy to get entangled with the embodied robot is detected, the size of the obstacle probability change value can be increased, and when a touchable obstacle such as a sofa is detected, the size of the obstacle probability change value can be reduced.

[0053] In a possible implementation, the obstacle probability can be expressed as: delta_p=norm(a*(b*norm(1 / dis)+c*conf).

[0054] Among them, delta_p represents the obstacle probability change value. Since this formula is used to update the obstacle probability of the grid where the obstacle exists when the obstacle is detected, the obstacle probability change value is also the obstacle probability increase value. Norm represents the normalization process to [0,1], dis represents the distance between the monocular camera and the obstacle, conf represents the obstacle confidence, a represents the coefficient related to the category, b represents the coefficient related to the distance, and c represents the coefficient related to the obstacle confidence.

[0055] In S106, the grid map is updated according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and path planning is performed according to the updated grid map.

[0056] The obstacle probability of the grid is determined according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map. After determining the two-dimensional point cloud information corresponding to the obstacle in the grid map, the grid that needs to increase the obstacle probability can be determined. According to the obstacle probability change value determined by the current scene image, combined with the obstacle probability of the grid determined by the previous frame scene image of the current scene image (for the convenience of description, it can be called the original obstacle probability), the obstacle probability change value is superimposed with the original obstacle probability to obtain the updated obstacle probability of the grid. For example, the original obstacle probability is 0.5, and the obstacle probability change value is 0.07 calculated by distance, obstacle category and obstacle confidence, then the updated obstacle probability is 0.57.

[0057] Based on the updated obstacle probability, it can be compared with the predetermined probability threshold. If the obstacle probability of the grid is greater than or equal to the predetermined probability threshold, it is determined that the grid has an obstacle, and the grid is updated to have an obstacle. If the obstacle probability of the grid is less than the predetermined probability threshold, it is determined that the grid does not have an obstacle, and the original state of the grid not having an obstacle is maintained. For example, if the obstacle threshold is 0.6, and the obstacle probability of the updated grid is 0.57, which is less than the probability threshold, it is determined that the grid does not have an obstacle, so it does not need to be updated, and the original state of the grid not having an obstacle is maintained.

[0058] After updating the obstacle probability of each grid in the obstacle area corresponding to the scene image, an updated grid map can be obtained. The task path is planned based on whether there are obstacles in the grid map to reduce the probability of collision between the embodied robot and obstacles and improve the robot's task execution efficiency.

[0059] In a possible implementation, after acquiring a scene image through a monocular vision sensor, the embodiment of the present application can improve the obstacle contour information by matching key point information, so that the embodied robot can perform path planning more accurately, such as Figure 3 As shown, the method includes: In S301, first key point information of an obstacle in the scene image is determined.

[0060] The first key point information in the embodiment of the present application may include feature points such as corner points, edge points, texture points, etc. of the obstacle. The SIFT feature extraction in the obstacle detection frame can be extracted by the SIFT (Scale-Invariant Feature Transform) method to obtain the descriptor of the first key point information.

[0061] In S302, the first key point information is matched with the second key point information of the obstacle stored in the map before the current time.

[0062] When the embodied robot performs a task, it will record the detected obstacle position and the key point information of the obstacle, i.e., the second key point information. The second key point information may include part of the key point information of the obstacle. The second key point information will gradually accumulate as the task is executed, i.e., the longer the execution time, the more data the second key point information accumulates.

[0063] When the first key point information is matched with the second key point information, the similarity between the local key point information in the second key point information and the first key point information can be determined. If the similarity meets a predetermined requirement, such as being greater than a predetermined similarity value, it is determined that the second key point information is matched.

[0064] In S303, in response to the first key point information and the second key point information matching, the obstacle confidence is increased and the obstacle contour information in the grid map is improved according to the contour information of the obstacle in the scene image at the current viewing angle.

[0065] Since the currently used scene image may include contour information that was not included before, when the first key point information matches the second key point information, the obstacle confidence can be increased, thereby increasing the obstacle probability change value of the grid, and according to the contour information of the current scene image, the contour information of the obstacles in the map is improved and supplemented, so that more accurate and comprehensive obstacle contour information can be obtained, which facilitates more accurate updating of obstacle probability and improves the accuracy and effectiveness of path planning.

[0066] It should be noted that when the obstacle probability of the grid is greater than or equal to the predetermined probability threshold, the grid is updated to indicate the existence of an obstacle. Only then will the outline information of the obstacle in the map be improved and supplemented according to the outline information of the current scene image.

[0067] In the embodiment of the present application, after the scene image is acquired by the monocular vision sensor, if no obstacle is detected in the scene image, that is, there is no obstacle in the field of view corresponding to the scene image, then the obstacle probability can be updated by the ray path method to improve the efficiency of the obstacle probability update. The update process can be as follows: Figure 4 As shown, including: In S401, if there are no obstacles in the scene image, a virtual point cloud is constructed at the bottom edge of the field of view within the field of view of the current monocular vision sensor.

[0068] When there is no obstacle in the scene image, that is, no obstacle is detected in the current field of view, at this time, only the obstacle probability of the grid within the field of view may be updated, that is, the obstacle probability of the grid may be reduced.

[0069] In order to improve the updating efficiency, the embodiment of the present application can construct a virtual point cloud according to the bottom edge of the field of view. Figure 5 As shown in the figure, the field of view of the monocular vision sensor of the embodied robot is a triangle, and the three sides are the left field boundary, the right field boundary and the bottom edge of the field of view. The bottom edge of the field of view is the edge of the three sides of the triangle that is farthest from the monocular vision sensor. Among them, the obstacles within the field of view are the original obstacles in the grid map, that is, the obstacles determined by the detection results before obtaining the current scene image.

[0070] In S402, the position of the monocular vision sensor is used as a starting point, and each point of the virtual point cloud is used as a connection point to determine a ray path emitted into the field of view.

[0071] After constructing a virtual point cloud according to the bottom line of the field of view, the position of the embodied robot can be used as the starting point, and each point of the virtual point cloud can be used as a connection point. The starting point and the connection point are connected into a line to obtain multiple ray paths within the field of view.

[0072] In S403, the original obstacle detection results of the grids in the grid map are determined according to the probability of each grid passed by the ray path.

[0073] After determining the position of each ray, the ray path can be determined according to the ray, and the original obstacle detection results of the grids within the field of view can be obtained one by one. The original obstacle detection results, that is, the obstacle detection results of the grids within the field of view determined before the current scene image, can include whether there are obstacles in the grid and the obstacle probability of the grid.

[0074] In S404, the obstacle probability of the grid is updated according to the original obstacle detection result of the grid.

[0075] According to the different original obstacle detection results, the obstacle probability of the grid can be updated according to the corresponding update method. For example, if the original obstacle detection result of the grid is that there is an obstacle in the grid, the obstacle probability of the grid can be reduced. If the original obstacle detection result of the grid is that there is no obstacle in the grid, since the obstacle probability of the grid is less than the probability threshold, the obstacle probability of the grid can be maintained unchanged, or the obstacle probability can be further reduced until it is reduced to a minimum value.

[0076] In the embodiment of the present application, when reducing the original obstacle probability, the probability reduction can be determined according to the original obstacle category and the distance between the original obstacle and the monocular vision sensor. According to the determined probability reduction, the original obstacle probability is reduced. For example, the original obstacle probability is 0.8, which is greater than the probability threshold of 0.6. According to the previously determined obstacle category and the distance between the current monocular vision sensor and the original obstacle, the probability reduction can be determined to be 0.5. The obstacle probability is updated according to the probability reduction, and the updated obstacle probability is 0.3, which is less than the probability threshold.

[0077] In summary, the embodiments of the present application can accurately and effectively identify and update obstacles through a monocular vision sensor, thereby improving the accuracy of obstacle avoidance at a low cost; the two-dimensional point cloud information is determined by the detected obstacle information, and the obstacle probability is updated multiple times through the distance between the monocular vision sensor and the obstacle, the obstacle category, and the obstacle confidence, which can effectively reduce the impact of noise on the detection results and help improve the accuracy of the obstacle detection results. By matching key point information, the outline information of the obstacle can be effectively improved and the accuracy of path planning can be improved. By updating the grid within the field of view in a ray manner, the update efficiency of the obstacle probability can be effectively improved.

[0078] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0079] Figure 6 A schematic diagram of an obstacle avoidance control device of an embodied robot provided in an embodiment of the present application, wherein the embodied robot includes a monocular vision sensor, and the device includes: The scene image acquisition unit 601 is used to acquire the scene image through a monocular vision sensor; The obstacle information determining unit 602 is used to determine the obstacle category, obstacle confidence and obstacle detection frame included in the scene image if there is an obstacle in the scene image; A distance determination unit 603 is used to determine the distance between the obstacle and the monocular vision sensor by a ground constraint method according to the position of the obstacle detection frame in the scene image; A two-dimensional information determination unit 604 is used to determine a preset size of the obstacle according to a preset correspondence between the obstacle category and the size range and the size of the obstacle detection frame, and to improve the obstacle range according to the preset size of the obstacle so as to determine the two-dimensional point cloud information of the obstacle in the grid map; The obstacle probability updating unit 605 is used to update the obstacle probability change value of the grid in the grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence; The grid map updating unit 606 is used to update the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and perform path planning according to the updated grid map.

[0080] Figure 6 The obstacle avoidance control device of the embodied robot shown in FIG. Figure 1 The obstacle avoidance control method of the embodied robot shown corresponds to this.

[0081] Figure 7 is a schematic diagram of an embodied robot provided in an embodiment of the present application. Figure 7 As shown, the embodied robot 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as an obstacle avoidance control program of the embodied robot. When the processor 70 executes the computer program 72, the steps of the above-mentioned obstacle avoidance control method embodiments of the embodied robot are implemented. Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0082] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 72 in the embodied robot 7.

[0083] The embodied robot may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7It is only an example of an embodied robot 7 and does not constitute a limitation of the embodied robot 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the embodied robot may also include input and output devices, network access devices, buses, etc.

[0084] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0085] The memory 71 may be an internal storage unit of the embodied robot 7, such as a hard disk or memory of the embodied robot 7. The memory 71 may also be an external storage device of the embodied robot 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the embodied robot 7. Furthermore, the memory 71 may also include both an internal storage unit and an external storage device of the embodied robot 7. The memory 71 is used to store the computer program and other programs and data required by the embodied robot. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0086] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0087] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0088] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0089] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0091] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0092] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0093] In addition, an embodiment of the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the methods in the above-mentioned implementation modes.

[0094] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An obstacle avoidance control method for an embodied robot, characterized in that: The embodied robot includes a monocular vision sensor, and the method includes: Acquire scene images through a monocular vision sensor; If there is an obstacle in the scene image, determining the obstacle category, obstacle confidence and obstacle detection frame included in the scene image; According to the position of the obstacle detection frame in the scene image, determining the distance between the obstacle and the monocular vision sensor by a ground constraint method; Determining a preset size of the obstacle according to a preset correspondence between the obstacle category and the size range and the size of the obstacle detection frame, and improving the obstacle range according to the preset size of the obstacle, thereby determining the two-dimensional point cloud information of the obstacle in the grid map; According to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence, updating the obstacle probability change value of the grid in the grid map corresponding to the scene image; The grid map is updated according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and the path planning is performed according to the updated grid map.

2. The method according to claim 1, characterized in that After acquiring the scene image through the monocular vision sensor, the method further includes: Determining first key point information of an obstacle in the scene image; Matching the first key point information with the second key point information of the obstacle stored in the map before the current time; In response to the first key point information and the second key point information matching, the obstacle confidence is increased and the obstacle contour information in the grid map is improved according to the contour information of the obstacle in the scene image at the current viewing angle.

3. The method according to claim 1, characterized in that After acquiring the scene image through the monocular vision sensor, it also includes: If there are no obstacles in the scene image, a virtual point cloud is constructed at the bottom edge of the field of view within the field of view of the current monocular vision sensor; Taking the position of the monocular vision sensor as a starting point and each point of the virtual point cloud as a connection point, determining a ray path emitted into the field of view; Determining original obstacle detection results of the grids in the grid map according to the probability of each grid passed by the ray path; The obstacle probability of the grid is updated according to the original obstacle detection result of the grid.

4. The method according to claim 3, characterized in that Updating the obstacle probability of the grid according to the original obstacle detection result of the grid includes: When the original obstacle detection result of the grid indicates that there is an obstacle in the grid, reducing the obstacle probability of the grid; When the original obstacle detection result of the grid is that there is no obstacle in the grid, the obstacle probability of the grid is maintained unchanged.

5. The method according to claim 4, characterized in that Reducing the probability of obstacles in the grid, including: Determining the probability reduction according to the original obstacle category in the grid map and the distance between the original obstacle and the monocular vision sensor; The obstacle probability of the grid is reduced according to the probability decrement.

6. The method according to claim 1, characterized in that Updating the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map includes: Determining the obstacle probability of the grid according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map; When the obstacle probability of the grid is greater than or equal to a predetermined probability threshold, updating the grid to indicate the existence of an obstacle; When the obstacle probability of the grid is less than a predetermined probability threshold, the grid is kept free of obstacles.

7. The method according to claim 1, characterized in that Determining the obstacle category, obstacle confidence, and obstacle detection frame included in the scene image, including: The scene image is input into a pre-trained obstacle detection model, and the obstacle detection model outputs the obstacle category, obstacle confidence and obstacle detection frame.

8. An obstacle avoidance control device for an embodied robot, characterized in that: The embodied robot includes a monocular vision sensor, and the device includes: A scene image acquisition unit, used for acquiring a scene image through a monocular vision sensor; An obstacle information determination unit, configured to determine the obstacle category, obstacle confidence and obstacle detection frame included in the scene image if there is an obstacle in the scene image; a distance determination unit, configured to determine the distance between the obstacle and the monocular vision sensor by a ground constraint method according to the position of the obstacle detection frame in the scene image; A two-dimensional information determination unit, configured to determine a preset size of an obstacle according to a preset correspondence between an obstacle category and a size range and a size of the obstacle detection frame, and to improve the obstacle range according to the preset size of the obstacle, thereby determining two-dimensional point cloud information of the obstacle in a grid map; An obstacle probability updating unit, configured to update an obstacle probability change value of a grid in a grid map corresponding to the scene image according to the obstacle category, the distance between the obstacle and the monocular vision sensor, and the obstacle confidence; The grid map updating unit is used to update the grid map according to the obstacle probability change value of the grid and the two-dimensional point cloud information of the obstacle in the grid map, and perform path planning according to the updated grid map.

9. An embodied robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the embodied robot implements the method according to any one of claims 1 to 7.

10. A computer program product comprising computer program instructions, characterized in that When the computer program is executed, the method according to any one of claims 1 to 7 is performed.

Images