Intelligent obstacle avoidance method, device, robot and medium based on binocular vision

Abstract

Embodiments of the present invention disclose an intelligent obstacle avoidance method, device, robot, and medium based on binocular vision. The method includes: obtaining a depth map of the work scene in the direction of the robot's movement through a binocular camera, and calculating a corresponding height map based on the depth map; determining first obstacle information in the work scene based on the depth map, and determining second obstacle information in the work scene based on the height map; determining target obstacles in the work scene and their position information based on the first obstacle information and the second obstacle information; and controlling the robot to avoid obstacles based on the position information and the robot's preset path. This method avoids the environmental interference problem that occurs when using depth maps alone for identification, thereby improving the accuracy of identifying obstacles ahead during the robot's movement and enabling the robot to effectively avoid obstacles.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of robotics technology, and in particular to an intelligent obstacle avoidance method, device, robot, and medium based on binocular vision. Background Art

[0002] In recent years, with the continuous advancement of science and technology and the need for industrial transformation and upgrading, more and more intelligent devices have entered people's work and lives, aiming to improve production efficiency and reduce labor costs. Mobile robots, as a representative intelligent product, are widely used in homes and various production sites. For example, a feed-pushing robot in a pasture faces a constantly changing environment due to the varying conditions of each pasture, and the frequent presence of feed spreaders or production vehicles parked in the cattle shed. To ensure that the pusher robot's work is not affected, it must identify and avoid these obstacles.

[0003] Currently, common obstacle avoidance methods include infrared, ultrasonic, laser, and visual. Both infrared and ultrasonic methods require the active emission of light and sound waves, which requires specific reflective properties. For example, infrared light is absorbed by black objects. While binocular vision is also affected by light, it is less susceptible to interference in a cattle shed, making it highly applicable. While lasers offer similar functions to binocular vision, lidar is more expensive and bulky. Furthermore, the heat generated by these devices can easily attract mosquitoes, compromising their effectiveness.

[0004] However, the currently used binocular vision obstacle avoidance method may identify some wrong obstacles, resulting in low obstacle recognition accuracy, which in turn affects the mobile robot's obstacle avoidance process and causes some errors. Summary of the Invention

[0005] Embodiments of the present invention provide a binocular vision-based intelligent obstacle avoidance method, device, robot, and medium to improve the accuracy of obstacle recognition, thereby enabling the robot to effectively avoid obstacles.

[0006] In a first aspect, an embodiment of the present invention provides an intelligent obstacle avoidance method based on binocular vision, the method comprising:

[0007] Obtain a depth map of the working scene in the direction of robot movement through a binocular camera, and calculate a corresponding height map based on the depth map;

[0008] Determining first obstacle information in the operation scene according to the depth map, and determining second obstacle information in the operation scene according to the height map;

[0009] determining a target obstacle existing in the operation scene and position information of the target obstacle according to the first obstacle information and the second obstacle information;

[0010] The robot is controlled to avoid obstacles according to the position information and the preset path of the robot.

[0011] Optionally, determining a target obstacle existing in the operation scene and location information of the target obstacle according to the first obstacle information and the second obstacle information includes:

[0012] Determining a first initial obstacle in the depth map according to the first obstacle information, and determining a second initial obstacle in the height map according to the second obstacle information;

[0013] Identifying an overlapping area between the first initial obstacle and the second initial obstacle, and calculating an overlapping rate between the first initial obstacle and the second initial obstacle based on the overlapping area;

[0014] The overlap rate is compared with a preset overlap rate threshold, and if the overlap rate is greater than or equal to the preset overlap rate threshold, the corresponding first initial obstacle or the second initial obstacle is determined as the target obstacle.

[0015] Optionally, identifying an overlapping area between the first initial obstacle and the second initial obstacle, and calculating an overlapping ratio between the first initial obstacle and the second initial obstacle based on the overlapping area includes:

[0016] Extracting a first minimum bounding rectangle of the first initial obstacle and a second minimum bounding rectangle of the second initial obstacle;

[0017] Identify the overlapping area according to the first minimum enclosing rectangle and the second minimum enclosing rectangle;

[0018] The overlap ratio is calculated according to the first coordinates of the first minimum bounding rectangle and the second coordinates of the second minimum bounding rectangle.

[0019] Optionally, the determining a target obstacle existing in the operation scene and location information of the target obstacle according to the first obstacle information and the second obstacle information further includes:

[0020] The position information is determined according to the first coordinates and / or the second coordinates corresponding to the target obstacle.

[0021] Optionally, after determining a first initial obstacle in the depth map according to the first obstacle information, the method further includes:

[0022] screening the first initial obstacles according to the first contour areas of the first initial obstacles to remove the first initial obstacles whose first contour areas are smaller than a first preset area threshold;

[0023] After determining the second initial obstacle in the height map according to the second obstacle information, the method further includes:

[0024] The second initial obstacles are screened according to the second contour areas of the second initial obstacles to remove the second initial obstacles whose second contour areas are smaller than a second preset area threshold.

[0025] Optionally, calculating a corresponding height map according to the depth map includes:

[0026] Converting the depth map into point cloud data of the operation scene;

[0027] The point cloud data is converted into the height map.

[0028] Optionally, determining first obstacle information in the operation scene according to the depth map, and determining second obstacle information in the operation scene according to the height map, includes:

[0029] Determine a first obstacle outline and coordinate information of the first obstacle outline in the depth map using a contour search algorithm or a clustering algorithm;

[0030] A contour search algorithm or a clustering algorithm is used to determine a second obstacle contour in the height map and coordinate information of the second obstacle contour.

[0031] In a second aspect, an embodiment of the present invention further provides an intelligent obstacle avoidance device based on binocular vision, the device comprising:

[0032] An image acquisition module is used to obtain a depth map of the working scene in the direction of the robot's movement through a binocular camera, and calculate a corresponding height map based on the depth map;

[0033] an obstacle information determination module, configured to determine first obstacle information in the operation scene according to the depth map, and to determine second obstacle information in the operation scene according to the height map;

[0034] an obstacle determination module, configured to determine a target obstacle existing in the operation scene and position information of the target obstacle based on the first obstacle information and the second obstacle information;

[0035] The obstacle avoidance control module is used to control the robot to avoid obstacles based on the position information and the preset path of the robot.

[0036] In a third aspect, an embodiment of the present invention further provides a robot, comprising:

[0037] one or more processors;

[0038] a memory for storing one or more programs;

[0039] When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent obstacle avoidance method based on binocular vision provided by any embodiment of the present invention.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent obstacle avoidance method based on binocular vision provided by any embodiment of the present invention.

[0041] An embodiment of the present invention provides an intelligent obstacle avoidance method based on binocular vision. First, a depth map of the working scene in the direction of the robot's movement is obtained through a binocular camera, and a corresponding height map is calculated based on the depth map. Then, the first obstacle information in the working scene is determined based on the depth map, and the second obstacle information in the working scene is determined based on the height map. Then, the target obstacles actually existing in the working scene and their position information are determined based on the obtained first obstacle information and second obstacle information. Thus, the robot can be controlled to avoid obstacles based on the position information of each target obstacle in combination with the robot's preset path. The intelligent obstacle avoidance method based on binocular vision provided by the embodiment of the present invention determines the final target obstacle and its position information by combining the obstacle information obtained by respectively identifying the depth map and the height map, thereby avoiding the environmental interference problem that exists in the process of identification using only the depth map, thereby improving the accuracy of identifying obstacles in front of the robot during movement, so that the robot can effectively avoid obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of the intelligent obstacle avoidance method based on binocular vision provided in Example 1 of the present invention;

[0043] Figure 2 A schematic structural diagram of an intelligent obstacle avoidance device based on binocular vision provided in the second embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of the robot provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0046] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0047] Example 1

[0048] Figure 1 This is a flow chart of the binocular vision-based intelligent obstacle avoidance method provided in the first embodiment of the present invention. This embodiment is applicable to various robots avoiding obstacles to prevent collisions during the movement process of performing tasks. This method can be performed by the binocular vision-based intelligent obstacle avoidance device provided in the embodiment of the present invention. The device can be implemented in hardware and / or software and can generally be integrated into various robots. Figure 1 As shown, the specific steps include:

[0049] S11. Obtain a depth map of the working scene in the direction of robot movement through a binocular camera, and calculate a corresponding height map based on the depth map.

[0050] Among them, the binocular camera can use a realsense camera, and the depth map can be directly read from the realsense camera through the corresponding software interface. By using the binocular camera, a depth map with a relatively high resolution can be obtained, and the detected objects are more three-dimensional, which makes obstacle avoidance more flexible and accurate. Specifically, when the robot (such as a pushing robot and a mowing robot) is in operation, it can use the binocular camera to shoot the working scene in the current moving direction in real time. The captured image can be directly calculated by the binocular camera to obtain the corresponding depth map, and then the depth map is transmitted to the obstacle avoidance processor as the input of the obstacle detection algorithm. After obtaining the depth map, the height map corresponding to the working scene can also be calculated based on the depth map, so that the final target obstacle can be determined by combining the depth map and the height map.

[0051] Optionally, calculating a corresponding height map based on the depth map includes: converting the depth map into point cloud data of the work scene; and converting the point cloud data into the height map. Specifically, the process of converting the depth map into point cloud data is the process of converting the image coordinate system into the world coordinate system. Similarly, the process of converting the point cloud data into the height map is the process of converting the world coordinate system back into the image coordinate system. The conversion process can be based on the intrinsic and extrinsic parameters of the binocular camera, which can be specifically read from the binocular camera. The specific conversion process can apply any existing conversion method, and this embodiment does not limit this.

[0052] S12: Determine first obstacle information in the operation scene according to the depth map, and determine second obstacle information in the operation scene according to the height map.

[0053] Specifically, after obtaining the depth map and the height map, obstacle information can be extracted from the depth map and the height map respectively, that is, first obstacle information and second obstacle information can be obtained respectively. The specific extraction process can apply any existing contour extraction method, and the extracted obstacle information can include the contour coordinate information of the obstacle.

[0054] Optionally, determining the first obstacle information in the work scene based on the depth map, and determining the second obstacle information in the work scene based on the height map, includes: using a contour search algorithm or a clustering algorithm to determine the first obstacle contour and the coordinate information of the first obstacle contour in the depth map; using a contour search algorithm or a clustering algorithm to determine the second obstacle contour and the coordinate information of the second obstacle contour in the height map. Specifically, the contour search algorithm or the clustering algorithm can be used to extract the contour in the image. Taking the contour search algorithm as an example, the findcontours function in OpenCV can be used. First, an initial starting point can be determined (such as the pixel point in the lower left corner of the image). Then, starting from the initial starting point, the boundary points between the foreground point and the background point are searched in the depth map according to a preset path finding method. The obtained boundary points are connected in sequence to form the first obstacle contour. The coordinate information of the first obstacle contour can be determined based on the coordinates of each boundary point. Among them, the foreground point is the pixel point belonging to the obstacle, and the background point is the pixel point not belonging to the obstacle. The basis for judging whether it is a boundary point is: if the pixel value difference between two adjacent pixel points is greater than a preset threshold, then the foreground point between the two pixel points can be used as a boundary point.

[0055] The above process can be used to extract the second obstacle outline and its coordinate information from the height map. However, due to the different heights of the obstacles, the height values of different parts of the same obstacle in the height map vary greatly. Therefore, the range of the height difference between the pixels at different positions on the second obstacle outline and the adjacent background points will also be relatively large. In this case, if a fixed preset threshold is used in analogy with the depth map, the result of boundary point determination may not be accurate. It can be seen that if the original height value of the height map is directly used, it will be difficult to determine whether it is a boundary point, and it is inconvenient to directly set a more appropriate threshold for comparison. Therefore, before searching for the above boundary points according to the preset path finding method, the height map can be first binarized. Specifically, the pixel values of the pixels in the height map with a height value greater than a preset pixel value (such as 1) can be assigned to 1, and the pixel values of other pixels can be assigned to 0. Therefore, when determining whether it is a boundary point, it is only necessary to determine whether the adjacent pixel values are 0 or 1. By binarizing the height map, the foreground and background points in the height map are directly distinguished, and the pixel value of each point is only 0 or 1. Therefore, there is no need to set an additional preset threshold for comparison, and the boundary points can be obtained more accurately.

[0056] S13: Determine a target obstacle in the operation scene and position information of the target obstacle based on the first obstacle information and the second obstacle information.

[0057] Specifically, a depth map indicates the distance from the camera to an object, while a height map indicates the height of an object. Because the first obstacle information obtained through the depth map can be affected by environmental factors such as the ground, non-obstacles such as the ground may be identified as obstacles. The second obstacle information obtained through the height map, however, lacks distance information and can result in the identification of distant obstacles that would not affect the robot's operation, causing the robot to stop at a considerable distance. Therefore, using either the depth map or the height map alone to determine the presence of obstacles in the forward working scene is generally inaccurate. In this embodiment, the first and second obstacle information can be combined to eliminate some of these misidentified obstacles, thereby filtering out the target obstacles that truly affect the robot's operation. Specifically, obstacles that only appear in the first or second obstacle information can be eliminated. Position information can include coordinate information and distance information. Coordinate information can be determined based on the coordinate position of the target obstacle in the depth map or height map, while distance information can be determined based on the depth values of each pixel at the coordinate position of the target obstacle in the depth map.

[0058] Optionally, determining a target obstacle in the work scene and the location information of the target obstacle based on the first obstacle information and the second obstacle information includes: determining a first initial obstacle in the depth map based on the first obstacle information, and determining a second initial obstacle in the height map based on the second obstacle information; identifying an overlapping area between the first initial obstacle and the second initial obstacle, and calculating an overlap ratio between the first initial obstacle and the second initial obstacle based on the overlapping area; and comparing the overlap ratio with a preset overlap ratio threshold. If the overlap ratio is greater than or equal to the preset overlap ratio threshold, determining the corresponding first initial obstacle or second initial obstacle as the target obstacle. Specifically, the first initial obstacle can be determined based on a first obstacle outline obtained in the first obstacle information. The first initial obstacle is an object in the robot work scene, such as a vehicle or a stone. The number of first initial obstacles can be one, two, or more. It should be noted that mutually independent closed contours within the first obstacle outline are determined as a single obstacle outline, thereby determining each first initial obstacle based on each contour. The method for determining the second initial obstacle is the same as the method for determining the first initial obstacle and is not further described here. In actual operational scenarios, the first initial obstacle obtained from the depth map may be the same vehicle as the second initial obstacle obtained from the height map. Alternatively, the first initial obstacle obtained from the depth map may be the ground, but the second initial obstacle obtained from the height map may not contain the ground. Common sense suggests that the overlap ratio of the overlapping area between the first and second initial obstacles is maximized only when they are the same obstacle.

[0059] After determining the first and second initial obstacles, the overlapping area between the first and second initial obstacles can be identified based on the outline coordinates of the first and second initial obstacles. As described above, the number of both the first and second initial obstacles can be one or more, and the resulting overlapping area can also be one or more. If only one overlapping area is obtained, the overlap ratio between the first and second initial obstacles can be directly calculated using this overlapping area. Specifically, the overlap ratio can be calculated by dividing the overlapping area of the overlapping area by the outline area of the first initial obstacle, the outline area of the second initial obstacle, or the area of the union of the outlines of the first and second initial obstacles. The overlapping area can be calculated based on the identified outline coordinates of the overlapping area. For example, if one pixel occupies one unit area, the overlapping area can be determined based on the number of pixels contained in the outline coordinates of the overlapping area. Alternatively, the overlapping area can be calculated using other graphic area calculation formulas, which are not specifically limited in this embodiment. When multiple overlapping areas are obtained, the area of each overlapping area can be calculated separately, and the contour coordinates of the first initial obstacle and the contour coordinates of the second initial obstacle corresponding to each overlapping area can be used for calculation. The specific calculation process is the same as before and will not be repeated here. After calculating one or more overlap rates, each overlap rate can be compared with a preset overlap rate threshold (such as 80%). If the overlap rate is greater than or equal to the preset overlap rate threshold, the first initial obstacle or the second initial obstacle corresponding to the overlap rate can be determined as the target obstacle. By identifying the overlapping areas, obstacles that only appear in the depth map or height map can be first screened out. Further, by comparing the overlap rate with the preset overlap rate threshold, it can be determined with a higher probability that the obstacles identified in the depth map and height map belong to the same obstacle, thereby making obstacle identification more accurate so that the robot can avoid obstacles more accurately. In particular, the ground may be identified as an obstacle in the depth map. In this case, when an obstacle actually exists in the ground area, it can be normally identified in the height map. If the distance between the obstacle and the robot is close, obstacle avoidance is required. However, the area of the ground area identified is usually large, while the area of the obstacle outline identified in the height map is not very large. In this case, it may be determined that obstacle avoidance is not required because the calculated overlap rate does not reach the preset overlap rate threshold. This is obviously inconsistent with the actual situation. Therefore, when shooting with a binocular camera, the area of the ground area that may be identified can be reduced by raising the camera angle so that the calculated overlap rate can fall within the preset overlap rate threshold, thereby achieving accurate identification.

[0060] Further optionally, identifying the overlapping area between the first initial obstacle and the second initial obstacle and calculating the overlap ratio between the first initial obstacle and the second initial obstacle based on the overlapping area includes: extracting a first minimum bounding rectangle (MBR) of the first initial obstacle and a second minimum bounding rectangle (MBR) of the second initial obstacle; identifying the overlapping area based on the first and second minimum bounding rectangles; and calculating the overlap ratio based on the first coordinates of the first and second minimum bounding rectangles. Specifically, when calculating the overlap ratio, the first and second minimum bounding rectangles of the first and second initial obstacle's outlines can be first calculated. The overlapping area between the first and second minimum bounding rectangles can then be used as the overlapping area between the first and second initial obstacles. The outline of the overlapping area can be identified based on the first and second coordinates of the first and second minimum bounding rectangles. The area of the overlapping area can then be calculated, and the overlap ratio can be calculated by dividing the area by the area of the first and second minimum bounding rectangles, or by the area of the union of the first and second minimum bounding rectangles. It should be noted that when there are multiple first and second initial obstacles, it is necessary to calculate the minimum bounding rectangles of each obstacle and further obtain multiple overlap ratios. This embodiment uses the minimum bounding rectangle to calculate the overlap ratio for subsequent judgment, making the calculation process simpler, thereby improving computer performance and conserving computer resources.

[0061] Further optionally, determining the target obstacle present in the work scene and the target obstacle's location information based on the first obstacle information and the second obstacle information further includes determining the location information based on the first coordinates and / or the second coordinates corresponding to the target obstacle. Specifically, after the target obstacles are determined using the above method and the minimum bounding rectangles of the first and second initial obstacles corresponding to each target obstacle are determined, the coordinate information in the location information of the corresponding target obstacle can be determined based on the coordinates of the minimum bounding rectangle of the selected initial obstacle, and the distance information in the location information of the target obstacle can be determined based on the depth values of each pixel within the minimum bounding rectangle of the first initial obstacle corresponding to the target obstacle. If, after completing the comparison of the overlapping areas, the corresponding first initial obstacle is determined as the target obstacle, the location information of the target obstacle can be determined based on the first coordinates of the first minimum bounding rectangle of the first initial obstacle. Specifically, the first coordinates can be directly used as the coordinate information of the target obstacle, and the average depth value of all pixels within the first coordinate range can be calculated as the distance information of the target obstacle. If, after completing the comparison of the overlapping areas, the corresponding second initial obstacle is determined as the target obstacle, the distance information of the target obstacle can be determined based on the first coordinates of the first minimum bounding rectangle of the first initial obstacle. Specifically, the average depth value of all pixels within the first coordinate range can be calculated as the distance information of the target obstacle, and the coordinate information of the target obstacle can be determined based on the second coordinates of the second minimum bounding rectangle of the second initial obstacle. Specifically, the second coordinates can be directly used as the coordinate information of the target obstacle. Preferably, the coordinates identified in the height map, i.e., the second coordinates, are more accurate and trustworthy. In addition, the coordinates of the overlapping area between the first minimum bounding rectangle and the second minimum bounding rectangle can be determined based on the first coordinates and the second coordinates, and the coordinates of the overlapping area can be used as the coordinate information of the target obstacle. By using the first coordinates or the second coordinates to determine each position information, the position information of the identified target obstacle is more regular and can be more adapted to the existing obstacle avoidance control method, so that the robot can better judge the next way of moving and further ensure the effectiveness of obstacle avoidance.

[0062] Further, optionally, after determining the first initial obstacles in the depth map based on the first obstacle information, the method further includes: screening the first initial obstacles based on their first contour areas to remove first initial obstacles whose first contour areas are smaller than a first preset area threshold; and after determining the second initial obstacles in the height map based on the second obstacle information, the method further includes: screening the second initial obstacles based on their second contour areas to remove second initial obstacles whose second contour areas are smaller than a second preset area threshold. The contour area refers to the area contained within the corresponding contour. Specifically, after determining the first and second initial obstacles, smaller obstacles in the work scene can be eliminated based on the contour area of each initial obstacle. These obstacles may not affect the robot's movement, eliminating the need for obstacle avoidance, thereby reducing the amount of subsequent computation. For example, for weeds and pebbles on the ground, the first and second preset area thresholds can be set based on the size of these objects that do not actually affect the robot's movement. This means that obstacles exceeding the first or second preset area thresholds require obstacle avoidance. The identified initial obstacles can then be screened by comparing them with the first and second preset area thresholds. Specifically, first initial obstacles whose first contour area is smaller than the first preset area threshold can be screened out, and second initial obstacles whose second contour area is smaller than the second preset area threshold can be screened out. The first and second preset area thresholds can be set to the same area value.

[0063] S14. Control the robot to avoid obstacles according to the position information and the preset path of the robot.

[0064] Specifically, after determining the position information of each target obstacle, it is possible to judge whether each target obstacle affects the normal operation of the robot based on the position information and the preset path of the robot. If it does not affect it, it can continue to operate normally. If it does, it can avoid the obstacle.

[0065] The technical solution provided by the embodiment of the present invention first obtains a depth map of the work scene in the direction of the robot's movement through a binocular camera, and calculates a corresponding height map based on the depth map. Then, the first obstacle information in the work scene is determined based on the depth map, and the second obstacle information in the work scene is determined based on the height map. Then, based on the obtained first obstacle information and second obstacle information, the actual target obstacles in the work scene and their position information are determined. This allows the robot to avoid obstacles based on the position information of each target obstacle in combination with the robot's preset path. By combining the obstacle information obtained by respectively identifying the depth map and the height map to determine the final target obstacle and its position information, the environmental interference problem that exists in the recognition process using only the depth map is avoided, thereby improving the accuracy of identifying obstacles in front of the robot during movement and enabling the robot to effectively avoid obstacles.

[0066] Example 2

[0067] Figure 2 This is a schematic diagram of the structure of the intelligent obstacle avoidance device based on binocular vision provided in the second embodiment of the present invention. The device can be implemented by hardware and / or software and can generally be integrated into a robot to execute the intelligent obstacle avoidance method based on binocular vision provided in any embodiment of the present invention. Figure 2 As shown, the device includes:

[0068] The image acquisition module 21 is used to obtain a depth map of the working scene in the direction of the robot's movement through a binocular camera, and calculate a corresponding height map based on the depth map;

[0069] an obstacle information determination module 22, configured to determine first obstacle information in the operation scene according to the depth map, and to determine second obstacle information in the operation scene according to the height map;

[0070] an obstacle determination module 23, configured to determine a target obstacle existing in the operation scene and position information of the target obstacle based on the first obstacle information and the second obstacle information;

[0071] The obstacle avoidance control module 24 is used to control the robot to avoid obstacles based on the position information and the preset path of the robot.

[0072] The technical solution provided by the embodiment of the present invention first obtains a depth map of the work scene in the direction of the robot's movement through a binocular camera, and calculates a corresponding height map based on the depth map. Then, the first obstacle information in the work scene is determined based on the depth map, and the second obstacle information in the work scene is determined based on the height map. Then, based on the obtained first obstacle information and second obstacle information, the actual target obstacles in the work scene and their position information are determined. This allows the robot to avoid obstacles based on the position information of each target obstacle in combination with the robot's preset path. By combining the obstacle information obtained by respectively identifying the depth map and the height map to determine the final target obstacle and its position information, the environmental interference problem that exists in the recognition process using only the depth map is avoided, thereby improving the accuracy of identifying obstacles in front of the robot during movement and enabling the robot to effectively avoid obstacles.

[0073] Based on the above technical solution, optionally, the obstacle determination module 23 includes:

[0074] an initial obstacle determining unit, configured to determine a first initial obstacle in the depth map according to the first obstacle information, and to determine a second initial obstacle in the height map according to the second obstacle information;

[0075] an overlap ratio determining unit, configured to identify an overlap area between the first initial obstacle and the second initial obstacle, and calculate an overlap ratio between the first initial obstacle and the second initial obstacle based on the overlap area;

[0076] A target obstacle determination unit is configured to compare the overlap rate with a preset overlap rate threshold, and if the overlap rate is greater than or equal to the preset overlap rate threshold, determine the corresponding first initial obstacle or the second initial obstacle as the target obstacle.

[0077] Based on the above technical solution, optionally, the overlap rate determination unit includes:

[0078] a minimum bounding rectangle determining subunit, configured to extract a first minimum bounding rectangle of the first initial obstacle and a second minimum bounding rectangle of the second initial obstacle;

[0079] an overlapping area identification subunit, configured to identify the overlapping area according to the first minimum bounding rectangle and the second minimum bounding rectangle;

[0080] The overlap ratio calculation subunit is configured to calculate the overlap ratio according to the first coordinates of the first minimum bounding rectangle and the second coordinates of the second minimum bounding rectangle.

[0081] Based on the above technical solution, optionally, the obstacle determination module 23 further includes:

[0082] A position information determining unit is configured to determine the position information according to the first coordinates and / or the second coordinates corresponding to the target obstacle.

[0083] Based on the above technical solution, optionally, the obstacle determination module 23 further includes:

[0084] a first screening unit configured to, after determining the first initial obstacles in the depth map according to the first obstacle information, screen the first initial obstacles according to first contour areas of the first initial obstacles to remove first initial obstacles having first contour areas smaller than a first preset area threshold;

[0085] a second screening unit configured to, after determining the second initial obstacles in the height map according to the second obstacle information, screen the second initial obstacles according to their second contour areas, to remove second initial obstacles having second contour areas smaller than a second preset area threshold.

[0086] On the basis of the above technical solution, optionally, the image acquisition module 21 includes:

[0087] a point cloud data conversion unit, configured to convert the depth map into point cloud data of the operation scene;

[0088] A height map conversion unit is used to convert the point cloud data into the height map.

[0089] Based on the above technical solution, optionally, the obstacle information determination module 22 is specifically configured to:

[0090] Determine a first obstacle outline and coordinate information of the first obstacle outline in the depth map using a contour search algorithm or a clustering algorithm;

[0091] A contour search algorithm or a clustering algorithm is used to determine a second obstacle contour in the height map and coordinate information of the second obstacle contour.

[0092] The intelligent obstacle avoidance device based on binocular vision provided in the embodiment of the present invention can execute the intelligent obstacle avoidance method based on binocular vision provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0093] It is worth noting that in the above-mentioned embodiment of the intelligent obstacle avoidance device based on binocular vision, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0094] Example 3

[0095] Figure 3 The schematic structural diagram of the robot provided in the third embodiment of the present invention shows a block diagram of an exemplary robot suitable for implementing the embodiments of the present invention. Figure 3 The robot shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention. Figure 3 As shown, the robot includes a processor 31, a memory 32, an input device 33 and an output device 34; the number of processors 31 in the robot can be one or more. Figure 3 Taking a processor 31 as an example, the processor 31, memory 32, input device 33 and output device 34 in the robot can be connected through a bus or other means. Figure 3 The bus connection is taken as an example.

[0096] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the binocular vision-based intelligent obstacle avoidance method in the embodiments of the present invention (for example, the image acquisition module 21, obstacle information determination module 22, obstacle determination module 23, and obstacle avoidance control module 24 in the binocular vision-based intelligent obstacle avoidance device). The processor 31 executes the software programs, instructions, and modules stored in the memory 32 to execute various functional applications and data processing of the robot, thereby implementing the aforementioned binocular vision-based intelligent obstacle avoidance method.

[0097] The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the robot's use, etc. Furthermore, the memory 32 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 32 may further include memory remotely located relative to the processor 31, and such remote memory may be connected to the robot via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The input device 33 can be used to collect images of the working scene in front of the robot, and generate key signal input related to the user settings and function control of the robot, etc. The output device 34 can be used to control the movement of the robot, etc.

[0099] Example 4

[0100] Embodiment 4 of the present invention further provides a storage medium containing computer-executable instructions. When the computer-executable instructions are executed by a computer processor, the computer-executable instructions are used to execute an intelligent obstacle avoidance method based on binocular vision, the method comprising:

[0101] Obtain a depth map of the working scene in the direction of robot movement through a binocular camera, and calculate a corresponding height map based on the depth map;

[0102] Determining first obstacle information in the operation scene according to the depth map, and determining second obstacle information in the operation scene according to the height map;

[0103] determining a target obstacle existing in the operation scene and position information of the target obstacle according to the first obstacle information and the second obstacle information;

[0104] The robot is controlled to avoid obstacles according to the position information and the preset path of the robot.

[0105] The storage medium can be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system that is connected to the computer system via a network (such as the Internet). The second computer system may provide program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.

[0106] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the binocular vision-based intelligent obstacle avoidance method provided in any embodiment of the present invention.

[0107] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0109] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0110] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. An intelligent obstacle avoidance method based on binocular vision, characterized in that: include: Obtain a depth map of the working scene in the direction of robot movement through a binocular camera, and calculate a corresponding height map based on the depth map; Determining first obstacle information in the operation scene according to the depth map, and determining second obstacle information in the operation scene according to the height map; determining target obstacles existing in the operation scene and location information of the target obstacles based on the first obstacle information and the second obstacle information, so as to filter out obstacles that exist only in the first obstacle information or the second obstacle information; controlling the robot to avoid obstacles according to the position information and a preset path of the robot; The determining, based on the first obstacle information and the second obstacle information, a target obstacle existing in the operation scene and location information of the target obstacle includes: Determining a first initial obstacle in the depth map according to the first obstacle information, and determining a second initial obstacle in the height map according to the second obstacle information; Identifying an overlapping area between the first initial obstacle and the second initial obstacle, and calculating an overlapping rate between the first initial obstacle and the second initial obstacle based on the overlapping area; The overlap rate is compared with a preset overlap rate threshold, and if the overlap rate is greater than or equal to the preset overlap rate threshold, the corresponding first initial obstacle or the second initial obstacle is determined as the target obstacle.

2. The intelligent obstacle avoidance method based on binocular vision according to claim 1, characterized in that: The identifying an overlapping area between the first initial obstacle and the second initial obstacle, and calculating an overlapping rate between the first initial obstacle and the second initial obstacle based on the overlapping area, includes: Extracting a first minimum bounding rectangle of the first initial obstacle and a second minimum bounding rectangle of the second initial obstacle; Identify the overlapping area according to the first minimum enclosing rectangle and the second minimum enclosing rectangle; The overlap ratio is calculated according to the first coordinates of the first minimum bounding rectangle and the second coordinates of the second minimum bounding rectangle.

3. The intelligent obstacle avoidance method based on binocular vision according to claim 2, characterized in that: The determining of a target obstacle existing in the operation scene and position information of the target obstacle according to the first obstacle information and the second obstacle information further includes: The position information is determined according to the first coordinates and / or the second coordinates corresponding to the target obstacle.

4. The intelligent obstacle avoidance method based on binocular vision according to claim 1, characterized in that: After determining the first initial obstacle in the depth map according to the first obstacle information, the method further includes: screening the first initial obstacles according to the first contour areas of the first initial obstacles to remove the first initial obstacles whose first contour areas are smaller than a first preset area threshold; After determining the second initial obstacle in the height map according to the second obstacle information, the method further includes: The second initial obstacles are screened according to the second contour areas of the second initial obstacles to remove the second initial obstacles whose second contour areas are smaller than a second preset area threshold.

5. The intelligent obstacle avoidance method based on binocular vision according to claim 1, characterized in that: The calculating a corresponding height map according to the depth map includes: Converting the depth map into point cloud data of the operation scene; The point cloud data is converted into the height map.

6. The intelligent obstacle avoidance method based on binocular vision according to claim 1, characterized in that: The determining first obstacle information in the operation scene according to the depth map, and determining second obstacle information in the operation scene according to the height map, includes: Determine a first obstacle outline and coordinate information of the first obstacle outline in the depth map using a contour search algorithm or a clustering algorithm; A contour search algorithm or a clustering algorithm is used to determine a second obstacle contour in the height map and coordinate information of the second obstacle contour.

7. An intelligent obstacle avoidance device based on binocular vision, characterized in that: include: An image acquisition module is used to obtain a depth map of the working scene in the direction of the robot's movement through a binocular camera, and calculate a corresponding height map based on the depth map; an obstacle information determination module, configured to determine first obstacle information in the operation scene according to the depth map, and to determine second obstacle information in the operation scene according to the height map; an obstacle determination module, configured to determine target obstacles present in the operation scene and position information of the target obstacles based on the first obstacle information and the second obstacle information, so as to filter out obstacles that only exist in the first obstacle information or the second obstacle information; an obstacle avoidance control module, configured to control the robot to avoid obstacles based on the position information and a preset path of the robot; The obstacle determination module includes: an initial obstacle determining unit, configured to determine a first initial obstacle in the depth map according to the first obstacle information, and to determine a second initial obstacle in the height map according to the second obstacle information; an overlap ratio determining unit, configured to identify an overlap area between the first initial obstacle and the second initial obstacle, and calculate an overlap ratio between the first initial obstacle and the second initial obstacle based on the overlap area; A target obstacle determination unit is configured to compare the overlap rate with a preset overlap rate threshold, and if the overlap rate is greater than or equal to the preset overlap rate threshold, determine the corresponding first initial obstacle or the second initial obstacle as the target obstacle.

8. A robot, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the binocular vision-based intelligent obstacle avoidance method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the intelligent obstacle avoidance method based on binocular vision as described in any one of claims 1 to 6 is implemented.

Images