Obstacle detection methods, devices, electronic equipment and machine-readable storage media
By acquiring binocular depth images of obstacles using a binocular camera and combining disparity values and depth information, the problem of false obstacle detection caused by repetitive textures is solved, improving detection reliability and reducing costs, while avoiding sensor fusion processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIKROBOT TECH CO LTD
- Filing Date
- 2023-02-21
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, mobile robots are easily affected by repetitive textures when detecting obstacles, resulting in a high probability of false detection. Furthermore, additional sensors such as LiDAR or ultrasonic sensors are required for fusion processing, which increases costs.
A binocular camera is used to acquire binocular depth images. By extracting the obstacle contour, the ROI region of the obstacle in the grayscale images of the left and right eyes is determined. By combining the disparity value and depth information, the credibility of the obstacle is judged, the probability of false detection is reduced, and sensor fusion processing is avoided.
It improves the reliability of obstacle detection, reduces the probability of false detection, and reduces detection costs, without relying on lidar or ultrasonic sensors.
Smart Images
Figure CN116188985B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of visual perception, and more particularly to an obstacle detection method, apparatus, electronic device, and machine-readable storage medium. Background Technology
[0002] Obstacle avoidance refers to the process by which a mobile robot, while moving, uses sensors to detect static or dynamic obstacles on its planned route, updates its path in real time according to a certain algorithm, bypasses the obstacles, and finally reaches the target point.
[0003] To achieve obstacle avoidance and ensure the safety of mobile robots, it is necessary to accurately detect obstacles in the path of the mobile robot. Summary of the Invention
[0004] In view of this, this application provides an obstacle detection method, apparatus, electronic device, and machine-readable storage medium.
[0005] According to a first aspect of the embodiments of this application, an obstacle detection method is provided, comprising:
[0006] Binocular depth images are acquired using binocular cameras deployed on mobile robots;
[0007] Obstacle contours are extracted based on the binocular depth images to obtain initial obstacle contours;
[0008] For any obstacle outline, determine the region corresponding to the obstacle outline, namely the first obstacle ROI region in the first grayscale image and the second obstacle ROI region in the second grayscale image; wherein, the first grayscale image is the left-eye grayscale image and the second grayscale image is the right-eye grayscale image; or, the first grayscale image is the right-eye grayscale image and the second grayscale image is the left-eye grayscale image.
[0009] Based on the first obstacle ROI region and the second obstacle ROI region, determine the first disparity value of the obstacle corresponding to the obstacle outline;
[0010] Based on the first disparity value and the second disparity value, it is determined whether the obstacle corresponding to the obstacle outline is a credible obstacle; wherein, the second disparity value is determined based on the depth information of the obstacle outline.
[0011] According to a second aspect of the embodiments of this application, an obstacle detection device is provided, comprising:
[0012] The acquisition unit is used to acquire binocular depth images through a binocular camera deployed on a mobile robot;
[0013] An extraction unit is used to extract obstacle contours based on the binocular depth image to obtain an initial obstacle contour.
[0014] The first determining unit is configured to, for any obstacle contour, determine the region corresponding to the obstacle contour as a first obstacle ROI region in a first grayscale image and a second obstacle ROI region in a second grayscale image; wherein, the first grayscale image is a left-eye grayscale image and the second grayscale image is a right-eye grayscale image; or, the first grayscale image is a right-eye grayscale image and the second grayscale image is a left-eye grayscale image.
[0015] The second determining unit is used to determine the first disparity value of the obstacle corresponding to the obstacle outline based on the first obstacle ROI region and the second obstacle ROI region;
[0016] The third determining unit is used to determine whether the obstacle corresponding to the obstacle outline is a credible obstacle based on the first disparity value and the second disparity value; wherein the second disparity value is determined based on the depth information of the obstacle outline.
[0017] According to a third aspect of the present application, an electronic device is provided, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.
[0018] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.
[0019] The obstacle detection method of this application embodiment acquires binocular depth images using a binocular camera deployed on a mobile robot; obstacle contours are extracted based on the binocular depth images to obtain initial obstacle contours; for any obstacle contour, on the one hand, the region corresponding to the obstacle is determined as a first obstacle ROI region in a first grayscale image and a second obstacle ROI region in a second grayscale image, and a first disparity value of the obstacle corresponding to the obstacle contour is determined based on the first obstacle ROI region and the second obstacle ROI region; on the other hand, a second disparity value of the obstacle corresponding to the obstacle contour is determined based on the depth information of the obstacle contour; furthermore, based on the first disparity value and the second disparity value, it is determined whether the obstacle corresponding to the obstacle contour is a credible obstacle, reducing the probability of false obstacle detection due to repetitive textures and improving the reliability of obstacle detection; in addition, it does not require fusion processing of sensors such as LiDAR and ultrasonic sensors, reducing the implementation cost of obstacle detection. Attached Figure Description
[0020] Figure 1This is a schematic flowchart of an obstacle detection method provided in an embodiment of this application;
[0021] Figure 2 This is a schematic diagram of a process for determining the region corresponding to the outline of an obstacle in a first grayscale image and the region corresponding to the second obstacle in a second grayscale image, according to an embodiment of this application.
[0022] Figure 3 This is a schematic diagram of the region to be verified in a third obstacle ROI region provided in an embodiment of this application;
[0023] Figure 4 This is a schematic flowchart of an obstacle detection method provided in an embodiment of this application;
[0024] Figure 5 This is a schematic diagram of the structure of an obstacle detection device provided in an embodiment of this application;
[0025] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0027] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.
[0028] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.
[0029] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0030] Please see Figure 1 This is a flowchart illustrating an obstacle detection method provided in an embodiment of this application. Figure 1 As shown, the obstacle detection method may include the following steps:
[0031] Step S100: Acquire binocular depth images using binocular cameras deployed on the mobile robot.
[0032] Step S110: Extract obstacle contours based on binocular depth images to obtain initial obstacle contours.
[0033] In this embodiment of the application, in order to achieve obstacle detection, a binocular camera can be deployed on a mobile robot, and a binocular depth image can be acquired through the binocular camera. Obstacle detection is then performed based on the acquired binocular depth image.
[0034] In this embodiment of the application, for the acquired binocular depth image, obstacle contours can be extracted based on the acquired binocular depth image to obtain the obstacle contours (hereinafter referred to as the initial obstacle contours).
[0035] Step S120: For any obstacle contour, determine the region corresponding to the obstacle contour, namely the first obstacle ROI region in the first grayscale image and the second obstacle ROI region in the second grayscale image; wherein, the first grayscale image is the left-eye grayscale image and the second grayscale image is the right-eye grayscale image; or, the first grayscale image is the right-eye grayscale image and the second grayscale image is the left-eye grayscale image.
[0036] Step S130: Based on the first obstacle ROI region and the second obstacle ROI region, determine the first parallax value of the obstacle corresponding to the obstacle outline.
[0037] Step S140: Determine whether the obstacle corresponding to the obstacle outline is a credible obstacle based on the first disparity value and the second disparity value; wherein, the second disparity value is determined based on the depth information of the obstacle outline.
[0038] In this embodiment of the application, it is considered that in real-world scenarios, the binocular depth image may be abnormal due to repeated textures, which may lead to the existence of unreliable obstacles (i.e., obstacles that do not actually exist) in the obstacles corresponding to the initial obstacle outline.
[0039] Accordingly, in order to improve the reliability of obstacle detection, after obtaining the initial obstacle contour in the above manner, for any obstacle contour, on the one hand, the obstacle ROI (Region of Interest) region (which can be referred to as the first obstacle ROI region and the second obstacle ROI region, respectively) corresponding to the region of the obstacle contour can be determined in the left / right eye grayscale images of the binocular camera (which can be referred to as the first grayscale image and the second grayscale image, respectively). Based on the first obstacle ROI region and the second obstacle ROI region, the disparity value of the obstacle corresponding to the obstacle contour (referred to as the first disparity value in this paper) is determined.
[0040] On the other hand, the disparity value of the obstacle profile (referred to as the second disparity value in this paper) can be determined based on the depth information of the obstacle profile.
[0041] Furthermore, based on the first and second disparity values, it can be determined whether the obstacle corresponding to the obstacle profile is a credible obstacle.
[0042] For example, if the difference between the first disparity value and the second disparity value exceeds a preset threshold (which may be called the sixth threshold), the obstacle corresponding to the obstacle outline can be determined as an untrusted obstacle; if the difference between the first disparity value and the second disparity value does not exceed the preset threshold, the obstacle corresponding to the obstacle outline can be determined as a trustworthy obstacle.
[0043] It should be noted that, in the embodiments of this application, when credible obstacles are determined, the outline of each credible obstacle can be transformed into the polar coordinate system of the mobile robot. Then, based on the coordinates of the obstacle outline in the polar coordinate system of the mobile robot, the travel route of the mobile robot is planned to avoid obstacles and improve the safety of the mobile robot.
[0044] It can be seen that, in Figure 1In the illustrated method, a binocular depth image is acquired using a binocular camera deployed on a mobile robot. Obstacle contours are extracted based on the binocular depth image to obtain initial obstacle contours. For any obstacle contour, on one hand, the corresponding region is determined as a first obstacle ROI region in the first grayscale image and a second obstacle ROI region in the second grayscale image. Based on the first and second obstacle ROI regions, a first disparity value is determined for the obstacle corresponding to the contour. On the other hand, based on the depth information of the obstacle contour, a second disparity value is determined. Then, based on the first and second disparity values, it is determined whether the obstacle corresponding to the contour is a credible obstacle. This reduces the probability of false detection due to repetitive textures and improves the reliability of obstacle detection. Furthermore, it eliminates the need for fusion processing using sensors such as LiDAR and ultrasonic sensors, reducing the implementation cost of obstacle detection.
[0045] In some embodiments, the above-mentioned obstacle contour extraction based on binocular depth images to obtain an initial obstacle contour may include:
[0046] Convert binocular depth images into 3D point clouds;
[0047] Based on the installation position of the binocular camera, the 3D point cloud is converted into a raster image;
[0048] Obstacle contours are extracted from the grid image to obtain the initial obstacle contours.
[0049] For example, in order to extract obstacle contours, the acquired binocular depth image can be converted into a 3D point cloud.
[0050] Considering that the complexity of 3D data processing is higher than that of 2D data processing, in order to reduce the complexity of data processing and improve the efficiency of data processing, the 3D point cloud can be converted into a raster map according to the installation position of the binocular camera. Then, the obstacle contour can be extracted from the raster map to obtain the initial obstacle contour.
[0051] It should be noted that, in the embodiments of this application, the acquired binocular depth image can also be converted into a 3D point cloud and the obstacle contour can be directly extracted from the 3D point cloud to obtain the initial obstacle contour. However, the obstacle contour extraction efficiency of this implementation method is lower than that of the above-mentioned implementation method of extracting obstacle contour from the grid image.
[0052] In one example, after extracting the obstacle contours from the grid image to obtain the initial obstacle contours, the process may further include:
[0053] Extract the cluster segments corresponding to the obstacles based on the initial obstacle contours;
[0054] Cluster segments with fewer than a first threshold number of points are identified as untrusted cluster segments, and the corresponding obstacles are identified as untrusted obstacles.
[0055] For example, considering that in real-world scenarios, there may be cases where flying points are mistakenly detected as obstacles, and considering that the number of flying points is relatively small compared to the number of points corresponding to real obstacles in the image, in order to avoid the influence of flying points on obstacle detection and improve the reliability of obstacle detection, after the initial obstacle contour is extracted in the above manner, the cluster segments corresponding to the obstacles can be extracted based on the initial obstacle contour, and the number of points included in each cluster segment can be counted. Cluster segments with fewer than a preset threshold (which can be called the first threshold) are identified as untrusted cluster segments (which can be called the first type of cluster segment), and the corresponding obstacles are identified as untrusted obstacles.
[0056] Accordingly, for the remaining cluster segments (which can be referred to as the second type of cluster segments) other than the first type of cluster segments, they can be further verified based on disparity. That is, on the one hand, based on the obstacle ROI region (i.e., the first obstacle ROI region mentioned above) in the first grayscale image and the obstacle ROI region (i.e., the second obstacle ROI region mentioned above) in the second grayscale image corresponding to the region of the second type of cluster segment, the disparity value of the obstacle corresponding to the second type of cluster segment (i.e., the first disparity value mentioned above) is determined. On the other hand, based on the depth information of the second cluster segment, the disparity value of the obstacle corresponding to the second type of cluster segment (i.e., the second disparity value mentioned above) is determined. Based on the first disparity value and the second disparity value, it is determined whether the obstacle corresponding to the second type of cluster segment is a credible obstacle.
[0057] In some embodiments, such as Figure 2 As shown, in step S120, determining the region corresponding to the obstacle outline—the first obstacle ROI region in the first grayscale image and the second obstacle ROI region in the second grayscale image—can be achieved through the following steps:
[0058] Step S121: Project the region corresponding to the obstacle outline onto the first grayscale image to obtain the first obstacle ROI region corresponding to the region corresponding to the obstacle outline in the first grayscale image;
[0059] Step S122: Using the phase correlation method or template matching method, determine the matching area of the first obstacle ROI region in the second grayscale image, and determine the matching area as the second obstacle ROI region.
[0060] For example, for any obstacle contour, the region corresponding to the obstacle contour can be projected onto a first grayscale image to obtain the first obstacle ROI region corresponding to the region corresponding to the obstacle contour in the first grayscale image.
[0061] In one example, given that the region corresponding to the obstacle outline is determined as the first obstacle ROI region in the first grayscale image, the phase correlation method can be used to determine the matching region of the first obstacle ROI region in the second grayscale image, and this matching region can be identified as the second obstacle ROI region.
[0062] In another example, if the region corresponding to the obstacle outline is determined to be the first obstacle ROI region in the first grayscale image, a template matching method can be used to determine the matching region of the first obstacle ROI region in the second grayscale image, and the matching region is determined as the second obstacle ROI region.
[0063] In some embodiments, after extracting the obstacle contour based on the binocular depth image to obtain the initial obstacle contour, the method may further include:
[0064] For any obstacle contour, determine the third obstacle ROI region corresponding to the region of the obstacle contour in the specified grayscale image; wherein, the specified grayscale image is either the first grayscale image or the second grayscale image;
[0065] The region to be verified is determined based on the ROI region of the third obstacle, and the obstacle corresponding to the outline of the obstacle is determined as a credible obstacle based on the pixel grayscale value of the region to be verified.
[0066] For example, considering that in real-world scenarios, depth map anomalies may occur due to image overexposure, which could lead to unreliable obstacles among the obstacles corresponding to the initial obstacle outline.
[0067] Accordingly, in order to improve the reliability of obstacle detection, after obtaining the initial obstacle contour in the manner described above, for any obstacle contour, the obstacle ROI region (which can be called the third obstacle ROI region) corresponding to the region of the obstacle contour in a specified grayscale image (such as the first grayscale image or the second grayscale image described above) can also be determined.
[0068] It should be noted that if the grayscale image is specified as the first grayscale image, then the third obstacle ROI region can be the aforementioned first obstacle ROI region; if the grayscale image is specified as the second grayscale image, then the third obstacle ROI region can be the aforementioned second obstacle ROI region.
[0069] For example, the area to be verified can be determined based on the third obstacle ROI region, and the obstacle corresponding to the obstacle contour can be determined as a credible obstacle based on the pixel grayscale value of the area to be verified.
[0070] For example, it can be determined whether the area to be verified is overexposed based on the pixel grayscale value of the area to be verified, thereby determining whether the obstacle corresponding to the obstacle outline is a credible obstacle.
[0071] For example, if the area to be verified is determined to be overexposed, the obstacle corresponding to the obstacle's outline is determined to be an untrusted obstacle; otherwise, the obstacle corresponding to the obstacle's outline is determined to be a trustworthy obstacle.
[0072] It should be noted that, in the embodiments of this application, the above-mentioned methods of verifying whether an obstacle is a credible obstacle based on disparity and the methods of verifying whether an obstacle is a credible obstacle based on the pixel grayscale value of the verification area can be performed concurrently. That is, for any obstacle, the method of verifying whether an obstacle is a credible obstacle based on disparity and the method of verifying whether an obstacle is a credible obstacle based on the pixel grayscale value of the verification area can be performed separately. If the obstacle is determined to be a credible obstacle under both methods, then the obstacle can be determined to be a credible obstacle; otherwise, the obstacle can be determined to be an untrustworthy obstacle.
[0073] Alternatively, the above-mentioned methods for verifying whether an obstacle is a credible obstacle based on disparity and for verifying whether an obstacle is a credible obstacle based on the pixel grayscale value of the verification area can be performed sequentially.
[0074] For example, for any obstacle, we can first check whether the obstacle is a credible obstacle based on the parallax. If it is determined to be a credible obstacle, we can further check whether the obstacle is a credible obstacle based on the pixel grayscale value of the check area. If the obstacle is determined to be an untrustworthy obstacle based on the parallax check, then we do not need to check the pixel grayscale value again.
[0075] Alternatively, for any obstacle, we can first verify whether the obstacle is a credible obstacle based on the pixel grayscale value of the verification area; if it is determined to be a credible obstacle, we can further verify whether the obstacle is a credible obstacle based on the parallax; if the obstacle is determined to be an untrustworthy obstacle based on the pixel grayscale value of the verification area, then we do not need to perform the parallax verification again.
[0076] In one example, determining the region to be verified based on the ROI region of a third obstacle can include:
[0077] If the size of the third obstacle ROI region meets the preset conditions, the third obstacle ROI region is expanded to obtain the corresponding region to be verified.
[0078] If the size of the third obstacle ROI region does not meet the preset conditions, the third obstacle ROI region will be identified as the region to be verified.
[0079] The size of the ROI region of the third obstacle must meet one or more of the following preset conditions:
[0080] The width of the third obstacle ROI region is less than the second threshold, the height of the third obstacle ROI region is less than the third threshold, and the area of the third obstacle ROI region is less than the fourth threshold.
[0081] For example, considering that the alignment between the obstacle ROI region and the depth map is not particularly accurate, there will be a certain deviation between the true exposure area and the obstacle ROI region, and the smaller the obstacle ROI region, the greater the impact will be.
[0082] To perform obstacle detection more accurately, for the identified third obstacle ROI region, it can be determined whether it meets the preset conditions based on the size of the third obstacle ROI region.
[0083] For example, the size of the ROI region of the third obstacle meets one or more of the following preset conditions:
[0084] The width of the third obstacle ROI region is less than the second threshold, the height of the third obstacle ROI region is less than the third threshold, and the area of the third obstacle ROI region is less than the fourth threshold.
[0085] For example, if the size of the third obstacle ROI region meets the preset conditions, the third obstacle ROI region can be expanded to obtain the corresponding region to be verified.
[0086] For example, the ROI region of the third obstacle can be expanded to include the 8-neighborhood of the ROI region and the ROI region itself as the region to be verified.
[0087] The third obstacle ROI region includes eight neighborhoods of the same size as the third obstacle ROI region: directly to its left, right, above, below, upper left, upper right, lower left, and lower right. A schematic diagram of this region can be seen as follows: Figure 3 As shown.
[0088] For example, if the size of the third obstacle ROI region does not meet the preset conditions, it is not necessary to expand the third obstacle ROI region, and the third obstacle ROI region can be determined as the region to be verified.
[0089] In one example, determining whether an obstacle corresponding to the outline of an obstacle is a credible obstacle based on the pixel grayscale values of the area to be verified can include:
[0090] Based on the pixel grayscale values of the area to be verified, determine the grayscale statistics of the area to be verified; the grayscale statistics include part or all of the average grayscale value of the area to be verified, the number of exposed pixels, and the proportion of exposed pixels, where exposed pixels are pixels with grayscale values greater than the fifth threshold.
[0091] Based on the grayscale statistical values of the area to be verified, determine whether the obstacle corresponding to the obstacle outline is a credible obstacle.
[0092] For example, in order to determine whether the area to be verified is overexposed, the grayscale statistics of the area to be verified can be determined based on the pixel grayscale values of the area to be verified.
[0093] For example, grayscale statistics may include the average grayscale value of the area to be verified, the number of exposed pixels, and some or all of the percentage of exposed pixels.
[0094] Among them, the exposed pixel is the pixel value whose grayscale value is greater than the preset threshold (which can be called the fifth threshold).
[0095] The exposure pixel ratio is the ratio of the number of exposed pixels in the area to be verified to the total number of pixels in the area to be verified.
[0096] For example, the grayscale statistical value of the area to be verified can be used to determine whether the obstacle corresponding to the obstacle outline is a credible obstacle.
[0097] For example, if the grayscale statistics of the area to be verified meet one or more of the following conditions, the obstacle corresponding to the obstacle profile can be determined as an untrusted obstacle:
[0098] The average grayscale value of the area to be verified is greater than the seventh threshold, the number of exposed pixels is greater than the eighth threshold, and the proportion of exposed pixels is greater than the ninth threshold.
[0099] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.
[0100] In this embodiment, in order to achieve obstacle detection, a binocular camera deployed on a mobile robot can be used to acquire binocular depth images, and the acquired binocular depth images can be converted into 3D point clouds. The coordinate transformation of the 3D point clouds can be performed according to the installation position of the binocular camera, and the point clouds can be converted into a grid map. The obstacle contour is extracted from the obtained grid map to obtain the initial obstacle contour.
[0101] Once the initial obstacle outlines are obtained, the corresponding cluster segments can be extracted based on each initial obstacle outline, and cluster segments with fewer than a first threshold are identified as untrusted cluster segments.
[0102] After excluding untrusted cluster segments, any cluster segment can be subjected to disparity verification and grayscale verification respectively.
[0103] For example, the disparity verification process may include: for any cluster segment, on the one hand, the region corresponding to the cluster segment can be projected onto the left-eye grayscale image (i.e., taking the first grayscale image mentioned above as the left-eye grayscale image as an example) to obtain the obstacle ROI region in the left-eye grayscale image (which can be called the left-eye obstacle ROI region, i.e. the first obstacle ROI region mentioned above). Through phase correlation method and template matching, the matching region of the left-eye obstacle ROI region in the right-eye grayscale image is determined to obtain the obstacle ROI region in the right-eye grayscale image (which can be called the right-eye obstacle ROI region, i.e. the second obstacle ROI region mentioned above). Then, based on the left-eye obstacle ROI region and the right-eye obstacle ROI region, the disparity value of the obstacle corresponding to the cluster segment (i.e., the first disparity mentioned above) is calculated. On the other hand, based on the depth information of the cluster segment, the disparity value of the obstacle corresponding to the cluster segment (i.e., the second disparity value mentioned above) can be determined. Then, the first disparity value and the second disparity value are compared. If the difference between the two exceeds a preset threshold (i.e., the ninth threshold mentioned above), the obstacle corresponding to the cluster segment can be determined to be an untrusted obstacle.
[0104] The grayscale verification process may include: for any cluster segment, the region corresponding to the cluster segment can be projected onto the left-eye grayscale image (i.e., taking the first grayscale image above as the left-eye grayscale image as an example) to obtain the obstacle ROI region in the left-eye grayscale image (i.e., the third obstacle ROI region above).
[0105] If the width and height of the third obstacle ROI region are both less than 40 pixels (i.e., taking the third obstacle ROI region as an example where the width and height of the third obstacle ROI region are both less than 40 pixels), then the third obstacle ROI region can be expanded, and the 8-neighborhood of the third obstacle ROI region and the third obstacle ROI region itself can be determined as the region to be verified.
[0106] If the width and / or height of the third obstacle ROI region is greater than or equal to 40 pixels, then the third obstacle ROI region is determined as the region to be verified.
[0107] Once the area to be verified is determined, the average grayscale value, the number of exposed pixels, and the percentage of exposed pixels in the area can be statistically analyzed.
[0108] For example, the exposed pixel can be a pixel with a grayscale value greater than 220 (i.e., taking the fifth threshold of 220 mentioned above as an example).
[0109] For example, if the average gray value of the area to be verified exceeds the sixth threshold, the number of exposed pixels exceeds the seventh threshold, and / or the proportion of exposed pixels exceeds the eighth threshold, the obstacle corresponding to the cluster segment can be determined as an untrusted obstacle.
[0110] In this embodiment, after eliminating untrusted obstacles through parallax and grayscale verification, for trustworthy obstacles, the obstacle contours can be transformed into the polar coordinate system of the mobile robot. Then, based on the coordinates of the obstacle contours in the mobile robot's polar coordinate system, the mobile robot's path is planned to avoid obstacles and improve its safety. A schematic diagram of this process is shown below. Figure 4 As shown.
[0111] The method provided in this application has been described above. The apparatus provided in this application is described below:
[0112] Please see Figure 5 This is a schematic diagram of the structure of an obstacle detection device provided in an embodiment of this application, as shown below. Figure 5 As shown, the depth data acquisition device may include:
[0113] The acquisition unit 510 is used to acquire binocular depth images through a binocular camera deployed on the mobile robot;
[0114] Extraction unit 520 is used to extract obstacle contours based on the binocular depth image to obtain an initial obstacle contour;
[0115] The first determining unit 530 is used to determine, for any obstacle contour, the region corresponding to the obstacle contour is a first obstacle ROI region in a first grayscale image and a second obstacle ROI region in a second grayscale image; wherein, the first grayscale image is a left-eye grayscale image and the second grayscale image is a right-eye grayscale image; or, the first grayscale image is a right-eye grayscale image and the second grayscale image is a left-eye grayscale image.
[0116] The second determining unit 540 is used to determine the first disparity value of the obstacle corresponding to the obstacle outline based on the first obstacle ROI region and the second obstacle ROI region.
[0117] The third determining unit 550 is used to determine whether the obstacle corresponding to the obstacle outline is a credible obstacle based on the first disparity value and the second disparity value; wherein the second disparity value is determined based on the depth information of the obstacle outline.
[0118] In some embodiments, the extraction unit 520 extracts obstacle contours based on the binocular depth image to obtain an initial obstacle contour, including:
[0119] Convert the binocular depth image into a 3D point cloud;
[0120] Based on the installation position of the binocular camera, the 3D point cloud is converted into a raster image;
[0121] Obstacle contours are extracted from the grid image to obtain initial obstacle contours.
[0122] In some embodiments, after the extraction unit 520 extracts obstacle contours from the grid image to obtain initial obstacle contours, it further includes:
[0123] Based on the initial obstacle contour, extract the cluster segments corresponding to the obstacles;
[0124] Cluster segments with fewer than a first threshold number of points are identified as untrusted cluster segments, and the corresponding obstacles are identified as untrusted obstacles.
[0125] In some embodiments, the first determining unit 530 determines the region corresponding to the obstacle contour as a first obstacle ROI region in the first grayscale image and a second obstacle ROI region in the second grayscale image, including:
[0126] Project the region corresponding to the obstacle outline onto the first grayscale image to obtain the first obstacle ROI region corresponding to the region corresponding to the obstacle outline in the first grayscale image;
[0127] Using phase correlation or template matching methods, the matching region of the first obstacle ROI region in the second grayscale image is determined, and the matching region is determined as the second obstacle ROI region.
[0128] In some embodiments, the third determining unit 550 is further configured to, for any obstacle contour, determine a third obstacle ROI region corresponding to the region corresponding to the obstacle contour in a specified grayscale image; wherein, the specified grayscale image is the first grayscale image or the second grayscale image; determine the region to be verified based on the third obstacle ROI region, and determine whether the obstacle corresponding to the obstacle contour is a credible obstacle based on the pixel grayscale value of the region to be verified.
[0129] In some embodiments, the third determining unit 550 determines the region to be verified based on the third obstacle ROI region, including:
[0130] If the size of the third obstacle ROI region meets the preset conditions, the third obstacle ROI region is expanded to obtain the corresponding region to be verified.
[0131] If the size of the third obstacle ROI region does not meet the preset conditions, the third obstacle ROI region is determined as the region to be verified.
[0132] In some embodiments, the size of the third obstacle ROI region satisfies one or more of the following preset conditions:
[0133] The width of the third obstacle ROI region is less than the second threshold, the height of the third obstacle ROI region is less than the third threshold, and the area of the third obstacle ROI region is less than the fourth threshold.
[0134] In some embodiments, the third determining unit 550 expands the third obstacle ROI region to obtain a corresponding region to be verified, including:
[0135] The third obstacle ROI region is expanded, and the 8-neighborhood of the third obstacle ROI region and the third obstacle ROI region are determined as the region to be verified.
[0136] In some embodiments, the third determining unit 550 determines whether the obstacle corresponding to the obstacle contour is a credible obstacle based on the pixel grayscale value of the area to be verified, including:
[0137] Based on the pixel grayscale values of the area to be verified, determine the grayscale statistics of the area to be verified; the grayscale statistics include part or all of the average grayscale value of the area to be verified, the number of exposed pixels, and the proportion of exposed pixels, wherein the exposed pixels are pixels with grayscale values greater than the fifth threshold.
[0138] Based on the grayscale statistical values of the area to be verified, it is determined whether the obstacle corresponding to the obstacle outline is a credible obstacle.
[0139] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the obstacle detection method described above.
[0140] Please see Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing machine-executable instructions. The processor 601 and the memory 602 can communicate via a system bus 603. Furthermore, by reading and executing the machine-executable instructions corresponding to obstacle detection logic in the memory 602, the processor 601 can execute the obstacle detection method described above.
[0141] The memory 602 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.
[0142] In some embodiments, a machine-readable storage medium, such as Figure 6 The memory 602 in the memory, which is a machine-readable storage medium, stores machine-executable instructions that, when executed by a processor, implement the obstacle detection method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.
[0143] It should be noted that, in this document, relational terms such as "objective" and "target" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0144] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. An obstacle detection method, characterized in that, include: Binocular depth images are acquired using binocular cameras deployed on a mobile robot, and these binocular depth images are used for obstacle detection. Obstacle contours are extracted based on the binocular depth images to obtain initial obstacle contours; For any obstacle outline, determine the region corresponding to the obstacle outline, namely the first obstacle ROI region in the first grayscale image and the second obstacle ROI region in the second grayscale image; wherein, the first grayscale image is the left-eye grayscale image and the second grayscale image is the right-eye grayscale image; or, the first grayscale image is the right-eye grayscale image and the second grayscale image is the left-eye grayscale image. Based on the first obstacle ROI region and the second obstacle ROI region, determine the first disparity value of the obstacle corresponding to the obstacle outline; Based on the first disparity value and the second disparity value, it is determined whether the obstacle corresponding to the obstacle outline is a credible obstacle; wherein, the second disparity value is determined based on the depth information of the obstacle outline.
2. The method according to claim 1, characterized in that, The step of extracting obstacle contours based on the binocular depth image to obtain an initial obstacle contour includes: The binocular depth image is converted into a 3D point cloud; Based on the installation position of the binocular camera, the 3D point cloud is converted into a raster image; Obstacle contours are extracted from the grid image to obtain initial obstacle contours.
3. The method according to claim 2, characterized in that, After extracting obstacle contours from the grid image to obtain initial obstacle contours, the process further includes: Based on the initial obstacle contour, extract the cluster segments corresponding to the obstacles; Cluster segments with fewer than a first threshold are identified as untrusted cluster segments, and the corresponding obstacles are identified as untrusted obstacles.
4. The method according to claim 1, characterized in that, The process of determining the region corresponding to the obstacle contour, namely the first obstacle ROI region in the first grayscale image and the second obstacle ROI region in the second grayscale image, includes: Project the region corresponding to the obstacle outline onto the first grayscale image to obtain the first obstacle ROI region corresponding to the region corresponding to the obstacle outline in the first grayscale image; Using phase correlation or template matching methods, the matching region of the first obstacle ROI region in the second grayscale image is determined, and the matching region is determined as the second obstacle ROI region.
5. The method according to any one of claims 1-4, characterized in that, After extracting the obstacle contour based on the binocular depth image to obtain the initial obstacle contour, the process further includes: For any obstacle contour, determine the third obstacle ROI region corresponding to the region of the obstacle contour in a specified grayscale image; wherein, the specified grayscale image is the first grayscale image or the second grayscale image; The region to be verified is determined based on the third obstacle ROI region, and the obstacle corresponding to the obstacle outline is determined to be a credible obstacle based on the pixel grayscale value of the region to be verified.
6. The method according to claim 5, characterized in that, The step of determining the region to be verified based on the third obstacle ROI region includes: If the size of the third obstacle ROI region meets the preset conditions, the third obstacle ROI region is expanded to obtain the corresponding region to be verified. If the size of the third obstacle ROI region does not meet the preset conditions, the third obstacle ROI region is determined as the region to be verified. The size of the third obstacle ROI region must satisfy one or more of the following preset conditions: The width of the third obstacle ROI region is less than the second threshold, the height of the third obstacle ROI region is less than the third threshold, and the area of the third obstacle ROI region is less than the fourth threshold.
7. The method according to claim 6, characterized in that, The process of expanding the ROI region of the third obstacle to obtain the corresponding region to be verified includes: The third obstacle ROI region is expanded, and the 8-neighborhood of the third obstacle ROI region and the third obstacle ROI region are determined as the region to be verified.
8. The method according to claim 5, characterized in that, The step of determining whether an obstacle corresponding to the obstacle contour is a credible obstacle based on the pixel grayscale values of the area to be verified includes: Based on the pixel grayscale values of the area to be verified, determine the grayscale statistics of the area to be verified; the grayscale statistics include part or all of the average grayscale value of the area to be verified, the number of exposed pixels, and the proportion of exposed pixels, wherein the exposed pixels are pixels with grayscale values greater than the fifth threshold. Based on the grayscale statistical values of the area to be verified, it is determined whether the obstacle corresponding to the obstacle outline is a credible obstacle.
9. An obstacle detection device, characterized in that, include: The acquisition unit is used to acquire binocular depth images through a binocular camera deployed on a mobile robot, and the binocular depth images are used for obstacle detection; An extraction unit is used to extract obstacle contours based on the binocular depth image to obtain an initial obstacle contour. The first determining unit is configured to, for any obstacle contour, determine the region corresponding to the obstacle contour as a first obstacle ROI region in a first grayscale image and a second obstacle ROI region in a second grayscale image; wherein, the first grayscale image is a left-eye grayscale image and the second grayscale image is a right-eye grayscale image; or, the first grayscale image is a right-eye grayscale image and the second grayscale image is a left-eye grayscale image. The second determining unit is used to determine the first disparity value of the obstacle corresponding to the obstacle outline based on the first obstacle ROI region and the second obstacle ROI region; The third determining unit is used to determine whether the obstacle corresponding to the obstacle outline is a credible obstacle based on the first disparity value and the second disparity value; wherein the second disparity value is determined based on the depth information of the obstacle outline.
10. The apparatus according to claim 9, characterized in that, The extraction unit extracts obstacle contours based on the binocular depth image to obtain initial obstacle contours, including: The binocular depth image is converted into a 3D point cloud; Based on the installation position of the binocular camera, the 3D point cloud is converted into a raster image; Obstacle contours are extracted from the grid image to obtain initial obstacle contours; The extraction unit extracts obstacle contours from the grid image to obtain initial obstacle contours, and then further includes: Based on the initial obstacle contour, extract the cluster segments corresponding to the obstacles; Cluster segments with fewer than the first threshold number of points are identified as untrusted cluster segments, and the corresponding obstacles are identified as untrusted obstacles. And / or, The first determining unit determines the region corresponding to the obstacle contour as a first obstacle ROI region in the first grayscale image and a second obstacle ROI region in the second grayscale image, including: Project the region corresponding to the obstacle outline onto the first grayscale image to obtain the first obstacle ROI region corresponding to the region corresponding to the obstacle outline in the first grayscale image; Using phase correlation or template matching methods, the matching region of the first obstacle ROI region in the second grayscale image is determined, and the matching region is determined as the second obstacle ROI region; And / or, The third determining unit is further configured to, for any obstacle contour, determine the third obstacle ROI region corresponding to the region corresponding to the obstacle contour in a specified grayscale image; wherein, the specified grayscale image is the first grayscale image or the second grayscale image; determine the region to be verified based on the third obstacle ROI region, and determine whether the obstacle corresponding to the obstacle contour is a credible obstacle based on the pixel grayscale value of the region to be verified. The third determining unit determines the region to be verified based on the third obstacle ROI region, including: If the size of the third obstacle ROI region meets the preset conditions, the third obstacle ROI region is expanded to obtain the corresponding region to be verified. If the size of the third obstacle ROI region does not meet the preset conditions, the third obstacle ROI region is determined as the region to be verified. The size of the third obstacle ROI region must satisfy one or more of the following preset conditions: The width of the third obstacle ROI region is less than the second threshold, the height of the third obstacle ROI region is less than the third threshold, and the area of the third obstacle ROI region is less than the fourth threshold. The third determining unit expands the third obstacle ROI region to obtain the corresponding region to be verified, including: The third obstacle ROI region is expanded, and the 8-neighborhood of the third obstacle ROI region and the third obstacle ROI region are determined as the region to be verified; The third determining unit determines whether an obstacle corresponding to the obstacle contour is a credible obstacle based on the pixel grayscale values of the area to be verified, including: Based on the pixel grayscale values of the area to be verified, determine the grayscale statistics of the area to be verified; the grayscale statistics include part or all of the average grayscale value of the area to be verified, the number of exposed pixels, and the proportion of exposed pixels, wherein the exposed pixels are pixels with grayscale values greater than the fifth threshold. Based on the grayscale statistical values of the area to be verified, it is determined whether the obstacle corresponding to the obstacle outline is a credible obstacle.
11. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-8.
12. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-8.