Robot movement environment detection method, electronic device, and storage medium

Abstract

The application discloses a robot mobile environment detection method, an electronic device and a storage medium, and relates to the field of robots.The method comprises the following steps: acquiring depth point cloud data in a first preset range of the surrounding environment of the robot based on a depth camera; segmenting the depth point cloud data to obtain each super voxel; and determining a first obstacle region in each super voxel based on a preset height condition and a preset slope condition. That is, the depth point cloud data of the surrounding environment of the robot is acquired by using the depth camera, and the depth point cloud data is segmented into a plurality of super voxels. Compared with a traditional obstacle detection scheme in the environment, the detection object when detecting the obstacle is refined in the application. Meanwhile, the application further refines the obstacle detection condition to a height condition and a slope condition. Compared with a traditional scheme, the granularity of the obstacle detection condition is reduced, so that the accuracy of the environment obstacle detection result is improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method for detecting the mobile environment of a robot, an electronic device, and a storage medium. Background Technology

[0002] Service robots are often used in environments with risks of falling, such as stairs and ramps, and may encounter pedestrians and other obstacles of varying heights and shapes in their path. Currently, obstacle detection for robots typically relies on LiDAR to acquire environmental data. However, most solutions use relatively simple detection logic and produce coarse-grained results, making it difficult to accurately assess complex work scenarios and thus affecting the robot's actions. Summary of the Invention

[0003] The main purpose of this application is to provide a robot mobile environment detection method, electronic device and storage medium, which aims to solve the technical problem that the current environmental obstacle detection logic is simple and the detection results are inaccurate in complex working scenarios.

[0004] To achieve the above objectives, this application provides a robot motion environment detection method, wherein the robot is equipped with a depth camera, and the robot motion environment detection method includes the following steps: The depth point cloud data of the robot's surrounding environment within a first preset range is acquired based on the depth camera; The depth point cloud data is segmented to obtain each hypervoxel; The first obstacle region in each of the supervoxels is determined based on preset height and preset slope conditions.

[0005] Optionally, the preset height condition includes a first height threshold and a second height threshold, wherein the first height threshold is greater than the second height threshold; the preset slope condition includes a first slope threshold and a second slope threshold, wherein the first slope threshold is greater than the second slope threshold. The step of determining the first obstacle region in each of the supervoxels based on preset height and preset slope conditions includes: For any hypervoxel, the first relative ground height and the first relative ground slope of the hypervoxel are determined based on the target ground plane where the robot is located. Extend the hypervoxel plane to the location of the robot or above and below the location of the robot, and determine the drop value between the location of the robot and the hypervoxel after the plane is extended; Determine whether the first relative ground height, the first relative ground slope, and the drop value satisfy the conditions in the first preset obstacle area condition set, wherein the first preset obstacle area condition set includes: the first relative ground height is greater than a first height threshold and the first relative ground slope is greater than a first slope threshold; the first relative ground height is greater than a first height threshold and the drop value is greater than a first height threshold; the first relative ground height is less than a second height threshold and the first relative ground slope is less than a second slope threshold; and the first relative ground height is less than a second height threshold and the drop value is less than a first height threshold. If the conditions in the first preset obstacle region condition set are met, then the hypervoxel is determined to be the first obstacle region.

[0006] Optionally, after the step of acquiring depth point cloud data within a first preset range of the robot's surrounding environment based on the depth camera, the method further includes: If there are point cloud hole regions in the depth point cloud data, then the image data of the point cloud hole regions is acquired, and the continuity of the point cloud hole regions and their neighborhoods is determined based on the image data. If the point cloud cavity region and its surrounding area are continuous, then point cloud data filling is performed on the point cloud cavity region.

[0007] Optionally, prior to the step of determining the hypervoxel as the first barrier region, the method further includes: If the conditions in the first preset obstacle area condition set are met, then based on the preset horizontal ground plane of the robot's working environment, the second relative height of the super voxel and the second relative slope of the ground are determined. Determine whether the relative height of the second ground, the relative slope of the second ground, and the drop value meet the conditions in the second preset obstacle area condition set, wherein the second preset obstacle area condition set includes: the relative height of the second ground is greater than a first height threshold and the relative slope of the second ground is greater than a first slope threshold; the relative height of the second ground is greater than a first height threshold and the drop value is greater than a first height threshold; the relative height of the second ground is less than a second height threshold and the relative slope of the second ground is less than a second slope threshold; and the relative height of the second ground is less than a second height threshold and the drop value is less than a first height threshold. If the conditions in the second preset obstacle region condition set are met, then the step of determining the hypervoxel as the first obstacle region is executed.

[0008] Optionally, before the step of acquiring image data of the point cloud hole region, the method includes: Obtain abnormal point clouds from the depth point cloud data; If the proportion of abnormal point clouds in the total depth point cloud exceeds a preset threshold, it is determined that there are point cloud void regions in the depth point cloud data.

[0009] Optionally, the step of determining the continuity of the point cloud hole region and its neighborhood based on the image data includes: Connecting adjacent abnormal point clouds in the depth point cloud data yields a connected region; The point cloud cavity region is determined based on the connected region; A reference color value is determined based on the color of the corresponding region in the image data of the point cloud cavity region; The image data is segmented based on the reference color value to obtain segmented regions; If the point cloud hole region is located within a segmented region, the continuity of the point cloud hole region and its neighborhood is determined, wherein the segmented region where the point cloud hole region is located is used for point cloud data filling of the point cloud hole region.

[0010] Optionally, the robot is equipped with a lidar, the obstacle detection result of the lidar is a second obstacle area, the shooting direction of the depth camera is towards the ground area, and the shooting direction forms a preset angle with the extension surface of the depth camera on the robot; After the step of determining the first obstacle region in each of the supervoxels based on preset height and preset slope conditions, the method further includes: The first obstacle region and the second obstacle region are compared and deduplicated to remove the first obstacle region that is the same as the second obstacle region; The first obstacle region obtained after comparison and deduplication is added to the path obstacle set. After the robot crosses the first obstacle region, the crossed first obstacle region will be removed from the path obstacle set.

[0011] Optionally, the robot movement environment detection method further includes: If the robot cannot cross the first obstacle area, the insurmountable first obstacle area is added to the set of temporary constraints when the robot plans its path.

[0012] In addition, to achieve the above objectives, this application also provides an electronic device, which includes: a memory, a processor, and a robot mobile environment detection program stored in the memory and executable on the processor. When the robot mobile environment detection program is executed by the processor, it implements the steps of the robot mobile environment detection method described above.

[0013] In addition, to achieve the above objectives, this application also provides a storage medium storing a robot mobile environment detection program, which, when executed by a processor, implements the steps of the robot mobile environment detection method described above.

[0014] This application proposes a robot mobile environment detection method, electronic device, and storage medium. In this embodiment, depth point cloud data within a first preset range of the robot's surrounding environment is acquired using a depth camera; the depth point cloud data is segmented to obtain supervoxels; and a first obstacle region within each supervoxel is determined based on preset height and slope conditions. That is, depth point cloud data of the robot's surrounding environment is acquired using a depth camera, and then the depth point cloud data is segmented into multiple supervoxels. Compared to traditional methods that detect obstacles in the environment, this embodiment refines the detection targets when detecting obstacles. Furthermore, this embodiment refines the obstacle detection conditions into height and slope conditions, reducing the granularity of the obstacle detection conditions compared to traditional methods, thereby improving the accuracy of environmental obstacle detection results. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of this application; Figure 2 This is a flowchart illustrating the first embodiment of the robot mobile environment detection method of this application; Figure 3 This is a detailed flowchart of step S20 in the robot mobile environment detection method of this application; Figure 4 This is a flowchart illustrating the second embodiment of the robot mobile environment detection method of this application; Figure 5 This is a flowchart illustrating the third embodiment of the robot mobile environment detection method of this application; Figure 6 This is a schematic diagram of the robot detection scenario in the robot mobile environment detection method of this application.

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0018] like Figure 1 As shown, Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of this application.

[0019] The electronic devices in the embodiments of this application can be mobile robots or unmanned vehicles, or electronic terminal devices such as smartphones, PCs, tablets, and portable computers.

[0020] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0021] Optionally, the electronic device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, etc. The terminal may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, and infrared sensor, which will not be elaborated upon here. Those skilled in the art will understand that... Figure 1 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0022] Those skilled in the art will understand that Figure 1 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0023] In addition, such as Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a robot mobile environment detection program.

[0024] exist Figure 1 In the illustrated electronic device, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the user terminal (user end) and communicate data with it; while processor 1001 can be used to call the robot movement environment detection program stored in memory 1005. The robot is equipped with a depth camera and performs the following operations: The depth point cloud data of the robot's surrounding environment within a first preset range is acquired based on the depth camera; The depth point cloud data is segmented to obtain each hypervoxel; The first obstacle region in each of the supervoxels is determined based on preset height and preset slope conditions.

[0025] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: The preset height condition includes a first height threshold and a second height threshold, wherein the first height threshold is greater than the second height threshold; the preset slope condition includes a first slope threshold and a second slope threshold, wherein the first slope threshold is greater than the second slope threshold. The step of determining the first obstacle region in each of the supervoxels based on preset height and preset slope conditions includes: For any hypervoxel, the first relative ground height and the first relative ground slope of the hypervoxel are determined based on the target ground plane where the robot is located. Extend the hypervoxel plane to the location of the robot or above and below the location of the robot, and determine the drop value between the location of the robot and the hypervoxel after the plane is extended; Determine whether the first relative ground height, the first relative ground slope, and the drop value satisfy the conditions in the first preset obstacle area condition set, wherein the first preset obstacle area condition set includes: the first relative ground height is greater than a first height threshold and the first relative ground slope is greater than a first slope threshold; the first relative ground height is greater than a first height threshold and the drop value is greater than a first height threshold; the first relative ground height is less than a second height threshold and the first relative ground slope is less than a second slope threshold; and the first relative ground height is less than a second height threshold and the drop value is less than a first height threshold. If the conditions in the first preset obstacle region condition set are met, then the hypervoxel is determined to be the first obstacle region.

[0026] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: Prior to the step of determining the hypervoxel as the first barrier region, the method further includes: If the conditions in the first preset obstacle area condition set are met, then based on the preset horizontal ground plane of the robot's working environment, the second relative height of the super voxel and the second relative slope of the ground are determined. Determine whether the relative height of the second ground, the relative slope of the second ground, and the drop value meet the conditions in the second preset obstacle area condition set, wherein the second preset obstacle area condition set includes: the relative height of the second ground is greater than a first height threshold and the relative slope of the second ground is greater than a first slope threshold; the relative height of the second ground is greater than a first height threshold and the drop value is greater than a first height threshold; the relative height of the second ground is less than a second height threshold and the relative slope of the second ground is less than a second slope threshold; and the relative height of the second ground is less than a second height threshold and the drop value is less than a first height threshold. If the conditions in the second preset obstacle region condition set are met, then the step of determining the hypervoxel as the first obstacle region is executed.

[0027] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: After the step of acquiring depth point cloud data within a first preset range of the robot's surrounding environment based on the depth camera, the method further includes: If there are point cloud hole regions in the depth point cloud data, then the image data of the point cloud hole regions is acquired, and the continuity of the point cloud hole regions and their neighborhoods is determined based on the image data. If the point cloud cavity region and its surrounding area are continuous, then point cloud data filling is performed on the point cloud cavity region.

[0028] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: Prior to the step of acquiring image data of the point cloud hole region, the method includes: Obtain abnormal point clouds from the depth point cloud data; If the proportion of abnormal point clouds in the total depth point cloud exceeds a preset threshold, it is determined that there are point cloud void regions in the depth point cloud data.

[0029] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: The step of determining the continuity of the point cloud hole region and its neighborhood based on the image data includes: Connecting adjacent abnormal point clouds in the depth point cloud data yields a connected region; The point cloud cavity region is determined based on the connected region; A reference color value is determined based on the color of the corresponding region in the image data of the point cloud cavity region; The image data is segmented based on the reference color value to obtain segmented regions; If the point cloud hole region is located within a segmented region, the continuity of the point cloud hole region and its neighborhood is determined, wherein the segmented region where the point cloud hole region is located is used for point cloud data filling of the point cloud hole region.

[0030] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: The step of filling the point cloud hole region with point cloud data includes: The point cloud cavity region is fitted as a plane to obtain the depth and location information of each abnormal point cloud in the point cloud cavity region, so as to fill the point cloud data in the cloud cavity region.

[0031] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: The robot is equipped with a lidar, and the obstacle detection result of the lidar is a second obstacle area. The shooting direction of the depth camera is towards the ground area, and the shooting direction forms a preset angle with the extension surface of the depth camera on the robot. After the step of determining the first obstacle region in each of the supervoxels based on preset height and preset slope conditions, the method further includes: The first obstacle region and the second obstacle region are compared and deduplicated to remove the first obstacle region that is the same as the second obstacle region; The first obstacle region obtained after comparison and deduplication is added to the path obstacle set. After the robot crosses the first obstacle region, the crossed first obstacle region will be removed from the path obstacle set.

[0032] In one feasible implementation, the processor 1001 may call the robot movement environment detection program stored in the memory 1005 and further perform the following operations: If the robot cannot cross the first obstacle area, the insurmountable first obstacle area is added to the set of temporary constraints when the robot plans its path.

[0033] Reference Figure 2 In the first embodiment of the robot mobile environment detection method of this application, the robot is equipped with a depth camera, the shooting direction of the depth camera is towards the ground area, and the shooting direction forms a preset angle with the extension surface of the depth camera on the robot; It should be noted that in this embodiment, a depth camera is deployed on the robot to assist the robot in detecting obstacles in its moving environment. For example, refer to... Figure 6 This is a schematic diagram of the robot detection scene in this embodiment. The robot 1 in the figure is equipped with a depth camera 2.

[0034] The robot mobile environment detection method includes steps S10-S30: Step S10: Acquire depth point cloud data within a first preset range of the robot's surrounding environment based on the depth camera; In this embodiment, the implementing entity of the above-described robot movement environment detection method can be the robot itself or a remote server controlling the robot. The robot movement environment detection method can be used to detect obstacles appearing on the robot's movement path and avoid them in a timely manner.

[0035] For example, a depth camera configured on the robot can acquire depth point cloud data within a first preset range of the robot's surrounding environment. This first preset range depends on the shooting direction of the depth camera and the field of view of the depth camera's lens. For example, refer to... Figure 6 The area captured by the depth camera in the image is the first preset range. The depth camera acquires environmental images of the robot's surroundings, and these images carry depth information. Based on the intrinsic parameters of the depth camera and the depth information in the environmental image, the camera coordinates of each pixel in the environmental image can be obtained in the camera coordinate system. It should be noted that, typically, to facilitate the determination of positional relationships, a transformation matrix can be used to further transform the camera coordinates of each pixel in the camera coordinate system to the world coordinate system, obtaining the world coordinates of each pixel in the world coordinate system. Therefore, depending on the actual situation, the aforementioned depth point cloud data can be either the camera coordinates of each pixel in the environmental image or the world coordinates of each pixel in the environmental image. The depth point cloud data will be used for obstacle detection, where obstacle areas refer to areas that the robot cannot enter, or areas where entering them poses a safety risk.

[0036] Step S20: Segment the depth point cloud data to obtain each hypervoxel; For example, based on the continuity and vectors of each point cloud in the deep point cloud data, the point cloud data can be segmented into multiple super voxels. The segmentation process using the super voxel segmentation method can refer to existing solutions, and will not be elaborated here.

[0037] Understandably, traditional obstacle detection solutions using depth cameras typically require fitting a plane, resulting in a large granularity of the detected object. In complex working environments, such as the coexistence of steps and ramps, or scenarios where a robot is angled towards a fall zone, the accuracy of the fitting results decreases. This is because the fitted plane treats steps and ramps as a single unit, leading to a large granularity of the detected object and consequently lower accuracy. However, in this application, the depth point cloud data is segmented into multiple supervoxels, thereby refining the granularity of the detected object. For example, supervoxel segmentation can divide adjacent steps and ramps into two or more supervoxels, which serve as the unit for obstacle detection. Therefore, the granularity of the detected object in this embodiment is smaller, improving the accuracy of obstacle detection.

[0038] Step S30: Determine the first obstacle region in each of the supervoxels based on preset height and preset slope conditions.

[0039] For example, for any segmented hypervoxel, in this embodiment, the hypervoxel will be judged based on preset height conditions and preset slope conditions, and the hypervoxel that meets the corresponding conditions will be regarded as the first obstacle area. The first obstacle area can be an area with obstacles, or a fall area with a drop, etc.

[0040] Understandably, traditional methods use relatively simple obstacle detection conditions. For example, they typically consider areas above or below ground level as obstacles. However, in some work scenarios, there might be gentle slopes or low steps that are passable by the robot. If all of these are categorically classified as obstacles, it limits the robot's mobility, resulting in granular detection. In this embodiment, the obstacle detection conditions are further subdivided into height and slope conditions. Hypervoxels are then detected to determine whether a hypervoxel is an obstacle. For example, in this embodiment, the height and slope of the hypervoxel relative to the ground level are considered together to determine if it is an obstacle. Under these conditions, it is ensured that gentle slopes and low steps are not considered obstacles, thus reducing limitations on robot movement. It is understood that by subdividing the obstacle conditions, the granularity of the obstacle detection results can be refined, ensuring accuracy.

[0041] In this embodiment, depth point cloud data within a first preset range of the robot's surrounding environment is acquired using the depth camera; the depth point cloud data is segmented to obtain individual supervoxels; and a first obstacle region within each supervoxel is determined based on preset height and slope conditions. That is, depth point cloud data of the robot's surrounding environment is acquired using a depth camera, and then the depth point cloud data is segmented into multiple supervoxels. Compared to traditional methods that detect obstacles in the environment, this embodiment refines the detection targets when detecting obstacles. Furthermore, this embodiment refines the obstacle detection conditions into height and slope conditions, reducing the granularity of the obstacle detection conditions compared to traditional methods, thereby improving the accuracy of environmental obstacle detection results.

[0042] In one feasible implementation, the preset height condition includes a first height threshold and a second height threshold, wherein the first height threshold is greater than the second height threshold; the preset slope condition includes a first slope threshold and a second slope threshold, wherein the first slope threshold is greater than the second slope threshold. Reference Figure 3 Step S20, which determines the first obstacle region in each of the supervoxels based on preset height and preset slope conditions, includes steps S21-S22: Step S21: For any hypervoxel, determine the first relative ground height and the first relative ground slope of the hypervoxel based on the target ground plane where the robot is located; Step S22: Extend the hypervoxel plane to the location of the robot or above and below the location of the robot, and determine the drop value between the location of the robot and the hypervoxel after the plane is extended. Step S23: Determine whether the first relative ground height, the first relative ground slope, and the drop value satisfy the conditions in the first preset obstacle area condition set, wherein the first preset obstacle area condition set includes: the first relative ground height is greater than a first height threshold and the first relative ground slope is greater than a first slope threshold; the first relative ground height is greater than a first height threshold and the drop value is greater than a first height threshold; the first relative ground height is less than a second height threshold and the first relative ground slope is less than a second slope threshold; and the first relative ground height is less than a second height threshold and the drop value is less than a first height threshold. Step S24: If the conditions in the first preset obstacle region condition set are met, then the hypervoxel is determined to be the first obstacle region.

[0043] It should be noted that in this embodiment, the preset height condition includes a first height threshold and a second height threshold, and the first height threshold is greater than the second height threshold. The preset slope condition includes a first slope threshold and a second slope threshold, and the first slope threshold is greater than the second slope threshold. Typically, the target ground plane where the robot is moving is calibrated. When the robot moves on different terrains, there will be different target ground planes.

[0044] For example, for any hypervoxel, the first relative ground height and the first relative ground slope of the hypervoxel can be determined based on the target ground plane where the robot is located. For instance, for the first relative ground height of the hypervoxel, the distance between the geometric center of the hypervoxel and the target ground plane can be selected as the distance between the hypervoxel and the plane. For the first relative ground slope of the hypervoxel, the normal vector of the surface formed by some points on the surface of the hypervoxel can be obtained, the angle between the normal vector and the ground plane can be determined, and the first relative ground slope of the hypervoxel can be determined based on the angle.

[0045] Furthermore, it should be noted that simply comparing the relative height and slope of the first ground level with preset thresholds to determine the presence of obstacles can easily lead to misjudgments. For example, setting a threshold for the relative height or slope to be considered an obstacle would result in some steep slopes being identified as obstacles, even though the robot could pass through them. Conversely, setting both the relative height and slope to be considered obstacles would lead to some steps with small drops being identified as obstacles, even though the robot could pass through them. Additionally, the obtained hypervoxels are not always entirely accurate. For instance, hypervoxels segmented from the point cloud data corresponding to steps should theoretically be parallel to the horizontal plane, but in reality, they may not. Therefore, it's difficult to distinguish whether a hypervoxel corresponds to a slope or a step. Consequently, further judgment based on the relative height and slope of the first ground level after distinguishing between steps and slopes is also inaccurate. Therefore, in this embodiment, the distinction between a ramp and a step is blurred. Instead of determining whether it is a ramp or a step, the hypervoxel plane is extended to the robot's location or above and below it. The difference in elevation between the robot's location and the extended hypervoxel is then determined, and this difference is used as one of the criteria for determining whether the robot can pass through the hypervoxel (i.e., whether the hypervoxel is the first obstacle area).

[0046] After determining the first relative ground height, first relative ground slope, and the drop value determined by the plane extension of the hypervoxel, it is then determined whether the first relative ground height, first relative ground slope, and drop value satisfy the conditions in the first preset obstacle area condition set. The first preset obstacle area condition set includes: the first relative ground height is greater than a first height threshold and the first relative ground slope is greater than a first slope threshold; the first relative ground height is greater than a first height threshold and the drop value is greater than the first height threshold; the first relative ground height is less than a second height threshold and the first relative ground slope is less than a second slope threshold; and the first relative ground height is less than a second height threshold and the drop value is less than the first height threshold. Generally, if the first relative ground height, first relative ground slope, and drop value of the hypervoxel satisfy any one of the conditions in the first preset obstacle area condition set, the hypervoxel can be determined as a first obstacle area. Conversely, if none of the conditions in the first preset obstacle area condition set are satisfied, the hypervoxel can be determined as not an obstacle area. It is understandable that the elevation difference value obtained by extending the hypervoxel in a plane can reflect the smoothness between the hypervoxel and the robot's location. If the relationship between the hypervoxel and the robot's location is relatively smooth, it indicates that the robot can pass through. Therefore, in this embodiment, by incorporating the elevation difference value as one of the judgment criteria when determining obstacle areas, the accuracy of obstacle area judgment is improved.

[0047] In one feasible implementation, before determining the hypervoxel as the first barrier region in step S24, the method further includes steps S241-S243: Step S241: If the conditions in the first preset obstacle area condition set are met, then based on the preset horizontal ground plane of the robot's working environment, determine the second relative height of the super voxel and the second relative slope of the ground. Step S242: Determine whether the second relative ground height, the second relative ground slope, and the drop value satisfy the conditions in the second preset obstacle area condition set, wherein the second preset obstacle area condition set includes: the second relative ground height is greater than a first height threshold and the second relative ground slope is greater than a first slope threshold; the second relative ground height is greater than a first height threshold and the drop value is greater than a first height threshold; the second relative ground height is less than a second height threshold and the second relative ground slope is less than a second slope threshold; and the second relative ground height is less than a second height threshold and the drop value is less than a first height threshold. Step S243: If the conditions in the second preset obstacle region condition set are met, then the step of determining the hypervoxel as the first obstacle region is executed.

[0048] It should be noted that the aforementioned relative height and relative slope of the first ground are determined based on the target ground plane where the robot is located. Therefore, the target ground plane may be a slope, which could mislead the final judgment. For example, in terrain where the uphill and downhill slopes are continuous, the robot is located on the uphill plane and can obtain the angle of the downhill plane. If the actual slopes of both the uphill and downhill slopes exceed the preset slope threshold, the slope of the downhill slope is determined based on the downhill plane during the actual determination process. Therefore, the superposition of the two slopes may result in a larger determined downhill slope, thus misclassifying a passable downhill slope as an obstacle area, leading to a misjudgment.

[0049] For example, in this embodiment, after determining that the first relative ground height, the first relative ground slope, and the drop value satisfy the conditions in the first preset obstacle area condition set, the second relative ground height and the second relative ground slope of the hypervoxel are determined based on the preset horizontal ground plane of the robot's working environment. The preset horizontal ground plane can be determined by an IMU (inertial motion unit) configured on the robot. Finally, the second relative ground height, the second relative ground slope, and the drop value are used to determine whether the conditions in the second preset obstacle area condition set are satisfied. The second preset obstacle area condition set includes: the second relative ground height is greater than a first height threshold and the second relative ground slope is greater than a first slope threshold; the second relative ground height is greater than a first height threshold and the drop value is greater than a first height threshold; the second relative ground height is less than a second height threshold and the second relative ground slope is less than a second slope threshold; and the second relative ground height is less than a second height threshold and the drop value is less than a first height threshold. Typically, if the second relative ground height, the second relative ground slope, and the drop value satisfy any one of the conditions in the second preset obstacle area condition set, the hypervoxel can be determined as the first obstacle area. Conversely, if none of the conditions in the set of conditions for the second preset obstacle area are met, then it can be determined as a non-obstacle area.

[0050] Understandably, in this embodiment, when the first relative ground height, first relative ground slope, and drop value of the hypervoxel satisfy any one of the conditions in the first preset obstacle area condition set, it is also determined whether the second relative ground height, second relative ground slope, and drop value of the hypervoxel satisfy any one of the conditions in the second preset obstacle area condition set. Once it is determined that any one of the conditions in the second preset obstacle area condition set is satisfied, the hypervoxel is identified as the first obstacle area. Since the second relative ground height and second relative ground slope are determined based on a preset horizontal ground plane, in the case of continuous uphill and downhill terrain, the two slopes will not be superimposed as the downhill slope, thus ensuring the accuracy of the final obstacle area.

[0051] Reference Figure 4 Based on the first embodiment of this application, a second embodiment of this application is proposed. The parts identical to those in the above embodiment can be referred to the above content, and will not be repeated here. After acquiring depth point cloud data within a first preset range of the robot's surrounding environment based on the depth camera in step S10, the method further includes steps S100-S200: Step S100: If there is a point cloud hole region in the depth point cloud data, then obtain the image data of the point cloud hole region, and determine the continuity of the point cloud hole region and its neighborhood based on the image data. Step S200: If the continuity of the point cloud cavity region and its surrounding area is continuous, then point cloud data filling is performed on the point cloud cavity region.

[0052] It should be noted that in practical applications, certain specific scenarios can affect the depth point cloud data acquired by the depth camera. For example, in some scenarios, black tiles may be distributed along the edges of steps, and the point cloud data acquired by the depth camera from the black areas may be anomalous (i.e., data holes exist, although some valid point clouds are also observed in actual measurements). When multiple machines are working simultaneously, the structured light of some depth cameras may be interfered with, and holes may also appear in overlapping areas of the field of view. Consequently, anomalous point clouds will interfere with the detection of obstacles in the environment. To avoid the influence of special environments, in this embodiment, the point cloud hole areas (i.e., the areas where anomalous point clouds are formed) will be filled in.

[0053] For example, in this embodiment, a point cloud hole region refers to the area formed by connecting hole point clouds, and a hole point cloud is an anomalous point cloud. Since the depth information of an anomalous point cloud is inaccurate, the depth information of the acquired anomalous point cloud is usually not considered; therefore, annomalous point clouds are also called hole point clouds. When there is a point cloud hole region in the depth point cloud data, the image data of the point cloud hole region is then acquired. It can be understood that, in this embodiment, a visible light camera can also be integrated on the depth camera. The corresponding image data is acquired through the visible light camera. Typically, since the depth camera and the visible light camera are integrated, the areas captured by the visible light camera and the depth camera are usually very close. Most manufacturers' SDK development kits also support the alignment of depth maps and visible light images. Therefore, when alignment is set, the image output by the depth camera and the image by the visible light camera are the same area. Thus, based on the relative position of the point cloud hole region on the depth image, the image data corresponding to the point cloud hole region can be obtained. The continuity of point cloud hole regions can be determined by analyzing their image data. For example, continuity can be judged by the similarity between pixel values ​​of the previous pixel in the image data of the hole region. If the pixels are all similar colors and there are no other colors blocking the view, the hole region can be considered continuous, and also continuous with its surrounding area. If the hole region is determined to be continuous, it can be treated as a planar structure. The depth information (or point cloud data, such as position coordinates) of each anomalous point cloud within the hole region can then be used to fill in the missing point cloud data.

[0054] In one feasible implementation, before acquiring the image data of the point cloud hole region in step S100, the method includes steps S101-S102: Step S101: Obtain abnormal point clouds from the depth point cloud data; Step S102: If the proportion of abnormal point clouds in the total depth point cloud exceeds a preset threshold, it is determined that there are point cloud hole regions in the depth point cloud data.

[0055] It should be noted that in practical applications, considering that a small number of holes (also considered anomalous point clouds) may appear in the detected point cloud, these usually do not affect obstacle detection. Therefore, in this embodiment, to avoid unnecessary filling, the size of the point cloud hole region will be judged. For example, the number of anomalous point clouds in the depth point cloud data and the total number of depth point cloud data are obtained, and it is determined whether the proportion of the number of anomalous point clouds in the total number of depth point cloud data exceeds a preset threshold. If it exceeds the threshold, it is determined that there is a point cloud hole region.

[0056] In one feasible implementation, step S100, which determines the continuity of the point cloud hole region and its neighborhood based on the image data, includes steps S110-S150: Step S110: Connect adjacent abnormal point clouds in the depth point cloud data to obtain a connected region; Step S120: Determine the point cloud cavity region based on the connected region; Step S130: Determine the reference color value based on the color of the corresponding area in the image data of the point cloud hole region; Step S140: Segment the image data based on the reference color value to obtain segmented regions; Step S150: If the point cloud hole region is located within a segmented region, then the continuity of the point cloud hole region and its neighborhood is determined, wherein the segmented region where the point cloud hole region is located is used for point cloud data filling of the point cloud hole region.

[0057] For example, adjacent anomalous point clouds in the depth point cloud data are connected to obtain a connected region. The area formed by the edge of the connected region can be regarded as the point cloud hole region. If we consider that there may be some normal point clouds, the connected region may be irregular. Considering that most black ground is black ceramic tile, the point cloud hole region can also be obtained by plane fitting of the connected region. The normal point cloud within the point cloud hole region is obtained. It can be multiple normal point clouds or a single normal point cloud. Generally, the result of combining multiple normal point clouds is more accurate, but the identification process for each normal point cloud is the same. Therefore, in this embodiment, a single point cloud hole H will be used as an example for explanation. The first pixel value (i.e., color) of the anomalous point cloud in the image data is obtained. Based on its average color value A, the image is thresholded according to a certain threshold (a common image processing method). The pixel regions in the image with a value less than A+delta are extracted. These may be multiple discontinuous regions, i.e., multiple segmented regions. Furthermore, if a point cloud cavity is located exactly within a certain segmented region, it indicates that the point cloud cavity is continuous, and that the point cloud cavity is also continuous with its neighborhood. Continuous point cloud cavity regions can then be filled in using plane fitting. For example, the segmented region containing the point cloud cavity can be fitted as a plane to obtain the location information of each anomalous point cloud within the cavity region.

[0058] Reference Figure 5Based on the first and second embodiments of this application, a third embodiment of this application is proposed. The parts identical to those described above can be referred to the above content and will not be repeated here. The robot is equipped with a 2D LiDAR, the obstacle detection result of the LiDAR is a second obstacle area, the shooting direction of the depth camera is towards the ground area, and the shooting direction forms a preset angle with the extension surface of the depth camera on the robot.

[0059] For example, refer to Figure 6 The depth camera 2 is oriented towards the ground area 3, and its shooting direction forms a preset angle α with the extension surface 4 of the depth camera 2 on the robot 1. The specific angle α can be set by technicians according to requirements. This angle is directly related to the angle between the depth camera and the ground. Errors exist during actual installation, and the specific installation position and angle of each camera will be obtained through external parameter calibration. It should be noted that in this embodiment, the robot is also equipped with a LiDAR. The LiDAR can acquire environmental data (i.e., radar point cloud data) from a farther or larger area compared to the depth camera. However, the radar point cloud data acquired by 2D LiDAR is usually only the point cloud data of the surrounding environment at a certain horizontal cross-section, so low-lying obstacles and suspended obstacles cannot be accurately identified. In this embodiment, the depth camera's shooting direction is towards the ground area, thus compensating for the blind spots of the LiDAR. By combining the LiDAR and depth camera for environmental detection, the accuracy of the detection results is ensured.

[0060] After determining the first obstacle region in each of the supervoxels in step S30 based on preset height and preset slope conditions, the method further includes steps S40-S50: Step S40: Compare the first obstacle region with the second obstacle region to remove duplicates, so as to remove the first obstacle region that is the same as the second obstacle region; Step S50: The first obstacle region obtained after comparison and deduplication is added to the path obstacle set. After the robot crosses the first obstacle region, the crossed first obstacle region will be removed from the path obstacle set.

[0061] For example, the obstacle detection result of the LiDAR is the second obstacle region. It is understood that the LiDAR detection result can be in the form of a point cloud. Since LiDAR and depth cameras are two different detection methods, there may be overlapping obstacle regions in the first and second obstacle regions. Therefore, after obtaining the first obstacle region based on the depth camera, a comparison and deduplication operation can be performed. For example, the first geometric center of the first obstacle region can be determined, the second geometric center of the second obstacle region can be determined, and the distance between the first and second geometric centers can be judged. If the distance is less than a preset distance threshold, then the first and second obstacle regions can be considered to be the same obstacle region. Accordingly, if the first obstacle region is determined to be the same as a certain second obstacle region, the first obstacle region can be deduplicated. After deduplication in the above manner, the remaining first obstacle regions can be added to the path obstacle set. The robot can avoid the first obstacle regions in the path obstacle set and the second obstacle regions detected by the LiDAR. Furthermore, it should be noted that if the robot crosses the first obstacle region, the crossed first obstacle region can be removed from the path obstacle set. Whether the robot has crossed the first obstacle area can be determined by the robot's current position on the movement path and the position of the first obstacle area on the movement path.

[0062] In one feasible implementation, the robot movement environment detection method further includes step S60: Step S60: If the robot cannot cross the first obstacle area, then the first obstacle area that cannot be crossed is added to the temporary constraint set when the robot plans its path.

[0063] It should be noted that in this embodiment, the obstacle area may include deep pits or steps, which the robot may be unable to overcome. Therefore, the robot needs to replan its route. For example, most current path planning is based on electronic maps. When the robot's positioning fails, it may move towards steps, steep slopes, or other areas where it could fall. After detecting an insurmountable fall area, the robot often changes direction and plans a new route. However, since the fall area has entered the robot's blind spot, the robot will plan a path towards the fall area (which appears as a normal area on the electronic map). This repeated process may cause the robot to gradually approach the edge of the fall area, eventually leading to a fall accident.

[0064] In this embodiment, to address the aforementioned problem, when a first obstacle area exists that the robot cannot overcome, this first obstacle area is added to the temporary constraint set used by the robot when planning its path. This ensures that the robot's path planning, based on the current electronic map and the temporary constraint set, will not include the first obstacle area, thus avoiding safety risks for the robot in the aforementioned work scenario.

[0065] In addition, it should be noted that on the electronic map used for the above-mentioned route planning, a safety distance will be set at certain locations to prevent the robot from approaching the specific location. For example, the specific location may be an escalator.

[0066] In addition, to achieve the above objectives, this application also provides an electronic device, which includes: a memory, a processor, and a robot mobile environment detection program stored in the memory and executable on the processor. When the robot mobile environment detection program is executed by the processor, it implements the steps of the robot mobile environment detection method as described above.

[0067] The specific implementation of the electronic device in this application is basically the same as the embodiments of the robot mobile environment detection method described above, and will not be repeated here.

[0068] In addition, to achieve the above objectives, this application also provides a storage medium storing a robot mobile environment detection program, which, when executed by a processor, implements the steps of the robot mobile environment detection method described above.

[0069] The specific implementation of the storage medium in this application is basically the same as the embodiments of the robot mobile environment detection method described above, and will not be repeated here.

[0070] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0071] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, robot, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0073] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for detecting the mobile environment of a robot, characterized in that, The robot is equipped with a depth camera; The robot mobile environment detection method includes the following steps: The depth point cloud data of the robot's surrounding environment within a first preset range is acquired based on the depth camera; The depth point cloud data is segmented to obtain each hypervoxel; The first obstacle region in each of the supervoxels is determined based on preset height conditions and preset slope conditions, wherein the preset height conditions include a first height threshold and a second height threshold, the first height threshold being greater than the second height threshold, and the preset slope conditions include a first slope threshold and a second slope threshold, the first slope threshold being greater than the second slope threshold; The step of determining the first obstacle region in each of the supervoxels based on preset height and preset slope conditions includes: For any hypervoxel, the first relative ground height and the first relative ground slope of the hypervoxel are determined based on the target ground plane where the robot is located. The first relative ground height is the distance between the geometric center of the hypervoxel and the target ground plane, and the first relative ground slope is determined by the angle between the normal vector of the surface formed by some points on the surface of the hypervoxel and the target ground plane. Extend the hypervoxel plane to the location of the robot or above and below the location of the robot, and determine the drop value between the location of the robot and the hypervoxel after the plane is extended; Determine whether the first relative ground height, the first relative ground slope, and the drop value satisfy the conditions in the first preset obstacle area condition set, wherein the first preset obstacle area condition set includes: the first relative ground height is greater than a first height threshold and the first relative ground slope is greater than a first slope threshold; the first relative ground height is greater than a first height threshold and the drop value is greater than a first height threshold; the first relative ground height is less than a second height threshold and the first relative ground slope is less than a second slope threshold; and the first relative ground height is less than a second height threshold and the drop value is less than a first height threshold. If the conditions in the first preset obstacle region condition set are met, then the hypervoxel is determined to be the first obstacle region.

2. The robot mobile environment detection method as described in claim 1, characterized in that, Prior to the step of determining the hypervoxel as the first barrier region, the method further includes: If the conditions in the first preset obstacle area condition set are met, then based on the preset horizontal ground plane of the robot's working environment, the second relative height of the super voxel and the second relative slope of the ground are determined, wherein the preset horizontal ground plane is determined by the IMU configured on the robot; Determine whether the relative height of the second ground, the relative slope of the second ground, and the drop value meet the conditions in the second preset obstacle area condition set, wherein the second preset obstacle area condition set includes: the relative height of the second ground is greater than a first height threshold and the relative slope of the second ground is greater than a first slope threshold; the relative height of the second ground is greater than a first height threshold and the drop value is greater than a first height threshold; the relative height of the second ground is less than a second height threshold and the relative slope of the second ground is less than a second slope threshold; and the relative height of the second ground is less than a second height threshold and the drop value is less than a first height threshold. If the conditions in the second preset obstacle region condition set are met, then the step of determining the hypervoxel as the first obstacle region is executed.

3. The robot mobile environment detection method as described in claim 1, characterized in that, After the step of acquiring depth point cloud data within a first preset range of the robot's surrounding environment based on the depth camera, the method further includes: If there are point cloud hole regions in the depth point cloud data, then the image data of the point cloud hole regions is acquired, and the continuity of the point cloud hole regions and their neighborhoods is determined based on the image data. If the point cloud cavity region and its neighborhood are continuous, then point cloud data filling is performed on the point cloud cavity region.

4. The robot movement environment detection method as described in claim 3, characterized in that, Prior to the step of acquiring image data of the point cloud hole region, the method includes: Obtain abnormal point clouds from the depth point cloud data; If the proportion of abnormal point clouds in the total depth point cloud exceeds a preset threshold, it is determined that there are point cloud void regions in the depth point cloud data.

5. The robot movement environment detection method as described in claim 3, characterized in that, The step of determining the continuity of the point cloud hole region and its neighborhood based on the image data includes: Connecting adjacent abnormal point clouds in the depth point cloud data yields a connected region; The point cloud cavity region is determined based on the connected region; A reference color value is determined based on the color of the corresponding region in the image data of the point cloud cavity region; The image data is segmented based on the reference color value to obtain segmented regions; If the point cloud hole region is located within a segmented region, the continuity of the point cloud hole region and its neighborhood is determined, wherein the segmented region where the point cloud hole region is located is used for point cloud data filling of the point cloud hole region.

6. The robot mobile environment detection method according to any one of claims 1 to 5, characterized in that, The robot is equipped with a lidar, and the obstacle detection result of the lidar is a second obstacle area. The shooting direction of the depth camera is towards the ground area, and the shooting direction forms a preset angle with the extension surface of the depth camera on the robot. After the step of determining the first obstacle region in each of the supervoxels based on preset height and preset slope conditions, the method further includes: The first obstacle region and the second obstacle region are compared and deduplicated to remove the first obstacle region that is the same as the second obstacle region; The first obstacle region obtained after comparison and deduplication is added to the path obstacle set. After the robot crosses the first obstacle region, the crossed first obstacle region will be removed from the path obstacle set.

7. The robot movement environment detection method as described in claim 6, characterized in that, The robot mobile environment detection method also includes: If the robot cannot cross the first obstacle area, the insurmountable first obstacle area is added to the set of temporary constraints when the robot plans its path.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a robot mobile environment detection program stored in the memory and executable on the processor, wherein: when the robot mobile environment detection program is executed by the processor, it implements the steps of the robot mobile environment detection method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium stores a robot mobile environment detection program, which, when executed by a processor, implements the steps of the robot mobile environment detection method as described in any one of claims 1 to 7.

Images