Obstacle avoidance processing method, robot and computer readable storage medium

Abstract

The invention provides an obstacle avoidance processing method, a robot and a computer readable storage medium, and the method comprises the steps: carrying out the semantic segmentation of a target image, and recognizing the type of an obstacle; if the type of the obstacle is a dynamic obstacle, determining the movement speed of the dynamic obstacle and a predicted movement track within a preset duration; and based on the predicted motion trajectory and the motion speed, performing expansion processing on a region corresponding to the dynamic obstacle, and determining an obstacle avoidance region corresponding to the dynamic obstacle. According to the embodiment of the invention, the accuracy of determining the obstacle avoidance area is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot obstacle avoidance, and in particular to an obstacle avoidance processing method, a robot and a computer readable storage medium. BACKGROUND

[0002] At present, in the process of moving operation, the robot generally detects obstacles through laser radar, ultrasonic wave, infrared sensor and the like, or detects moving obstacles (such as animals and pedestrians) through continuous frames, and expands the region where the obstacle is located by a fixed range (such as uniformly expanding by 20 cm radius) to determine the obstacle avoidance range. However, for some obstacles, the obstacle avoidance range based on the fixed range expansion is not large enough, and the robot may still have a collision risk; and for other obstacles, the obstacle avoidance range based on the fixed range expansion is too large, which affects the operation efficiency of the robot. SUMMARY

[0003] The present application provides an obstacle avoidance processing method, a robot and a computer readable storage medium, aiming to improve the accuracy of determining the obstacle avoidance region.

[0004] To achieve the above-mentioned purpose, the present application provides an obstacle avoidance processing method, comprising: performing semantic segmentation processing on a target image to identify the category of an obstacle; if the category of the obstacle is a dynamic obstacle, determining the movement speed of the dynamic obstacle and the predicted movement trajectory within a preset time length; based on the predicted movement trajectory and the movement speed, performing expansion processing on the region corresponding to the dynamic obstacle to determine the obstacle avoidance region corresponding to the dynamic obstacle.

[0005] In addition, to achieve the above-mentioned purpose, the present application also provides a robot, which comprises a vision device, a walking device and an obstacle avoidance processing device; wherein the walking device is used to control the movement of the robot; the vision device is used to acquire the image of the working environment of the robot; the obstacle avoidance processing device comprises a processor and a memory, the memory stores a computer program executable by the processor, and the computer program is executed by the processor to implement the steps of the obstacle avoidance processing method as described above.

[0006] In addition, to achieve the above-mentioned purpose, the present application also provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the obstacle avoidance processing method as described above.

[0007] The obstacle avoidance processing method, the robot and the computer readable storage medium provided by the embodiments of the present application can improve the accuracy of determining the obstacle avoidance area, thereby reducing the collision risk and improving the safety by performing semantic segmentation processing on the target image, identifying the category of the obstacle, determining the motion speed of the dynamic obstacle and the predicted motion trajectory of the dynamic obstacle within a preset time length if the category of the obstacle is a dynamic obstacle, performing inflation processing on the region corresponding to the dynamic obstacle based on the predicted motion trajectory and the motion speed, and determining the obstacle avoidance region corresponding to the dynamic obstacle, i.e., determining the appropriate range of the obstacle avoidance region based on the motion condition of the dynamic obstacle.

[0008] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory and are not restrictive of the application. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0010] Figure 1 is a flowchart of an obstacle avoidance processing method provided by an embodiment of the present application; Figure 2 is a flowchart of an obstacle avoidance processing method provided by an embodiment of the present application; Figure 3 is a flowchart of determining the motion speed of the dynamic obstacle provided by an embodiment of the present application; Figure 4 is a flowchart of distinguishing the motion abnormal feature points and the motion normal feature points of each feature point of the dynamic obstacle based on the translation speed model provided by an embodiment of the present application; Figure 5 is a flowchart of determining the obstacle avoidance region corresponding to the dynamic obstacle provided by an embodiment of the present application; Figure 6 is a flowchart of another obstacle avoidance processing method provided by an embodiment of the present application; Figure 7 is a structural schematic diagram of a robot provided by an embodiment of the present application; Figure 8 is a schematic block diagram of an obstacle avoidance processing device of a robot provided by an embodiment of the present application. DETAILED DESCRIPTION

[0011] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0012] The flowcharts shown in the drawings are only illustrative, and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further decomposed, combined or partially merged, so that the actual execution order may be changed according to the actual situation.

[0013] It should be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0014] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0015] Embodiments of the present application provide an obstacle avoidance processing method, a robot and a computer readable storage medium, for improving the accuracy of determining an obstacle avoidance area.

[0016] Please refer to Figure 1 , Figure 1 is a flowchart of an obstacle avoidance processing method provided by an embodiment of the present application. The method can be applied to a robot or a control device (such as a remote controller, a mobile phone, etc.) of a robot, and the application scenario of the method is not limited in the present application.

[0017] As Figure 1 shown, the obstacle avoidance processing method specifically includes steps S101 to S103.

[0018] S101, performing semantic segmentation processing on a target image to identify the category of an obstacle.

[0019] The target image can be an original image / video frame collected by a visual device installed on a target device, or the target image can be an image obtained after pre-processing such as de-warping on the original image / video frame, which is not specifically limited in the present application. The target device includes but is not limited to a lawn mowing robot, an autonomous driving vehicle, an indoor mobile robot, a patrol robot, etc.; the visual device includes but is not limited to a camera, a camera, a laser radar, etc.

[0020] For example, taking a lawn mowing robot as an example, a binocular camera is installed on the lawn mowing robot, as shown in Figure 2 As shown, first, a first image is acquired by the left camera and a second image is acquired by the right camera, based on a pre-labeled camera internal parameter calibration file (such as the focal length, principal point coordinates, and distortion coefficient of the camera, etc.), the first image and the second image are rectified to remove image distortion. Then, taking one of the rectified first image and the second image (such as the first image) as a reference, the disparity corresponding to each pixel point is determined, and then the depth of each pixel point is calculated based on the relationship between the disparity and the depth, such as according to the following formula (1), to generate a binocular depth image.

[0021] Z = f * B / d (1) Wherein, Z is the depth, f is the camera focal length, B is the camera baseline distance, and d is the disparity.

[0022] For example, invalid disparity is removed, and pixel points with a depth less than or equal to an effective depth (such as 3m) are selected to generate a binocular depth image, without the need to process all depths, thereby reducing the amount of calculation and processing.

[0023] And taking one of the rectified first image and the second image (such as the first image) as a target image, the target image is subjected to semantic segmentation processing to obtain a semantic segmentation image corresponding to the target image. For example, the target image is input into a trained semantic segmentation model, and the target image is subjected to semantic segmentation processing based on the semantic segmentation model to obtain a semantic segmentation image. Wherein, the semantic segmentation model can replace the backbone network of deeplabv3 with a lightweight network mobilenet to balance the accuracy and speed, and can meet the real-time requirements of embedded platforms. A large number of sample images used for model training are collected, the objects in the sample images are classified and semantically labeled to obtain training samples, and the training samples are used to train the semantic segmentation model to obtain a trained semantic segmentation model. For example, taking the lawn mowing robot as an example, a large number of outdoor lawn scene images are collected for classification and semantic labeling (such as hedgehogs, ground squirrels, cats, dogs, people, vehicles, stones, shrubs, trees, sky, background, lawn, etc.), and the labeled image data is used to train the semantic segmentation model, and the trained semantic segmentation model is saved.

[0024] In practical applications, considering that the time marker information (such as a timestamp) of the binocular depth image is not the same as the time marker information of the semantic segmentation image, two queues (buffer areas, which can accumulate multiple frames of data) can be used to store different image data, for example, queue 1 is used to store the target image timestamp and the corresponding binocular depth image (hereinafter referred to as an element), and queue 2 is used to store the target image timestamp and the corresponding semantic segmentation image. It is judged whether the timestamp corresponding to the first element in the queue 1 is equal to the timestamp corresponding to the first element in the queue 2. If they are equal, the first element in the queue 1 and the first element in the queue 2 are output, and after output, the first element in the queue 1 and the first element in the queue 2 are deleted, and the timestamp is output in a loop. If the timestamp corresponding to the first element in the queue 1 is greater than the timestamp corresponding to the first element in the queue 2, the first element in the queue 2 is deleted, and the loop is judged again. If the timestamp corresponding to the first element in the queue 1 is less than the timestamp corresponding to the first element in the queue 2, the first element in the queue 1 is deleted, and the loop is judged again. In this way, it is ensured that the timestamp of the output binocular depth image is consistent with the timestamp of the semantic segmentation image, that is, the binocular depth image and the semantic segmentation image are time-synchronized.

[0025] According to the time-synchronized binocular depth image and the semantic segmentation image, the obstacle and the category of the obstacle are determined, wherein the determined obstacle corresponds to a depth less than or equal to a preset depth threshold (such as 2 m). It should be noted that the preset depth threshold can be flexibly set according to actual conditions, and is not specifically limited in the present application. The category of the obstacle includes static obstacles and dynamic obstacles, for example, still taking the lawn working scene of the lawn mower robot as an example, the static obstacles include but are not limited to stones, shrubs, trees, etc., and the dynamic obstacles include but are not limited to pedestrians, vehicles, animals, etc.

[0026] For example, the intersection of the mask corresponding to the binocular depth image and the mask corresponding to the semantic segmentation image is obtained, and then it is judged whether the obstacle semantics exist in the intersection. If the obstacle semantics do not exist, it is determined that there is no obstacle, and the processing can be ended. For example, a prompt information of no obstacle can be output, and subsequent processing is not required, thereby reducing resource occupation. If the obstacle semantics exist, a certain range of space regions (such as a space region of 2 m*2 m*2 m) is divided into multiple grids (voxels) according to a preset grid resolution (such as a grid resolution of 0.02 m*0.02 m*0.02 m), and only the space region is analyzed and processed, thereby saving CPU processing amount.

[0027] The binocular depth image and the semantic segmentation image are projected into voxels, each voxel is assigned a semantic label and an occupancy probability. For example, the occupancy probability is calculated according to the following formula (2): (2) wherein, is the current occupancy probability; is the last occupancy probability; is the observation likelihood probability, which is related to the sensor accuracy, for example, it can be set to 0.9, of course, it can also be set to other values, which are not specifically limited in the present application.

[0028] Suppose the coordinates of a pixel point are (x, y, z) (unit: m), and the grid resolution is 0.02m*0.02m*0.02m, then the corresponding voxel position is (x / 0.02, y / 0.02, z / 0.02) m, and (x / 0.02, y / 0.02, z / 0.02) m is converted to an integer to obtain the corresponding voxel coordinates.

[0029] In addition, the semantic label of each pixel point in the voxel can be obtained by superimposing the semantic segmentation result corresponding to each pixel point. For example, still taking the lawn working scene of the lawn mower as an example, the obtained semantic labels include but are not limited to shrubs, trees, animals, people, vehicles, stones, sky, background, lawn, etc.

[0030] In the case where one or more semantic labels exist in a voxel, the semantic label of the voxel is given according to the following situations: ①Judge whether the maximum occupancy probability in the occupancy probability corresponding to multiple semantic labels is greater than or equal to a preset probability threshold, wherein the preset probability threshold is set to 2.0, for example, of course, it can also be set to other values, which are not specifically limited in the present application. If the maximum occupancy probability is greater than or equal to the preset probability threshold, the semantic label of the voxel is set to the semantic label corresponding to the maximum occupancy probability, and the voxel is marked as occupied. For example, still taking the lawn working scene of the lawn mower as an example, if the corresponding voxel exists lawn semantic label and obstacle (dynamic obstacle / static obstacle) semantic label, and the lawn semantic label and the obstacle semantic label are both greater than or equal to the preset probability threshold, the semantic label of the voxel is set to the obstacle semantic label as the standard, and the lawn occupancy is ignored.

[0031] ②If the maximum occupancy probability is less than the preset probability threshold, and the sum of the occupancy probabilities corresponding to multiple semantics is greater than or equal to the preset probability threshold, the semantic label of the voxel is set to a mixed semantic label, and the voxel is marked as occupied; otherwise, the voxel is set as a passable area.

[0032] ③Judge the occupancy probability of the passable area (such as the lawn), if the occupancy probability of the passable area is greater than or equal to the preset probability threshold, the semantic label of the passable area is set to the first semantic label (such as the lawn semantic label), otherwise, it is set to the second semantic label (such as the unknown area semantic label).

[0033] For example, a plane (such as a lawn plane) is fitted around each obstacle by the semantic label of the voxel, for example, an area (such as an area within 10 cm / 5 voxel range) around the obstacle can be taken out for plane fitting, and the plane fitting can be performed in various ways, such as least square method, RANSAC plane fitting, etc., which are not specifically limited in the present application. Based on the fitted plane, it is determined whether there is a dynamic obstacle semantic label, and if so, the category of the obstacle is determined as a dynamic obstacle.

[0034] S102, if the category of the obstacle is a dynamic obstacle, determining the motion speed of the dynamic obstacle and the predicted motion trajectory within a preset time length.

[0035] In the case where the semantic label of the voxel is a dynamic obstacle (such as a pedestrian, a vehicle, an animal, etc.), i.e., the category of the obstacle is a dynamic obstacle, since the motion state of the dynamic obstacle is uncertain, such as it can be stationary, it can be slow motion, or it can be fast motion, etc., therefore, for the dynamic obstacle, the obstacle avoidance region inflation processing is differentiated according to its current motion state. First, the motion speed of the dynamic obstacle is determined. In some embodiments, as shown in Figure 3 S102 includes sub-step S1021 and sub-step S1022.

[0036] S1021, based on the translation speed model, distinguishing each feature point of the dynamic obstacle into motion abnormal feature points and motion normal feature points; S1022, calculating the speed average corresponding to the motion normal feature points, and determining the speed average as the motion speed of the dynamic obstacle.

[0037] The translation speed model is mainly used to describe the mathematical model of the target device moving at a constant speed, which can be fitted by the least square method. Based on the motion of each feature point of the dynamic obstacle, each feature point is distinguished into motion abnormal feature points and motion normal feature points by referring to the translation speed model. The motion normal feature point refers to the feature point whose actual motion is consistent with or has a small deviation from the predicted motion of the translation speed model, and the small deviation refers to that the difference between the normal feature point and the predicted feature point is less than a budget threshold, which can be defined by the user; the motion abnormal feature point refers to the feature point whose actual motion is inconsistent with or has a large deviation from the predicted motion of the translation speed model, and the large deviation refers to that the difference between the normal feature point and the predicted feature point is less than a budget threshold, which can be defined by the user.

[0038] In some embodiments, as shown in Figure 4As shown, step S1021 includes sub-step S10211 to sub-step S10214.

[0039] S10211, acquiring a velocity vector corresponding to each feature point of the dynamic obstacle; S10212, calculating a residual between the velocity vector corresponding to each feature point and the velocity corresponding to the translation velocity model; S10213, if the residual is greater than a preset value, determining that the feature point corresponding to the residual is the motion abnormal feature point; S10214, if the residual is less than or equal to the preset value, determining that the feature point corresponding to the residual is the motion normal feature point.

[0040] For example, first, the pose change T of the target device of the current frame and the voxel coordinates P0 of the dynamic obstacle of the previous frame are acquired, and based on T and P0, the voxel coordinates P1 of the dynamic obstacle of the current frame are predicted as P1=T*P0, and based on P1 and the voxel coordinates P2 of the dynamic obstacle of the current frame, the coincidence degree of the voxels corresponding to the dynamic obstacle is judged, and if the coincidence degree is greater than or equal to a coincidence degree threshold, it is considered that the dynamic obstacles of the two frames are the same obstacle. Considering that in the case of high frame rate, the dynamic obstacle motion distance between two frames will not be too far, and the coincidence degree of the voxels corresponding to the dynamic obstacle will be higher, the coincidence degree threshold can be set to 80%, and when the coincidence degree >=80%, it is determined that the dynamic obstacles corresponding to the two frames are the same obstacle. It should be noted that the coincidence degree threshold can also be set to other values, which are not limited in the present application.

[0041] After it is determined that the dynamic obstacles corresponding to the two frames are the same obstacle, the segmented mask region of the dynamic obstacle of the previous frame is extracted, and the sparse optical flow method (such as Lucas-Kanade) is used to track the feature points of the previous frame to the current frame, and then the matching feature points of the previous frame are projected to the 3D coordinates of the first coordinate system (such as the camera coordinate system) through the binocular depth image of the previous frame, and the matching feature points of the current frame are projected to the 3D coordinates of the first coordinate system through the binocular depth image of the current frame; wherein the feature points that cannot directly obtain the depth can use the bilinear difference method to obtain the corresponding depth. Then, the 3D coordinates of the first coordinate system are converted to the second coordinate system (such as the map coordinate system) to obtain the coordinates P' of the matching feature points of the previous frame in the second coordinate system and the coordinates P of the matching feature points of the current frame in the second coordinate system, and then the displacement vector ΔP=P P', so as to obtain the corresponding velocity v of the matching feature point in the current frame according to the displacement vector AP, and the corresponding velocity v' of the matching feature point in the previous frame is obtained in the same manner, and the corresponding velocity vector AV = v - v' is calculated according to v and v'.

[0042] The residual of the corresponding velocity vector AV of each feature point and the corresponding velocity of the translation velocity model is calculated, if the residual of the feature point is less than or equal to a preset value (such as 0.1), the feature point is determined as a motion normal feature point, otherwise, if the residual of the feature point is greater than the preset value, the feature point is determined as a motion abnormal feature point. The motion normal feature points are retained, and the corresponding velocity mean is calculated based on the corresponding velocity vector AV of the motion normal feature points, and the velocity mean is determined as the motion velocity of the dynamic obstacle. By eliminating the motion abnormal feature points, the motion velocity is determined by the motion normal feature points, which ensures the accuracy of the motion velocity.

[0043] The dynamic obstacle has a clear position change trend and potential collision risk. In addition to determining the motion velocity of the dynamic obstacle, the predicted motion trajectory of the dynamic obstacle within a preset time length is also determined. For example, based on the historical position, velocity (linear velocity, angular velocity) and corresponding motion model (such as constant velocity model) of the dynamic obstacle, the predicted motion trajectory of the dynamic obstacle within a future short preset time T_pred (such as 1-2 seconds) is predicted.

[0044] S103, based on the predicted motion trajectory and the motion velocity, performing inflation processing on the region corresponding to the dynamic obstacle to determine an obstacle avoidance region corresponding to the dynamic obstacle.

[0045] Based on the predicted motion trajectory and the motion velocity, the inflation processing of the obstacle avoidance region corresponding to the dynamic obstacle is differentiated. For example, the greater the motion velocity of the dynamic obstacle, the greater the range of inflation of the region corresponding to the dynamic obstacle, that is, the greater the determined obstacle avoidance region, and the target device performs obstacle avoidance movement based on the obstacle avoidance region, thereby reducing the collision risk and improving the safety.

[0046] For example, after determining the obstacle avoidance region, the corresponding obstacle avoidance information is output, and the obstacle avoidance information includes but is not limited to the obstacle and the corresponding obstacle avoidance region. In this way, the user can intuitively understand the obstacle avoidance situation by checking the obstacle avoidance information, thereby further improving the user experience.

[0047] In some embodiments, as shown in FIG. 1, Figure 5 As shown in FIG. 1, step S103 includes sub-step S1031 to sub-step S1033.

[0048] S1031, determining the motion state of the dynamic obstacle based on the motion velocity; S1032, determining the inflation radius according to the motion state; S1033, based on the predicted motion trajectory and the inflation radius, performing inflation processing on the region corresponding to the dynamic obstacle to determine an obstacle-avoiding region corresponding to the dynamic obstacle.

[0049] The motion state of the dynamic obstacle includes static, motion, etc. For example, a preset speed range for judging the motion state of the dynamic obstacle is preset. It should be noted that the preset speed range can be flexibly set according to actual conditions, and is not specifically limited in this application. If the motion speed of the dynamic obstacle is within the preset speed range, it is determined that the dynamic obstacle is static. Otherwise, if the motion speed of the dynamic obstacle is outside the preset speed range, it is determined that the dynamic obstacle is in motion. Then, based on the motion state of the dynamic obstacle, an inflation radius for performing inflation processing on the region corresponding to the dynamic obstacle is determined.

[0050] In the case of a static dynamic obstacle, it is first determined whether the dynamic obstacle is an obstacle-avoiding object that needs to be avoided. If the minimum height of the dynamic obstacle is less than or equal to the height of the target device, it is indicated that a collision may occur, and it is determined that the dynamic obstacle is an obstacle-avoiding object. If the minimum height of the dynamic obstacle is greater than the height of the target device, the minimum height region of the dynamic obstacle is inflated to obtain a corresponding inflated minimum height. The inflated minimum height is compared with the height of the target device. If the inflated minimum height is less than or equal to the height of the target device, it is determined that the dynamic obstacle is an obstacle-avoiding object. Otherwise, if the inflated minimum height is greater than the height of the target device, it is determined that the dynamic obstacle is not an obstacle-avoiding object. For example, taking a lawn mowing robot as the target device, if the minimum height (based on the difference in height of the plane fitted by the obstacle) of the voxel corresponding to the dynamic obstacle marked as occupied is less than or equal to the maximum height of the lawn mowing robot, it is determined that the dynamic obstacle is an obstacle-avoiding object, and obstacle marking is performed on the map. If the minimum height of the voxel corresponding to the dynamic obstacle marked as occupied is greater than the maximum height of the lawn mowing robot, it is indicated that the dynamic obstacle is a floating obstacle, and the lawn mowing robot can pass normally. At this time, the dynamic obstacle can be inflated based on a preset reference radius (such as 5 cm) only to inflate the lowest contour of the dynamic obstacle in the three-dimensional world, and then it is determined whether the inflated minimum height of the voxel corresponding to the dynamic obstacle after inflation is greater than the maximum height of the lawn mowing robot. If the inflated minimum height is greater than the maximum height of the lawn mowing robot, it is indicated that the lawn mowing robot can still pass normally, and it is determined that the dynamic obstacle is not an obstacle-avoiding object. Otherwise, if the inflated minimum height is less than or equal to the maximum height of the lawn mowing robot, it is determined that the dynamic obstacle is an obstacle-avoiding object, and obstacle marking is performed on the map.

[0051] Through the dynamic obstacle height determination and contour inflation correction mechanism, the problem of easy misjudgment of floating obstacles by traditional height threshold-based obstacle avoidance strategies is avoided, the understanding ability of the robot for a complex three-dimensional environment is improved, and the path planning is more safe and efficient.​

[0052] For the dynamic obstacle in a static state determined as an obstacle-avoiding object, the first inflation radius is determined according to formula (3): (3) wherein, is the first inflation radius corresponding to the dynamic obstacle in a static state; is a reference radius (for example, 5 cm); is a first inflation coefficient (for example, 1.3). It should be noted that, , can be flexibly set according to actual conditions, which are not specifically limited in the present application.

[0053] Then, the region corresponding to the dynamic obstacle in a static state determined as an obstacle-avoiding object is inflated based on the first inflation radius , to determine the obstacle-avoiding region corresponding to the dynamic obstacle. That is, only the region corresponding to the dynamic obstacle needing to avoid obstacles is inflated, and the region corresponding to the dynamic obstacle (passable floating obstacle) not needing to avoid obstacles is not inflated, thereby saving CPU processing amount and improving efficiency.

[0054] In the case of dynamic obstacle movement, the second inflation radius is determined according to formula (4): (4) wherein, is the second inflation radius corresponding to the dynamic obstacle in motion; is a reference radius; is a second inflation coefficient, is greater than , for example, can be set to 1.5; is the current speed of the dynamic obstacle, is a preset safe reaction time (for example, 0.5 s), represents the conservative response time required for system planning and control, the term introduces a speed-related longitudinal safety buffer, and the faster the speed, the greater the buffer distance; is an adjustment parameter, is a prediction time, is a prediction uncertainty radius, which becomes larger with the prediction time , to represent that the longer the prediction time is, the greater the uncertainty is, and the greater the inflation processing is, to obtain an obstacle-avoiding region in a conical shape (narrow at the starting end and wide at the ending end), for example. It should be noted that, , , 、 The flexible setting can be made according to actual conditions, and no specific limitation is made in the present application.

[0055] Then, based on the second inflation radius and the predicted motion trajectory, the dynamic obstacle corresponding region is inflated based on the predicted motion trajectory as the center line to determine the dynamic obstacle corresponding obstacle avoidance region, and the target device moves based on the obstacle avoidance region to reduce the collision risk and improve safety.

[0056] In some embodiments, as Figure 6 shown, step S101 is followed by step S104.

[0057] S104, if the category of the obstacle is a static obstacle, the static obstacle corresponding region is inflated based on a third inflation radius to determine the static obstacle corresponding obstacle avoidance region.

[0058] In the case where the semantic label of the voxel is a static obstacle (such as a stone, a shrub, a tree, etc.), that is, the category of the obstacle is a static obstacle, the static obstacle corresponding region is inflated based on a third inflation radius to determine the static obstacle corresponding obstacle avoidance region. For example, the third inflation radius is determined according to formula (5): (5) wherein, is the third inflation radius, is a reference radius, is a third inflation coefficient, is less than , for example, can be set to 1.2, of course can also be set to other values, and no specific limitation is made in the present application.

[0059] Then, the static obstacle corresponding region is inflated based on the third inflation radius to determine the static obstacle corresponding obstacle avoidance region. Compared with the dynamic obstacle, the static obstacle corresponding region is inflated in a smaller range, so that more target device moving areas can be retained, thereby reducing the target device missed working area. For example, more mowing areas can be retained to reduce the mowing robot missed mowing area.

[0060] In some embodiments, before the static obstacle corresponding region is inflated based on the third inflation radius to determine the static obstacle corresponding obstacle avoidance region, it includes: determining whether the static obstacle is an obstacle avoidance object; if the static obstacle is an obstacle avoidance object, the third inflation radius is determined.

[0061] For example, taking the mowing robot as an example, if the lowest height of the voxel corresponding to the static obstacle marked as occupied is less than or equal to the highest height of the mowing robot, the static obstacle is determined as an obstacle-avoiding object, and obstacle marking is performed on the map. If the lowest height of the voxel corresponding to the static obstacle marked as occupied is greater than the highest height of the mowing robot, it indicates that the static obstacle is a floating obstacle, and the mowing robot can pass through normally. At this time, the third inflation radius R 3 can be determined based on the preset reference radius R 0 and the highest height of the mowing robot. Only the lowest contour of the static obstacle in the three-dimensional world is inflated, and then it is determined whether the lowest inflation height of the voxel corresponding to the inflated static obstacle is greater than the highest height of the mowing robot. If the lowest inflation height is greater than the highest height of the mowing robot, it indicates that the mowing robot can still pass through normally, and at this time, the static obstacle is determined as not an obstacle-avoiding object. Otherwise, if the lowest inflation height is less than or equal to the highest height of the mowing robot, the static obstacle is determined as an obstacle-avoiding object, and obstacle marking is performed on the map.

[0062] For the static obstacle determined as an obstacle-avoiding object, the third inflation radius R 3 can be determined according to the above formula (5) Then, the third inflation radius R 3 is used to perform inflation processing on the static obstacle corresponding region to determine the obstacle-avoiding region corresponding to the static obstacle. The static obstacle corresponding region is inflated to determine the obstacle-avoiding region corresponding to the static obstacle. That is, only the static obstacle corresponding region that needs to avoid obstacles is inflated, and the static obstacle corresponding region that does not need to avoid obstacles (passable floating obstacle) is not inflated, thereby saving CPU processing amount and improving efficiency.

[0063] In the above embodiment, by performing semantic segmentation processing on the target image, the category of the obstacle is recognized. If the category of the obstacle is a dynamic obstacle, the movement speed and the predicted movement trajectory in a preset time length of the dynamic obstacle are determined. Based on the predicted movement trajectory and the movement speed, the dynamic obstacle corresponding region is inflated to determine the obstacle-avoiding region corresponding to the dynamic obstacle. That is, the appropriate range of the obstacle-avoiding region is determined based on the movement of the dynamic obstacle, which improves the accuracy of determining the obstacle-avoiding region, thereby reducing the collision risk and improving the safety.

[0064] Please refer to Figure 7 , Figure 7 is a structural schematic diagram of a robot provided by an embodiment of the present application. The robot includes but is not limited to a mowing robot, an indoor mobile robot, a patrol robot, etc. As shown in Figure 7As shown, the robot 1000 includes a walking device 100, a vision device 200, and an obstacle avoidance processing device (not shown in the figure). The walking device 100 includes, but is not limited to, a driving wheel, a guide wheel, etc., and is used to control the movement of the robot 1000. The vision device 200 includes, but is not limited to, a camera, a camera lens, a laser radar, etc., and is used to obtain an image of the working environment of the robot 1000. The obstacle avoidance processing device determines an obstacle avoidance area corresponding to an obstacle in the working environment of the robot 1000 according to the image of the working environment of the robot 1000, and thus moves in an obstacle avoidance manner based on the obstacle avoidance area.

[0065] For example, a schematic block diagram of the obstacle avoidance processing device is as shown in Figure 8 As shown, the obstacle avoidance processing device 300 can include a processor 310 and a memory 320, wherein the processor 310 and the memory 320 are connected through a bus, such as an I2C (Inter-integrated Circuit) bus.

[0066] Specifically, the processor 310 can be a micro-controller unit (MCU), a central processing unit (CPU), or a digital signal processor (DSP), etc.

[0067] Specifically, the memory 320 can be a flash chip, a read-only memory (ROM) disk, an optical disk, a U disk, or a mobile hard disk, etc. The memory 320 stores various computer programs for the processor 310 to execute.

[0068] The processor 310 is configured to run the computer programs stored in the memory and implement the following when executing the computer programs: performing semantic segmentation processing on a target image to identify a category of an obstacle; if the category of the obstacle is a dynamic obstacle, determining a movement speed of the dynamic obstacle and a predicted movement trajectory of the dynamic obstacle within a preset time length; based on the predicted movement trajectory and the movement speed, performing inflation processing on a region corresponding to the dynamic obstacle to determine an obstacle avoidance area corresponding to the dynamic obstacle.

[0069] In some embodiments, when implementing the inflation processing on the region corresponding to the dynamic obstacle based on the predicted movement trajectory and the movement speed to determine the obstacle avoidance area corresponding to the dynamic obstacle, the processor 310 is configured to implement: based on the movement speed, determining a movement state of the dynamic obstacle; determine an inflation radius according to the motion state; perform inflation processing on the region corresponding to the dynamic obstacle based on the predicted motion trajectory and the inflation radius, to determine an obstacle-avoiding region corresponding to the dynamic obstacle.

[0070] In some embodiments, the processor 310, in implementing the determining of the motion speed of the dynamic obstacle, is configured to: distinguish between motion abnormal feature points and motion normal feature points of each feature point of the dynamic obstacle based on a translation speed model; calculate a speed average corresponding to the motion normal feature points, and determine the speed average as the motion speed of the dynamic obstacle; the determining of the motion state of the dynamic obstacle based on the motion speed comprises: if the motion speed is within a preset speed range, determining that the dynamic obstacle is static; if the motion speed is outside the preset speed range, determining that the dynamic obstacle is moving.

[0071] In some embodiments, the processor 310, in implementing the distinguishing between motion abnormal feature points and motion normal feature points of each feature point of the dynamic obstacle based on a translation speed model, is configured to: obtain a speed vector corresponding to each feature point of the dynamic obstacle; calculate a residual error between the speed vector corresponding to each feature point and a speed corresponding to the translation speed model; if the residual error is greater than a preset value, determining that the feature point corresponding to the residual error is the motion abnormal feature point; if the residual error is less than or equal to the preset value, determining that the feature point corresponding to the residual error is the motion normal feature point.

[0072] In some embodiments, the processor 310, in implementing the determining of the inflation radius according to the motion state, is configured to: if the dynamic obstacle is static, determining whether the dynamic obstacle is an obstacle-avoiding object; if the dynamic obstacle is an obstacle-avoiding object, determining a first inflation radius according to the formula . wherein, is the first inflation radius corresponding to the static dynamic obstacle, is a reference radius, is a first inflation coefficient.

[0073] In some embodiments, the processor 310, in implementing the determining of the inflation radius according to the motion state, is configured to: If the dynamic obstacle moves, a second inflation radius is determined according to a formula wherein, is the second inflation radius corresponding to the moving dynamic obstacle, is a reference radius, is a second inflation coefficient, is a current speed of the dynamic obstacle, is a preset safety reaction time, is an adjustment parameter, is a prediction time.

[0074] In some embodiments, the processor 310, after implementing the semantic segmentation processing on the target image to identify the category of the obstacle, is configured to implement: If the category of the obstacle is a static obstacle, an inflation processing is performed on a region corresponding to the static obstacle based on a third inflation radius to determine an obstacle avoidance region corresponding to the static obstacle.

[0075] In some embodiments, the processor 310, before implementing the inflation processing on the region corresponding to the static obstacle based on the third inflation radius to determine the obstacle avoidance region corresponding to the static obstacle, is configured to implement: determine whether the static obstacle is an obstacle avoidance object; If the static obstacle is an obstacle avoidance object, the third inflation radius is determined.

[0076] The robot 1000 can perform the obstacle avoidance processing method provided by the embodiments of the present application, and thus can achieve the beneficial effects of the obstacle avoidance processing method provided by the embodiments of the present application. Details are described above, and will not be repeated here.

[0077] For example, in the scenario of a lawn mowing robot mowing a lawn, the lawn mowing robot is installed with a binocular camera. First, a first image is obtained by a left camera of the binocular camera, and a second image is obtained by a right camera of the binocular camera. The first image and the second image are corrected to remove image distortion. Then, one of the corrected first image and the second image (for example, the first image) is taken as a reference, and a binocular depth image is generated based on the disparity. Furthermore, the one of the corrected first image and the second image (for example, the first image) is taken as a target image, and the target image is input into a trained semantic segmentation model. The target image is subjected to semantic segmentation processing based on the semantic segmentation model to obtain a semantic segmentation image.

[0078] ​Next, based on the time-synchronized binocular depth image and semantic segmentation image, the intersection of the mask corresponding to the depth of the binocular depth image (where the depth is less than or equal to a preset depth threshold) and the corresponding mask of the semantic segmentation image is taken. It is then determined whether obstacle semantics exist in the intersection. If no obstacle semantics exist, the processing can end; for example, a prompt indicating no obstacle can be output, eliminating the need for further processing and reducing resource consumption. If obstacle semantics exist, the binocular depth image and semantic segmentation image are projected onto voxels. Each voxel is assigned a semantic label and occupancy probability. When the semantic label of a voxel is a dynamic obstacle (e.g., pedestrian, vehicle, animal), the movement speed of the dynamic obstacle and its predicted trajectory are determined. If the movement speed of the dynamic obstacle is within a preset speed range, the dynamic obstacle is determined to be stationary, and the process proceeds accordingly. Formula calculation determined Then based on The system expands the corresponding area of ​​stationary dynamic obstacles identified as obstacle avoidance targets to determine the obstacle avoidance zone. If the speed of the dynamic obstacle is outside the preset speed range, the system determines that the dynamic obstacle is moving and proceeds accordingly. Formula calculation determined Then based on The system predicts the motion trajectory and uses the predicted trajectory as the center line to expand the corresponding area of ​​the moving dynamic obstacle to determine the obstacle avoidance area corresponding to the dynamic obstacle.

[0079] When the semantic label of a voxel is a static obstacle (such as a rock, bush, or tree), if the lowest height of the voxel corresponding to the static obstacle is less than or equal to the highest height of the lawnmower robot, then the static obstacle is determined to be an obstacle avoidance target; if the lowest height of the voxel corresponding to the static obstacle is greater than the highest height of the lawnmower robot, it indicates that the static obstacle is a floating obstacle, and the lawnmower robot can pass through it normally. In this case, it can be based on... Only the lowest point of the static obstacle's outline in the 3D world is expanded. Then, it's determined whether the expanded minimum height of the corresponding voxel of the static obstacle is greater than the maximum height of the lawnmower robot. If the expanded minimum height is greater than the lawnmower robot's maximum height, the lawnmower robot can still pass normally, and in this case, the static obstacle is determined not to be an obstacle avoidance target. Conversely, if the expanded minimum height is less than or equal to the lawnmower robot's maximum height, the static obstacle is determined to be an obstacle avoidance target. If the static obstacle is determined to be an obstacle avoidance target, then... Formula calculation determined ,based on The area corresponding to static obstacles is expanded to determine the obstacle avoidance zone. By determining the appropriate range of the obstacle avoidance zone based on the obstacle conditions, the accuracy of obstacle avoidance zone determination is improved, thereby reducing the collision risk of Robot 1000 and enhancing safety.

[0080] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the obstacle avoidance processing method.

[0081] The computer readable storage medium can be an internal storage unit of the obstacle avoidance processing device or the robot, for example, a hard disk or a memory of the obstacle avoidance processing device or the robot. The computer readable storage medium can also be an external storage device of the obstacle avoidance processing device or the robot, for example, a plug-in hard disk, a smart media card (SMC), a secure digital card (SD card), a flash card or the like.

[0082] The computer program stored in the storage medium can execute any of the obstacle avoidance processing methods provided by the embodiments of the present application, and thus the beneficial effects of any of the obstacle avoidance processing methods provided by the embodiments of the present application can be achieved. Details are described in the foregoing embodiments, and thus will not be described here.

[0083] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0084] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. An obstacle avoidance method, characterized in that, include: Perform semantic segmentation on the target image to identify the category of obstacles; If the obstacle is classified as a dynamic obstacle, then the movement speed of the dynamic obstacle and its predicted movement trajectory within a preset time period are determined. Based on the predicted motion trajectory and the motion speed, the region corresponding to the dynamic obstacle is expanded to determine the obstacle avoidance region corresponding to the dynamic obstacle.

2. The obstacle avoidance method according to claim 1, characterized in that, The step of expanding the region corresponding to the dynamic obstacle based on the predicted motion trajectory and the motion speed to determine the obstacle avoidance region corresponding to the dynamic obstacle includes: Based on the speed of movement, the motion state of the dynamic obstacle is determined; Determine the expansion radius based on the described motion state; Based on the predicted motion trajectory and the expansion radius, the region corresponding to the dynamic obstacle is expanded to determine the obstacle avoidance area corresponding to the dynamic obstacle.

3. The obstacle avoidance method according to claim 2, characterized in that, Determining the speed of the dynamic obstacle includes: Based on the translational velocity model, the feature points of the dynamic obstacle are distinguished as abnormal motion feature points and normal motion feature points. Calculate the average velocity corresponding to the normal motion feature points, and determine the average velocity as the motion velocity of the dynamic obstacle; Determining the motion state of the dynamic obstacle based on the motion speed includes: If the speed of movement is within a preset speed range, then the dynamic obstacle is determined to be stationary; If the movement speed is outside the preset speed range, then the dynamic obstacle movement is determined.

4. The obstacle avoidance method according to claim 3, characterized in that, The translational velocity model distinguishes between abnormal and normal motion feature points of the dynamic obstacle, including: Obtain the velocity vectors corresponding to each feature point of the dynamic obstacle; Calculate the residual between the velocity vector corresponding to each feature point and the velocity corresponding to the translational velocity model; If the residual is greater than a preset value, then the feature point corresponding to the residual is determined to be the motion anomaly feature point; If the residual is less than or equal to the preset value, then the feature point corresponding to the residual is determined to be a normal motion feature point.

5. The obstacle avoidance method according to claim 2, characterized in that, Determining the expansion radius based on the motion state includes: If the dynamic obstacle is stationary, then determine whether the dynamic obstacle is an obstacle to avoid; If the dynamic obstacle is an obstacle to avoid, then according to the formula Determine the first expansion radius; in, The first expansion radius corresponds to the stationary dynamic obstacle. As a reference radius, It is the first expansion coefficient.

6. The obstacle avoidance method according to claim 2, characterized in that, Determining the expansion radius based on the motion state includes: If the dynamic obstacle moves, then according to the formula Determine the second expansion radius; in, The second expansion radius corresponds to the moving dynamic obstacle. As a reference radius, The second coefficient of thermal expansion. The current velocity of the dynamic obstacle. To preset a safe response time, To adjust the parameters, For predicting time.

7. The obstacle avoidance method according to claim 1, characterized in that, The process of semantic segmentation of the target image and identification of obstacle categories includes: If the obstacle is classified as a static obstacle, the corresponding area of ​​the static obstacle is expanded based on the third expansion radius to determine the obstacle avoidance area corresponding to the static obstacle.

8. The obstacle avoidance method according to claim 7, characterized in that, Before determining the obstacle avoidance area corresponding to the static obstacle by expanding the region corresponding to the static obstacle based on the third expansion radius, the process includes: Determine whether the static obstacle is an obstacle to avoid; If the static obstacle is an obstacle to be avoided, then the third expansion radius is determined.

9. A robot, characterized in that, The robot includes a vision device, a walking device, and an obstacle avoidance device; wherein the walking device is used to control the movement of the robot; the vision device is used to acquire images of the robot's working environment; the obstacle avoidance device includes a processor and a memory, the memory storing a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the steps of the obstacle avoidance method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the obstacle avoidance processing method as described in any one of claims 1 to 8.

Images