Quadruped robot obstacle avoidance method and system based on cooperation of laser radar and depth camera

Abstract

The invention discloses a quadruped robot obstacle avoidance method and system based on cooperation of a laser radar and a depth camera, and relates to the technical field of robot obstacle avoidance. The system comprises a quadruped robot, and the quadruped robot is provided with an industrial personal computer, a laser radar, a laser radar protection device and a depth camera. When the robot walks, the laser radar obstacle avoidance method is executed firstly, when a laser radar obstacle avoidance program fails due to the conditions of large blind areas and the like, the robot continues to walk until the laser radar obstacle avoidance program appears in the detection range of the depth camera, the depth camera obstacle avoidance method is executed for obstacle avoidance judgment again, and selection is made. The laser radar and the depth camera are combined for use to realize obstacle avoidance, and the depth camera is used as a blind compensation means and supplementation of laser obstacle avoidance, so that the safety is further improved. Processing depth data acquired by the depth camera by using a random forest; the operation amount is small and the processing speed is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot obstacle avoidance, and relates to a method and system for avoiding obstacles of a quadruped robot in coordination with a laser radar and a depth camera. Background Art

[0002] In recent years, with the rapid development of robotics technology, quadruped robots have shown great potential in the fields of line inspection, military reconnaissance, disaster relief, and home services. However, the existing quadruped robots still face many challenges in their anti-collision capabilities in complex environments, such as the depth camera of the perception layer being greatly affected by light, the high cost of laser radar, blind spots in detection, inaccurate obstacle recognition, and insufficient safety protection capabilities. Summary of the invention

[0003] In order to make up for the deficiencies of the prior art, the present invention provides a method and system for avoiding obstacles for a quadruped robot using a laser radar and a depth camera. The present invention uses a laser radar and a depth camera in combination to achieve obstacle avoidance. The depth camera is used as a blind spot compensation method and a supplement to laser obstacle avoidance to further improve safety. A random forest is used to process the depth data obtained by the depth camera; the amount of calculation is small and the processing speed is fast.

[0004] The present invention is implemented by the following technical solution: The present invention provides a quadruped robot obstacle avoidance method in which a laser radar and a depth camera are coordinated, comprising: LiDAR obstacle avoidance method: Program start, LiDAR data acquisition, using LiDAR as a sensor to acquire LiDAR data; Pre-filter the laser radar data. In the robot operating system ROS, the laser radar data is an array of length n, and each value in the array is a distance information; traverse this array, judge by distance, delete the values ​​in the array with a distance greater than 100cm and less than 10cm, and retain the data in the key area, which is pre-filtering; Data cropping is to further filter the pre-filtered data. For robot obstacle avoidance, it is only necessary to obtain the presence or absence of obstacles within the range of 60 degrees HFOV (horizontal field of view) and -16 to 15 degrees VFOV (vertical field of view) in front of the robot. The pre-filtered data is traversed to determine which values ​​are within this range by angle. The data that affects the obstacle avoidance range is retained. Determine whether there is an obstacle in front of the robot. If not, continue to move forward until the program ends; If there is an obstacle in front of the robot within the threshold, the robot will stop and wait for 30 seconds, and then continue to determine whether the obstacle still exists. If not, it will continue to move forward until the program ends. If the obstacle still exists, it will return and re-plan the route; Depth camera obstacle avoidance method: Program start; The depth camera is used as a sensor to obtain depth data directly in front of and below the quadruped robot; Filter out areas that affect traffic; Evenly extract a number of point cloud data from the area; Use random forest terrain to determine whether it is passable; The uniformly extracted depth data are passed as parameters to the random forest model, which gives classification results, including obstacles, passable roads and cliffs; If the obstacle in front of the robot is determined to be a cliff, the robot returns and replans the route to avoid this stop; If the robot is judged to be on a passable road ahead, it will continue to move forward until the program ends; If it is determined that there is an obstacle in front of the robot, the robot will stop and wait for 30 seconds, then judge again. If the obstacle still exists, it will return to the last stop and re-plan the route. If the obstacle disappears, it will continue to move forward until the program ends; The laser radar obstacle avoidance method is executed first. When the laser radar obstacle avoidance program fails due to a large blind spot or other reasons, the robot will continue to walk until it appears in the detection range of the depth camera. Then the depth camera obstacle avoidance method is executed to make an obstacle avoidance judgment again and make a choice.

[0005] A quadruped robot obstacle avoidance system in which laser radar and depth camera cooperate, comprises a quadruped robot, on which an industrial computer, a laser radar, a laser radar protection device and a depth camera are installed; the laser radar is connected to the industrial computer via an RJ45 interface, the depth camera is connected to the industrial computer via a universal serial bus USB, and the industrial computer sends control commands to the quadruped robot control module via the RJ45 interface after performing random forest prediction processing; the laser radar is fixed to the head of the quadruped robot via a threaded connection, and the depth camera is fixed to the bottom of the head of the quadruped robot via a bayonet connection; the laser radar protection device is arranged on the outer ring of the laser radar, and is fixed to the head of the quadruped robot via a threaded connection, so as to protect the laser radar.

[0006] The beneficial effects of the present invention are: The present invention integrates laser radar and depth camera data to identify obstacles, road conditions and other information of the quadruped robot in a complex and changeable environment, and realizes the functions of obstacle avoidance, collision prevention and fall prevention, thereby improving the autonomous operation capability and safety of the quadruped robot.

[0007] The present invention integrates laser SLAM mapping, intelligent path planning, laser radar and camera joint obstacle detection, collision prevention, and a quadruped robot system to improve the robot's autonomous navigation capability and safety protection performance in complex environments.

[0008] Most existing technologies only use a single sensor for perception without redundant backup. The present invention uses two different sensors, laser radar and depth camera. The two sensors have different principles and will not be interfered at the same time. The two obstacle avoidance methods are processed in parallel and complement each other. The random forest model is used for prediction, which takes up less computing power than the neural network-based processing method used in the existing technology, has lower hardware cost, faster processing speed, and better anti-overfitting effect.

[0009] The present invention uses a depth camera, which is installed below the laser radar, and the direction of the depth camera is 45 degrees diagonally downward in front of the robot; the blind spot problem of the laser radar is supplemented based on the data of the depth camera.

[0010] The present invention uses a laser radar and a depth camera to achieve obstacle avoidance. The depth camera is used as a blind spot compensation method and a supplement to the laser obstacle avoidance to further improve safety. The depth data obtained by the depth camera is processed using a random forest; the amount of calculation is small and the processing speed is fast. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be further described below in conjunction with the accompanying drawings; Figure 1 This is a flow chart of the laser radar obstacle avoidance method of the present invention; Figure 2 This is a flow chart of the obstacle avoidance method of the depth camera of the present invention; Figure 3 This is a main structural diagram of the quadruped robot of the present invention; Figure 4 A schematic diagram of uniformly extracting depth data according to the present invention; Figure 5 This is a diagram showing the range of interest of the depth camera of the present invention; In the picture, 1 is an industrial computer, 2 is a laser radar, 3 is a laser radar protection device, and 4 is a depth camera. DETAILED DESCRIPTION

[0012] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0013] The accompanying drawings are specific embodiments of the quadruped robot obstacle avoidance method and system in which the laser radar and depth camera are coordinated according to the present invention.

[0014] An embodiment of a quadruped robot obstacle avoidance method using a laser radar and a depth camera, including a laser radar obstacle avoidance method: The program starts.

[0015] LiDAR data acquisition, uses LiDAR as a sensor to acquire LiDAR data.

[0016] Pre-filter the LiDAR data. In the robot operating system ROS, the LiDAR data is an array of length n. The size of n depends on the performance of the LiDAR. Each value in the array is a distance information. Traverse the array and delete the values ​​in the array with a distance greater than 100 cm and less than 10 cm by distance judgment, and retain the data in the key area. This is called pre-filtering.

[0017] Data cropping is to further screen the pre-filtered data. The index of each value in the LiDAR array is related to the angle corresponding to this value. The depth information at which angle this value corresponds is analyzed through the number of LiDAR lines, angular resolution and array length. To achieve robot obstacle avoidance, it is only necessary to obtain whether there are obstacles within the range of the robot's HFOV (horizontal field of view) of 60 degrees and VFOV (vertical field of view) of -16 to 15 degrees. The pre-filtered data is traversed to determine which values ​​are within this range by angle. The data within the range that affects obstacle avoidance is retained.

[0018] Determine whether there is an obstacle in front of the robot. If not, continue to move forward until the program ends. If there is an obstacle in front of the robot within the threshold, the robot stops and waits for 30 seconds, then continues to determine whether the obstacle still exists. If not, continue to move forward until the program ends. If the obstacle still exists, return and re-plan the route.

[0019] The threshold is determined by the blind spot of the laser radar. The threshold of the laser radar in this embodiment is calculated to be 1.4 meters through the height of the laser radar from the ground and the downward field of view angle of the laser radar.

[0020] If there is an obstacle within 1.4 meters in front of the robot for more than 30 seconds, considering that people or other movable obstacles may disappear within 30 seconds, the robot is considered impassable and returns to the last stop, deletes this stop, and replans the path to bypass this impassable stop; if the obstacle disappears within 30 seconds, it resumes moving forward.

[0021] Depth camera obstacle avoidance method: The program starts.

[0022] The depth camera is used as a sensor to obtain depth data directly in front of and slightly below the quadruped robot.

[0023] Filter out areas that affect traffic.

[0024] Evenly extract some point cloud data from the area. The specific method of evenly extracting depth data is as follows: Figure 4 A point cloud data is taken from the center of each small white box, thus forming a set of point cloud data. This data is used as the basis for random forest training and judgment.

[0025] The random forest terrain is used to determine whether the road is passable.

[0026] Random forest is a machine learning algorithm that processes large-scale, high-dimensional, missing or unbalanced data sets, and is used to solve classification and regression problems. Several uniformly extracted depth data are passed as parameters to the random forest model, which gives classification results, including obstacles, drivable roads, and cliffs.

[0027] If the obstacle in front of the robot is determined to be a cliff, the robot returns and replans the route to avoid this stop; If the robot is judged to be on a passable road ahead, it will continue to move forward until the program ends; If it is determined that there is an obstacle in front of the robot, the robot will stop and wait for 30 seconds, then judge again. If the obstacle still exists, it will return to the last stop and re-plan the route. If the obstacle disappears, it will continue to move forward until the program ends.

[0028] The information obtained by the lidar is depth information, which is represented as a point cloud. The processing of lidar data is the processing of point cloud data. The point cloud data that the quadruped robot cannot cross in front of it is found and clustered into line segments or circles. The obstacle information obtained from the processed data is added to the obstacle map. After data cropping, the system has taken out the data that the robot is concerned about obstacle avoidance. If there is an obstacle in this area, it is added to the obstacle map in the robot operating system ROS, and a local path planning algorithm is used to implement local path planning based on this map to avoid obstacles.

[0029] When the obstacle is in the blind spot of the LiDAR or the LiDAR obstacle avoidance method fails, the quadruped robot will continue to walk until the obstacle appears in the detection range of the depth camera. At this time, the depth camera will obtain depth data, evenly extract several valid values ​​from the field of view of the depth camera, and use the pre-trained random forest model to make predictions to determine whether there are obstacles or obstacle terrain in the field of view of the depth camera, and issue commands to stop, cross obstacles, bypass obstacles, or return.

[0030] The training steps of the random forest model include: moving the robot to obstacles, cliffs, and passable roads. Obstacles, cliffs, and passable roads are called labels. The types of labels can be increased as needed. Considering the completeness of the data set, each label should cover various situations that the robot may encounter. Use a depth camera to collect depth data in these situations, and evenly extract several points from it. The extracted points and corresponding labels are made into a data set; the data set is a file with the suffix .csv, and each line of the file consists of depth information and its corresponding label. The data set file path is passed as a parameter to the training program to train the random forest model.

[0031] When using a depth camera to fill in the blind spots; after acquiring the point cloud data, first remove the invalid values, as follows Figure 5 The depth camera focus range diagram shown in the figure removes the data from the vertical 1 / 2 of the depth image to the top of the image, and only retains the data in the red box in the lower half. The purpose is to reduce the amount of calculation and speed up the processing; find out the points in this area that the quadruped robot cannot cross and affect the passage, use the clustering algorithm to abstract these points into line segments or circles, and use them as abstract representations of obstacles, and map them to the obstacle map to facilitate obstacle avoidance or parking in the navigation program.

[0032] As attached Figure 3 As shown, a quadruped robot obstacle avoidance system in which a laser radar and a depth camera cooperate, comprises a quadruped robot, on which an industrial computer 1, a laser radar 2, a laser radar protection device 3 and a depth camera 4 are installed; the laser radar is connected to the industrial computer via an RJ45 interface, and the depth camera is connected to the industrial computer via a universal serial bus USB, and the industrial computer sends a control command to the quadruped robot control module via the RJ45 interface after performing random forest prediction processing; the laser radar is fixed to the head of the quadruped robot via a threaded connection, and the depth camera is fixed to the bottom of the head of the quadruped robot via a bayonet connection; the laser radar protection device is arranged on the outer ring of the laser radar, and is fixed to the head of the quadruped robot via a threaded connection, so as to protect the laser radar.

[0033] The depth camera is installed below the laser radar, and the direction of the depth camera is 45 degrees diagonally downward in front of the quadruped robot; the depth data obtained by the depth camera supplements the blind spot problem of the laser radar.

[0034] The industrial computer includes a data processing unit, which is used to process the lidar data and the depth data obtained by the depth camera, perform random forest prediction, determine whether the current road surface is a communicative road surface, a cliff, or an obstacle, and issue control commands.

[0035] Laser radar is a device for obtaining distance information. One of the forms of expressing distance information is point cloud data. The detection range of the laser radar is related to the wiring harness and installation position of the laser radar. In the present invention, the laser radar is installed at the head position in front of the body of the quadruped robot.

[0036] During the navigation process, if the quadruped robot encounters an insurmountable obstacle on the road, the quadruped robot should make the following choices: 1. Stop, 2. Return to the last navigation point and re-plan the route, 3. Detour. To achieve these controls, you must first sense the existence of obstacles. The present invention uses laser radar and depth camera as sensors. The two sensors complement each other. The laser radar has a large detection range and a wide field of view. As the main sensor, the depth camera is used as a blind spot sensor for the laser radar. In the obstacle avoidance scheme, the laser radar obstacle avoidance method is executed first. When the laser radar obstacle avoidance program fails due to a large blind spot, the robot will continue to walk until it appears in the detection range of the depth camera, and then execute the depth camera obstacle avoidance method to make an obstacle avoidance judgment again and make a choice. Since the two sensors work on different principles and the probability of being interfered with is low, the probability of successful obstacle avoidance is also greatly improved.

[0037] The information obtained by the laser radar is depth information, which is expressed as a point cloud. The processing of the laser radar data is the processing of the point cloud data. The point cloud data is not as fully developed as the image processing technology, and there are fewer processing and analysis methods. Therefore, this method does not use the laser radar as the main obstacle avoidance sensor. The present invention only performs simple distance and angle filtering on the laser radar data to obtain the depth information of a small area in front of the robot. This area is the area that affects the robot's progress. It is determined whether there are obstacles in this small area. If there are obstacles, the obstacle avoidance processing program is entered. If the laser radar obstacle avoidance method fails, the depth camera sensor of this scheme becomes a supplement to the laser radar scheme at this time. The data of the depth camera includes both image data and point cloud data. After obtaining the point cloud data of the depth camera, a number of depth points are uniformly taken out from the field of view of the depth camera; uniform here means that the points taken out in this way will not appear dense and concentrated in a certain area, while sparse in other areas; using the pre-trained random forest model, prediction is performed to determine whether there are obstacles or obstacle terrains in the field of view of the depth camera, such as deep pits, cliffs and other terrains.

[0038] Existing path planning algorithms usually use the shortest path or the shortest time as the evaluation algorithm for the optimal path, and the planned path will not consider terrain factors. This may cause the robot to cross multiple terrains during walking. The present invention implements a path planning algorithm with straight lines between the stop points, which is manifested as the path planned by the robot in the inspection task being connected by line segments connected by the stop points. The stop points are represented as xy coordinates on the map coordinate system. Since the stop points are manually selected, the construction personnel will consider the road surface passability when creating them, so most of the stop points are planned on paved roads or terrain that is easy for robots to pass. When creating an inspection task, it will include a series of stop points composed of xy coordinates. The path planning algorithm will connect these stop points in series and assign serial numbers, and the robot will run towards these stop points in sequence. In this way, the overlap between the planned path and the paved road surface at the inspection site is greatly improved, reducing the frequency of the robot planning a route that is difficult to walk.

[0039] Integrated laser SLAM mapping means that the quadruped robot uses laser radar as the main sensor and is equipped with a matching SLAM algorithm to achieve the function of drawing maps in unknown environments. The SLAM algorithm is used to achieve robot positioning and draw environmental maps on this basis.

[0040] Intelligent path planning means that after obtaining the environment map through the lidar sensor and slam algorithm, the robot adds multiple stops based on this environment map and task requirements, uses the path planning algorithm to plan an optimal path, and passes through these stops in sequence.

[0041] The present invention has an effective anti-fall mechanism. The depth camera can obtain image data and depth data. One of the presentation forms of the depth data is point cloud. First, the semantic segmentation method in image data processing is used to finely understand and analyze the image content, and paved roads and different terrains are classified; if it is an unpaved road, deep information processing based on random forest is performed to predict whether it is a cliff. If so, obstacle stop processing is entered. Obstacle stop processing means that the robot stops immediately and returns to the original route after a period of time.

[0042] The effective obstacle avoidance mechanism of the present invention: the present invention adopts two sensors, a laser radar and a depth camera, to obtain the information of detecting obstacles. The two obstacle avoidance schemes run in parallel and complement each other. When the laser radar obstacle avoidance scheme fails, the depth camera scheme becomes a supplement to the laser radar scheme; the reliability of anti-collision is greatly improved.

Claims

1. A quadruped robot obstacle avoidance method using laser radar and depth camera, characterized in that: include LiDAR obstacle avoidance method: Program start, LiDAR data acquisition, using LiDAR as a sensor to acquire LiDAR data; Pre-filter the laser radar data. In the robot operating system ROS, the laser radar data is an array of length n, and each value in the array is a distance information; traverse this array, judge by distance, delete the values ​​in the array with a distance greater than 100cm and less than 10cm, and retain the data in the key area, which is pre-filtering; Data cropping is to further filter the pre-filtered data. For robot obstacle avoidance, it is only necessary to obtain the presence or absence of obstacles within the range of 60 degrees HFOV horizontal field of view and -16 to 15 degrees VFOV vertical field of view directly in front of the robot. The pre-filtered data is traversed to determine which values ​​are within this range by angle. The data that affects the obstacle avoidance range is retained. Determine whether there is an obstacle in front of the robot. If not, continue to move forward until the program ends; If there is an obstacle in front of the robot within the threshold, the robot will stop and wait for 30 seconds, and then continue to determine whether the obstacle still exists. If not, it will continue to move forward until the program ends. If the obstacle still exists, it will return and re-plan the route; Depth camera obstacle avoidance method: Program start; The depth camera is used as a sensor to obtain depth data directly in front of and below the quadruped robot; Filter out areas that affect traffic; Evenly extract a number of point cloud data from the area; Use random forest terrain to determine whether it is passable; The uniformly extracted depth data are passed as parameters to the random forest model, which gives classification results, including obstacles, passable roads and cliffs; If the obstacle in front of the robot is determined to be a cliff, the robot returns and replans the route to avoid this stop; If the robot is judged to be on a passable road ahead, it will continue to move forward until the program ends; If it is determined that there is an obstacle in front of the robot, the robot will stop and wait for 30 seconds, then judge again. If the obstacle still exists, it will return to the last stop and re-plan the route. If the obstacle disappears, it will continue to move forward until the program ends; The laser radar obstacle avoidance method is executed first. When the laser radar obstacle avoidance program fails due to a large blind spot or other reasons, the robot will continue to walk until it appears in the detection range of the depth camera. Then the depth camera obstacle avoidance method is executed to make an obstacle avoidance judgment again and make a choice.

2. The laser radar and depth camera coordinated quadruped robot obstacle avoidance method according to claim 1, characterized in that: The information obtained by the laser radar is depth information, which is represented as a point cloud. The processing of the laser radar data is the processing of the point cloud data. The point cloud data that the quadruped robot cannot cross in front of it is found and clustered into line segments or circles. The obstacle information obtained from the processed data is added to the obstacle map. After data trimming, the system has extracted the data that the robot is concerned about in obstacle avoidance. If there are obstacles in this area, they will be added to the obstacle map in the robot operating system ROS. The local path planning algorithm will be used to implement local path planning based on this map to avoid obstacles.

3. The laser radar and depth camera coordinated quadruped robot obstacle avoidance method according to claim 1, characterized in that: When the obstacle is in the blind spot of the LiDAR or the LiDAR obstacle avoidance method fails, the quadruped robot will continue to walk until the obstacle appears in the detection range of the depth camera. At this time, the depth camera will obtain depth data, evenly extract several valid values ​​from the field of view of the depth camera, and use the pre-trained random forest model to make predictions to determine whether there are obstacles or obstacle terrain in the field of view of the depth camera, and issue commands to stop, cross obstacles, bypass obstacles, or return.

4. The laser radar and depth camera coordinated quadruped robot obstacle avoidance method according to claim 1, characterized in that: The training steps of the random forest model include: moving the robot to obstacles, cliffs, and passable roads, using a depth camera to collect depth data in these situations, and evenly extracting several points from them, making a data set with the extracted points and corresponding labels; passing the data set file path as a parameter to the training program to train the random forest model.

5. The laser radar and depth camera coordinated quadruped robot obstacle avoidance method according to claim 1, characterized in that: When using a depth camera for blind spot filling; after obtaining the point cloud data, first remove the invalid values, and crop the area that affects the walking of the quadruped robot from the valid values, that is, from the vertical 1 / 2 of the depth image to the bottom of the image; find out the points in this area that the quadruped robot cannot cross and that affect the passage, and use the clustering algorithm to abstract these points into line segments or circles as an abstract representation of obstacles, and map them to the obstacle map to facilitate obstacle avoidance or parking in the navigation program.

6. The laser radar and depth camera-coordinated quadruped robot obstacle avoidance system is characterized by: It includes a quadruped robot, which is equipped with an industrial computer, a laser radar, a laser radar protection device and a depth camera; the laser radar is connected to the industrial computer through an RJ45 interface, and the depth camera is connected to the industrial computer through a universal serial bus USB. After the industrial computer performs random forest prediction processing, it sends a control command to the quadruped robot control module through the RJ45 interface; the laser radar is fixed to the head of the quadruped robot through a threaded connection, and the depth camera is fixed to the bottom of the head of the quadruped robot through a bayonet connection; the laser radar protection device is arranged on the outer ring of the laser radar, and is fixed to the head of the quadruped robot through a threaded connection, so as to protect the laser radar.

7. The laser radar and depth camera coordinated quadruped robot obstacle avoidance system according to claim 6, characterized in that: The depth camera is installed below the laser radar, and the direction of the depth camera is 45 degrees diagonally downward in front of the quadruped robot.

8. The laser radar and depth camera coordinated quadruped robot obstacle avoidance system according to claim 6, characterized in that: The industrial computer includes a data processing unit, which is used to process the lidar data and the depth data obtained by the depth camera, perform random forest prediction, determine whether the current road surface is a communicative road surface, a cliff, or an obstacle, and issue control commands.

Images