Robot anti-falling detection method based on depth camera and single-point laser

Through the combination of depth cameras and single-point laser sensors, real-time detection of the robot environment is achieved, solving the problem of insufficient detection accuracy of traditional methods in complex scenarios, and improving the safety and detection accuracy of the robot in stair environments.

CN119017431AActive Publication Date: 2024-11-26SHENYANG SIASUN ROBOT & AUTOMATION
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310599313.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-26
Estimated Expiration
2043-05-25

AI Technical Summary

Technical Problem

Traditional robot anti-fall detection methods have insufficient detection accuracy in complex scenarios and cannot effectively identify transient environments such as stairs, resulting in safety hazards of robots and may lead to overturn or fall.

Method used

Using a detection method based on depth camera and single-point laser, the environment data is periodically collected through the depth camera and laser sensor, combined with depth information and grayscale image processing, determine whether there is a fall environment, and feedback it to the robot main controller in real time to stop the operation. .

Benefits of technology

It improves the safety of the robot in complex scenarios, prevents overturning caused by the stair environment, enhances detection accuracy, avoids hidden dangers of a single detection method, and ensures stable operation of the robot.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119017431A_ABST
    Figure CN119017431A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of robot visual processing, and particularly relates to a robot anti-falling detection method based on a depth camera and a single-point laser, which comprises the following steps: a robot main controller receives environmental data collected by the depth camera and a laser sensor respectively, and performs data processing analysis on the environmental data respectively; judging whether the environment in front of the distribution robot is a falling environment or not; if one of the processed depth camera acquisition data or the laser sensor acquisition data is judged to be a falling environment, judging that the environment in front of the distribution robot is the falling environment, and feeding back the current dangerous state of the robot to the background server, and stopping the operation of the distribution robot by the background server; if the processed data collected by the depth camera or the data collected by the laser sensor are not judged to be in a falling environment, operation continues. Under the complex application scene of the robot, the situation that when the robot advances or retreats, due to the height transient environment such as stairs, the robot turns over during running, and losses are caused is prevented.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of robot vision processing, and in particular is a robot anti-fall detection method based on a depth camera and a single-point laser. Background Art

[0002] With the progress of society and the continuous development of technology, intelligent robots have also played an important role in people's lives, and robots can be seen in various public places. Some application scenarios are better, involving only the avoidance and obstacle handling of some people or objects, while some application scenarios may have stairs, escalators, and pits, which cannot be detected by the horizontally installed chassis laser. Therefore, there will be safety hazards when the robot operates in this scenario. It is necessary to use external sensor equipment to detect environments such as stairs, stop the robot operation, and feedback the dangerous status, waiting for the upper computer logic processing.

[0003] The traditional method of detecting and preventing falls is to install an anti-fall detection sensor on the robot chassis and calculate the distance through analog signals. The detection accuracy is not enough, which will affect the emergency stop effect of the robot and cause it to tip over or fall down the stairs. Summary of the invention

[0004] The purpose of the present invention is to provide a method for anti-fall detection based on a depth camera and a single-point laser, so as to prevent the robot from tipping over and causing losses when moving forward or backward due to a highly transient environment such as stairs.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a method for robot anti-fall detection based on a depth camera and a single-point laser, comprising the following steps:

[0006] 1) Install a depth camera and a laser sensor on the delivery robot, and connect the depth camera and the laser sensor to the robot main controller, so that during the driving process of the delivery robot, the depth camera and the laser sensor periodically collect environmental data and send it to the robot main controller;

[0007] 2) The robot main controller receives the environmental data collected by the depth camera and the laser sensor respectively, and processes and analyzes the environmental data respectively to determine whether the environment in front of the delivery robot is a falling environment;

[0008] 3) If one of the processed depth camera data or laser sensor data is determined to be a falling environment, the front of the delivery robot is determined to be a falling environment, and the current dangerous state of the robot is fed back to the background server, and the background server stops the delivery robot operation;

[0009] 4) If the processed depth camera data or laser sensor data are determined not to be a falling environment, the delivery robot continues to operate.

[0010] The depth camera and the laser sensor are both installed on the same horizontal mounting surface on the top of the delivery robot, and the mounting heights are consistent;

[0011] The field of view direction of the depth camera and the laser emission direction of the laser sensor are arranged in the same direction as the travel direction of the delivery robot;

[0012] The laser sensor irradiates obliquely downward, and the angle formed between the laser emission direction of the laser sensor and the horizontal plane is 45°;

[0013] The depth camera is installed at an angle of 45° and shines obliquely downward. The field of view of the depth camera covers an area 500m in front of the delivery robot.

[0014] The laser sensor is a laser radar detection module, which is used to collect laser data of the environment in front of the delivery robot.

[0015] The laser data includes: laser detection distance data and reflectivity data.

[0016] In step 2), the robot main controller receives the environmental data collected by the depth camera and performs data processing and analysis to determine whether the delivery robot is in a falling environment, specifically:

[0017] The depth camera captures image frames and reads depth information, and sends the depth information to the robot main controller;

[0018] The robot main controller converts the depth information into three-dimensional data information relative to the center of the delivery robot, namely:

[0019] x=Contour[i].fx+CamParam.HDis

[0020] y=Contour[i].fy+CamParam.VDis

[0021] z=Contour[i].fz+CamParam.ZDis

[0022] Where x, y, and z are the coordinates relative to the center of the robot, Contour[i].fx, Contour[i].fy, and Contour[i].fz are the coordinate values ​​of the i-th visual data, CamParam.HDis is the lateral offset of the depth camera installation position, CamParam.VDis is the longitudinal offset of the depth camera installation position, and CamParam.ZDis is the protrusion offset of the depth camera installation position;

[0023] The robot main controller determines the y coordinate data based on the y coordinate data in the three-dimensional data information of the robot center. When the y coordinate data is greater than the set height threshold, it is determined that the environment in front is a falling environment. Otherwise, the robot main controller continues to control the depth camera to collect image frames in real time.

[0024] In step 2), the robot main controller receives the environmental data collected by the laser sensor and performs data processing and analysis to determine whether the environment in front of the delivery robot is a falling environment, specifically:

[0025] The laser sensor collects laser detection distance data and reflectivity data of the environment in front of the delivery robot and sends it to the robot main controller;

[0026] The robot main controller selects the laser detection distance data of two adjacent cycles that meet the preset reflectivity range according to the preset reflectivity range and makes a difference, that is:

[0027] temp=dis[i]-dis[i-1]

[0028] Where i is the number of cycles, dis[i] is the laser detection distance data of the i-th cycle;

[0029] When the laser detection distance data difference temp is greater than 100mm within n consecutive cycles, it is determined that there is a falling environment in front of the delivery robot; otherwise, it is determined that there is no falling environment in front, and the robot main controller continues to control the laser sensor to cyclically collect environmental data.

[0030] It also includes: when the falling environment is a staircase environment, the depth camera cannot determine the environment in front of the delivery robot alone, and the robot main controller performs the following steps:

[0031] The information collection of the stair environment is obtained by combining the grayscale image with the depth image. When the depth information contains a number less than 0, the grayscale image of the staircase in the current frame is collected. The grayscale image is first binarized, and the image is blurred by Gaussian filtering. Then, the grayscale image is subjected to Hough feature transformation to extract the lines and edges in the environment.

[0032] After the Hough feature transformation, multiple repeated lines are obtained. According to the parallel line characteristics of the stairs and the distance standards between them, the coordinates of the lines are compared and judged, and then the extracted lines are screened and filtered to extract the line contour information of the stairs;

[0033] Based on the judgment of the grayscale image and depth image information, when the depth value is greater than the height threshold and there is a contour line in the grayscale image that meets the characteristics of the stairs, it is considered that the front is a falling environment.

[0034] The present invention has the following beneficial effects and advantages:

[0035] 1. In complex robot application scenarios, the present invention prevents the robot from tipping over and causing losses when the robot is moving forward or backward due to a highly transient environment such as stairs.

[0036] 2. The present invention utilizes depth information and a laser sensor to collect and process data in a cyclic manner, and uses one of the two to determine the falling environment, thereby preventing a situation in which a single method cannot be used to identify the falling environment.

[0037] 3. The present invention greatly increases the accuracy of detection by combining depth information with grayscale images, thereby preventing the hidden danger of a falling environment being undetected by both depth information and laser sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the overall logic flow of detecting a falling environment implemented by the present invention;

[0039] Figure 2 This is a schematic diagram of the cycle flow of the depth camera of the present invention;

[0040] Figure 3 It is a schematic diagram of the cycle flow of the laser sensor of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0042] The present invention uses the depth data value obtained by the depth camera to detect the information of the ground cracks, stairs and other places below the ground level in front of the robot, and sends the information to the robot main controller, so that the delivery robot can avoid danger and operate stably. The principle of the depth camera is that the infrared transmitting end of the camera sends a fixed wavelength, encoded infrared signal, and the infrared receiving end receives the returned signal to obtain the obstacle distance information. Therefore, for scenes with obvious infrared interference and light source reflection interference, the data information obtained by the depth camera is inaccurate;

[0043] Therefore, the present invention integrates a single-point laser sensor on the basis of the depth camera to detect 500mm in front of the robot. According to the detection distance and reflectivity data returned by the sensor, it is analyzed and judged that when the reflectivity meets the normal value range and the distance is greater than the rated range, it is considered that there is a falling environment in front of the robot, which plays a role in fall protection. After the depth camera or laser sensor detects the staircase environment in front, the delivery robot stops working and feedbacks the dangerous state to the upper computer, waiting for the upper computer logic processing.

[0044] like Figure 1 As shown, it is a schematic diagram of the overall logic flow of detecting a falling environment implemented by the present invention. The present invention implements a robot anti-fall detection method based on a depth camera and a single-point laser, comprising the following steps:

[0045] 1) Install a depth camera and a laser sensor on the delivery robot, and connect the depth camera and the laser sensor to the robot main controller, so that during the driving process of the delivery robot, the depth camera and the laser sensor periodically collect environmental data and send it to the robot main controller;

[0046] The depth camera and laser sensor are both installed on the same horizontal mounting surface on the top of the delivery robot, and the mounting heights are consistent; the field of view direction of the depth camera and the laser emission direction of the laser sensor are set in the same direction as the travel direction of the delivery robot;

[0047] In this embodiment, the depth camera uses an external Intel depth camera to collect depth images within the field of view and send them to the robot main controller;

[0048] The laser sensor irradiates obliquely downward, and the angle formed by the laser emission direction of the laser sensor and the horizontal plane is 45°; the depth camera is installed at an angle of 45° and irradiates obliquely downward, and the field of view of the depth camera covers an area 500m in front of the delivery robot.

[0049] The laser sensor is a laser radar detection module. In this embodiment, the laser sensor uses a TFminiPlus laser sensor model to collect laser data of the environment in front of the delivery robot. The laser data includes: laser detection distance data and reflectivity data.

[0050] 2) The robot main controller receives the environmental data collected by the depth camera and the laser sensor respectively, and processes and analyzes the environmental data respectively to determine whether the environment in front of the delivery robot is a falling environment;

[0051] 3) If one of the processed depth camera data or laser sensor data is determined to be a falling environment, the front of the delivery robot is determined to be a falling environment, and the current dangerous state of the robot is fed back to the background server, and the background server stops the delivery robot operation;

[0052] 4) If the processed depth camera data or laser sensor data are determined not to be a falling environment, the delivery robot continues to operate.

[0053] like Figure 2 As shown, it is a schematic diagram of the cycle process of the depth camera of the present invention. In the present invention, the robot main controller receives the environmental data collected by the depth camera for data processing and analysis to determine whether the delivery robot is in a falling environment, specifically:

[0054] The depth camera captures image frames and reads depth information, and sends the depth information to the robot main controller;

[0055] The robot main controller converts the depth information into three-dimensional data information relative to the center of the delivery robot, namely:

[0056] x=Contour[i].fx+CamParam.HDis

[0057] y=Contour[i].fy+CamParam.VDis

[0058] z=Contour[i].fz+CamParam.ZDis

[0059] Where x, y, and z are the coordinates relative to the center of the robot, Contour[i].fx, Contour[i].fy, and Contour[i].fz are the coordinate values ​​of the i-th visual data, CamParam.HDis is the lateral offset of the depth camera installation position, CamParam.VDis is the longitudinal offset of the depth camera installation position, and CamParam.ZDis is the protrusion offset of the depth camera installation position;

[0060] The robot main controller determines the y coordinate data based on the y coordinate data in the three-dimensional data information of the robot center. When the y coordinate data is greater than the set height threshold, it is determined that the environment in front is a falling environment. Otherwise, the robot main controller continues to control the depth camera to collect image frames in real time.

[0061] like Figure 3 The figure is a schematic diagram of the laser sensor cycle process of the present invention. In the present invention, the robot main controller receives the environmental data collected by the laser sensor and performs data processing and analysis to determine whether the environment in front of the delivery robot is a falling environment, specifically:

[0062] The laser sensor collects laser detection distance data and reflectivity data of the environment in front of the delivery robot and sends it to the robot main controller;

[0063] The robot main controller selects the laser detection distance data of two adjacent cycles that meet the preset reflectivity range according to the preset reflectivity range and makes a difference, that is:

[0064] temp=dis[i]-dis[i-1]

[0065] Where i is the number of cycles, dis[i] is the laser detection distance data of the i-th cycle;

[0066] When the laser detection distance data difference temp is greater than 100mm within n consecutive cycles, it is determined that there is a falling environment in front of the delivery robot; otherwise, it is determined that there is no falling environment in front, and the robot main controller continues to control the laser sensor to cyclically collect environmental data.

[0067] In this embodiment, the laser sensor cyclically collects distance data and reflectivity data. Some reflective and transparent object environments can be screened out based on the reflectivity. The difference between the distance data of two adjacent cycles in the qualified range of reflectivity is taken. If the data difference is greater than the set distance threshold of 100mm and this condition is met for 5 consecutive cycles, it is considered that the environment ahead is a falling environment, the robot operation is stopped and the dangerous state is fed back to the upper computer. If the distance data difference of adjacent cycles is not greater than 100mm for 5 consecutive cycles, it is considered that the environment ahead is not a falling environment and the robot continues to operate.

[0068] In order to improve the accuracy of detecting the falling environment and prevent the hidden danger of not detecting the falling environment using both the depth information and the laser sensor, the present invention also includes that when the falling environment is a stair environment, the depth camera cannot independently determine the environment in front of the delivery robot, and the robot main controller performs the following steps:

[0069] The information collection of the stair environment is obtained by combining the grayscale image with the depth image. When the depth information contains a number less than 0, the grayscale image of the staircase in the current frame is collected. The grayscale image is first binarized, and the image is blurred by Gaussian filtering. Then, the grayscale image is subjected to Hough feature transformation to extract the lines and edges in the environment.

[0070] After the Hough feature transformation, multiple repeated lines are obtained. According to the parallel line characteristics of the stairs and the distance standards between them, the coordinates of the lines are compared and judged, and then the extracted lines are screened and filtered to extract the line contour information of the stairs;

[0071] Based on the judgment of the grayscale image and depth image information, when the depth value is greater than the height threshold and there is a contour line in the grayscale image that meets the characteristics of the stairs, it is considered that the front is a falling environment.

[0072] In summary, the present invention prevents the robot from tipping over and causing damage when the robot is moving forward or backward due to a highly transient environment such as stairs in complex robot application scenarios. By using depth information and laser sensor cyclic acquisition and processing, one of the two is used to determine the falling environment, thereby preventing the situation that cannot be identified by a single method.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robot anti-fall detection method based on a depth camera and a single-point laser, characterized in that: The following steps are involved: 1) Install a depth camera and a laser sensor on the delivery robot, and connect the depth camera and the laser sensor to the robot main controller, so that during the driving process of the delivery robot, the depth camera and the laser sensor periodically collect environmental data and send it to the robot main controller; 2) The robot main controller receives the environmental data collected by the depth camera and the laser sensor respectively, and processes and analyzes the environmental data respectively to determine whether the environment in front of the delivery robot is a falling environment; 3) If one of the processed depth camera data or laser sensor data is determined to be a falling environment, the front of the delivery robot is determined to be a falling environment, and the current dangerous state of the robot is fed back to the background server, and the background server stops the delivery robot operation; 4) If the processed depth camera data or laser sensor data are determined not to be a falling environment, the delivery robot continues to operate.

2. According to claim 1, a method for robot anti-fall detection based on a depth camera and a single-point laser is characterized in that: The depth camera and the laser sensor are both installed on the same horizontal mounting surface on the top of the delivery robot, and the mounting heights are consistent; The field of view direction of the depth camera and the laser emission direction of the laser sensor are arranged in the same direction as the travel direction of the delivery robot; The laser sensor irradiates obliquely downward, and the angle formed between the laser emission direction of the laser sensor and the horizontal plane is 45°; The depth camera is installed at an angle of 45° and shines obliquely downward. The field of view of the depth camera covers an area 500m in front of the delivery robot.

3. The method for robot anti-fall detection based on depth camera and single-point laser according to claim 1 is characterized in that: The laser sensor is a laser radar detection module, which is used to collect laser data of the environment in front of the delivery robot.

4. The method for robot anti-fall detection based on depth camera and single-point laser according to claim 3 is characterized in that: The laser data includes: laser detection distance data and reflectivity data.

5. The method for robot anti-fall detection based on depth camera and single-point laser according to claim 1, characterized in that: In step 2), the robot main controller receives the environmental data collected by the depth camera and performs data processing and analysis to determine whether the delivery robot is in a falling environment, specifically: The depth camera captures image frames and reads depth information, and sends the depth information to the robot main controller; The robot main controller converts the depth information into three-dimensional data information relative to the center of the delivery robot, namely: x=Contour[i].fx+CamParam.HDis y=Contour[i].fy+CamParam.VDis z=Contour[i].fz+CamParam.ZDis Where x, y, and z are the coordinates relative to the center of the robot, Contour[i].fx, Contour[i].fy, and Contour[i].fz are the coordinates and depth values ​​of the i-th visual data, CamParam.HDis is the lateral offset of the depth camera installation position, CamParam.VDis is the longitudinal offset of the depth camera installation position, and CamParam.ZDis is the protrusion offset of the depth camera installation position; The robot main controller determines the y coordinate data based on the y coordinate data in the three-dimensional data information of the robot center. When the y coordinate data is greater than the set height threshold, it is determined that the environment in front is a falling environment. Otherwise, the robot main controller continues to control the depth camera to collect image frames in real time.

6. The method for robot anti-fall detection based on depth camera and single-point laser according to claim 1, characterized in that: In step 2), the robot main controller receives the environmental data collected by the laser sensor and performs data processing and analysis to determine whether the environment in front of the delivery robot is a falling environment, specifically: The laser sensor collects laser detection distance data and reflectivity data of the environment in front of the delivery robot and sends it to the robot main controller; The robot main controller selects the laser detection distance data of two adjacent cycles that meet the preset reflectivity range according to the preset reflectivity range and makes a difference, that is: temp=dis[i]-dis[i-1] Where i is the number of cycles, dis[i] is the laser detection distance data of the i-th cycle; When the laser detection distance data difference temp is greater than 100mm within n consecutive cycles, it is determined that there is a falling environment in front of the delivery robot; otherwise, it is determined that there is no falling environment in front, and the robot main controller continues to control the laser sensor to cyclically collect environmental data.

7. The method for robot anti-fall detection based on depth camera and single-point laser according to claim 1, characterized in that: Also includes: When the falling environment is a staircase environment, the depth camera cannot accurately determine the environment in front of the delivery robot alone, so the robot main controller performs the following steps: The information collection of the stair environment is obtained by combining the grayscale image with the depth image. Combined with the depth data information, when the depth information value is less than 0, the grayscale image of the current frame of the stair is collected. First, the grayscale image is binarized, and then the image is blurred by Gaussian filtering. Then, the grayscale image is subjected to Hough feature transformation to extract the lines and edges in the environment. After the Hough feature transformation, multiple repeated lines are obtained. According to the parallel line characteristics of the stairs and the distance standards between them, the coordinates of the lines are compared and judged, and then the extracted lines are screened and filtered to extract the line contour information of the stairs; Based on the judgment of the grayscale image and depth image information, when the depth value is greater than the height threshold and there is a contour line in the grayscale image that meets the characteristics of the stairs, it is considered that the front is a falling environment.

Citation Information

Patent Citations

  • AGV obstacle avoidance method based on data fusion of laser radar and depth camera

    CN112925326A

  • Autonomous obstacle avoidance method based on deep learning and stereoscopic vision

    CN113031597A

  • Depth camera and single-line laser radar fused mobile robot obstacle avoidance method

    CN113110451A

  • Method and Server for Obstacle Avoidance Driving Control of Moving Vehicle

    KR102449047B1

  • Epidemic-prevention disinfection and cleaning robot

    WO2021208381A1