A method for preventing robot falls based on depth cameras and single-point lasers

Abstract

This invention belongs to the field of robot vision processing technology, specifically a method for preventing robot falls based on a depth camera and a single-point laser. The method includes the following steps: The robot's main controller receives environmental data collected by the depth camera and the laser sensor, respectively, and processes and analyzes the data to determine whether the environment in front of the delivery robot is a fall-prone environment. If either the processed data from the depth camera or the laser sensor indicates a fall-prone environment, the robot is deemed to be in a fall-prone environment, and the current dangerous state of the robot is reported to the backend server, which then stops the robot's operation. If neither the processed data from the depth camera nor the laser sensor indicates a fall-prone environment, the robot continues operation. This invention prevents robots from tipping over and causing damage in complex application scenarios, such as on stairs where height changes rapidly.

Description

Technical Field

[0001] This invention belongs to the field of robot vision processing technology, specifically a method for preventing robot falls based on a depth camera and a single-point laser. Background Technology

[0002] With social progress and continuous technological development, intelligent robots are playing a vital role in people's lives, and their presence can be seen in various public places. Some application scenarios are relatively straightforward, involving only the avoidance and obstacle avoidance of people or objects. However, other application scenarios involve environments with stairs, escalators, and pits, which are undetectable by horizontally mounted laser chassis. Therefore, robots operating in such scenarios pose safety hazards and require external sensor devices to detect environments like stairs, stop robot operations, report the danger status, and await processing by the host computer.

[0003] Traditional methods for detecting fall prevention involve installing fall prevention sensors on the robot's chassis and calculating distance using analog signals. However, the detection accuracy is insufficient, which can affect the robot's emergency stopping ability and cause it to tip over or fall down stairs. Summary of the Invention

[0004] The purpose of this invention is to provide a method for fall detection based on a depth camera and a single-point laser, so as to prevent the robot from tipping over and causing losses when moving forward or backward due to the rapidly changing height of the environment such as stairs.

[0005] The technical solution adopted by the present invention to achieve the above objectives is: a method for preventing robot falls based on a depth camera and a single-point laser, comprising the following steps:

[0006] 1) Install depth cameras and laser sensors on the delivery robot and connect them to the robot's main controller so that the depth cameras and laser sensors can periodically collect environmental data and send it to the robot's main controller during the delivery robot's operation.

[0007] 2) The robot's main controller receives environmental data collected by the depth camera and laser sensor respectively, and performs data processing and analysis on the environmental data to determine whether the environment in front of the delivery robot is a fall environment;

[0008] 3) If any of the processed depth camera data or laser sensor data is determined to be a fall environment, then the delivery robot is determined to be in a fall environment and the current dangerous status of the robot is reported to the backend server, and the backend server stops the delivery robot from operating.

[0009] 4) If the processed depth camera data or laser sensor data both indicate that the environment is not a fall environment, the delivery robot continues to operate.

[0010] The depth camera and laser sensor are both mounted on the same horizontal mounting surface on top of the delivery robot, and at the same mounting height.

[0011] The field of view of the depth camera and the laser emission direction of the laser sensor are set in the same direction as the direction of travel of the delivery robot;

[0012] The laser sensor is irradiated obliquely downwards, and the angle between the laser emission direction of the laser sensor and the horizontal plane is 45°.

[0013] The depth camera is installed at a 45° angle and shines obliquely downwards, and its field of view covers an area 500m in front of the delivery robot.

[0014] The laser sensor is a lidar detection module used to collect laser data of the environment in front of the delivery robot.

[0015] The laser data includes: laser detection distance data and reflectivity data.

[0016] In step 2), the robot's main controller receives environmental data collected by the depth camera, processes and analyzes the data, and determines whether the delivery robot is in a fall environment. Specifically:

[0017] The depth camera acquires image frames and reads depth information, then sends the depth information to the robot's main controller;

[0018] The robot's main controller converts depth information into three-dimensional data information relative to the center of the delivery robot, that is:

[0019] x=Contour[i].fx+CamParam.HDis

[0020] y=Contour[i].fy+CamParam.VDis

[0021] z=Contour[i].fz+CamParam.ZDis

[0022] Where x, y, and z are coordinates relative to the robot's center, Contour[i].fx, Contour[i].fy, and Contour[i].fz are the coordinate values ​​of the i-th visual data, CamParam.HDis is the lateral offset of the depth camera's mounting position, CamParam.VDis is the longitudinal offset of the depth camera's mounting position, and CamParam.ZDis is the protrusion offset of the depth camera's mounting position.

[0023] The robot's main controller determines the y-coordinate data based on the 3D data information at the robot's center. If the y-coordinate data is greater than the set height threshold, it determines that the environment in front is a fall environment. Otherwise, the robot's main controller continues to control the depth camera to collect image frames in real time.

[0024] In step 2), the robot's main controller receives environmental data collected by the laser sensor, processes and analyzes the data, and determines whether the environment in front of the delivery robot is a fall-prone environment. Specifically:

[0025] The laser sensor collects laser detection distance data and reflectivity data of the environment in front of the delivery robot and sends them to the robot's main controller;

[0026] The robot's main controller selects and calculates the difference between laser detection distance data from two adjacent cycles that fall within a preset reflectivity range, i.e.:

[0027] temp = dis[i] - dis[i-1]

[0028] Where i is the number of cycles, and dis[i] is the laser detection distance data of the i-th cycle;

[0029] If the difference in laser detection distance data temp is greater than 100mm for n consecutive cycles, it is determined that the area in front of the delivery robot is a fall environment; otherwise, it is determined that the area in front is not a fall environment, and the robot's main controller continues to control the laser sensor to collect environmental data in a loop.

[0030] This also includes: When the fall environment is a staircase, and the depth camera cannot independently determine the environment in front of the delivery robot, the robot's main controller executes the following steps:

[0031] Information about the staircase environment is obtained by combining grayscale images and depth images. When there is a value less than 0 in the depth information, the grayscale image of the staircase in the current frame is acquired. First, the grayscale image is binarized. After blurring the image by Gaussian filtering, the grayscale image is subjected to Hough feature transformation to extract lines and edges in the environment.

[0032] After Hough feature transformation, multiple repeating lines are obtained. Based on the parallel line characteristics of the staircase and the distance standard between them, the coordinates of the lines are compared and judged, and then the extracted lines are screened and filtered to extract the line outline information of the staircase.

[0033] Based on the judgment of grayscale and depth image information, when the depth value is greater than the height threshold and the grayscale image has an outline that matches the characteristics of stairs, it is considered that the area in front is a fall environment.

[0034] The present invention has the following beneficial effects and advantages:

[0035] 1. This invention prevents robots from tipping over and causing losses in complex application scenarios, such as staircases, when moving forward or backward.

[0036] 2. This invention utilizes a method of cyclically collecting and processing depth information and laser sensors, employing either one to determine the fall environment, thus preventing situations where a single method cannot identify the fall.

[0037] 3. This invention greatly increases the accuracy of detection by combining depth information with grayscale images, preventing potential risks of failing to detect the fall environment when both depth information and laser sensors are used. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall logic flow for detecting drop environments implemented in this invention;

[0039] Figure 2 This is a schematic diagram of the depth camera loop process of the present invention;

[0040] Figure 3 This is a schematic diagram of the cyclic process of the laser sensor of the present invention. Detailed Implementation

[0041] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0042] This invention utilizes depth data acquired by a depth camera to detect obstacles below ground level, such as cracks in the ground or stairs, and sends this information to the robot's main controller. This allows the delivery robot to avoid hazards and operate stably. The depth camera works by sending a fixed-wavelength, encoded infrared signal from its infrared emitter, which is then received by an infrared receiver to obtain obstacle distance information. Therefore, in scenarios with significant infrared interference or light source reflection interference, the data acquired by the depth camera may be inaccurate.

[0043] Therefore, this invention integrates a single-point laser sensor with a depth camera to detect a distance of 500mm in front of the robot. Based on the detection distance and reflectivity data returned by the sensor, it analyzes and determines that if the reflectivity meets the normal range and the distance is greater than the rated range, it considers the area in front of the robot to be a fall environment, thus providing fall protection. After the depth camera or laser sensor detects a staircase environment ahead, the delivery robot stops operating and reports the danger status to the host computer, awaiting logical processing by the host computer.

[0044] like Figure 1 The diagram shown illustrates the overall logic flow of the fall detection method implemented by this invention. This invention provides a robot fall prevention detection method based on a depth camera and a single-point laser, comprising the following steps:

[0045] 1) Install depth cameras and laser sensors on the delivery robot and connect them to the robot's main controller so that the depth cameras and laser sensors can periodically collect environmental data and send it to the robot's main controller during the delivery robot's operation.

[0046] The depth camera and laser sensor are both mounted on the same horizontal mounting surface on top of the delivery robot, and at the same height; the field of view of the depth camera and the laser emission direction of the laser sensor are set in the same direction as the delivery robot's travel direction;

[0047] In this embodiment, an external Intel depth camera is used to acquire depth images within the field of view and send them to the robot's main controller.

[0048] The laser sensor emits light at an angle downwards, and the angle between the laser emission direction of the laser sensor and the horizontal plane is 45°; the depth camera is installed at a 45° angle and emits light at an angle downwards, and the field of view of the depth camera covers an area 500m in front of the delivery robot.

[0049] The laser sensor is a lidar detection module. In this embodiment, the laser sensor is a TFminiPlus laser sensor, which is used to collect laser data of the environment in front of the delivery robot. The laser data includes laser detection distance data and reflectivity data.

[0050] 2) The robot's main controller receives environmental data collected by the depth camera and laser sensor respectively, and performs data processing and analysis on the environmental data to determine whether the environment in front of the delivery robot is a fall environment;

[0051] 3) If any of the processed depth camera data or laser sensor data is determined to be a fall environment, then the delivery robot is determined to be in a fall environment and the current dangerous status of the robot is reported to the backend server, and the backend server stops the delivery robot from operating.

[0052] 4) If the processed depth camera data or laser sensor data both indicate that the environment is not a fall environment, the delivery robot continues to operate.

[0053] like Figure 2 The diagram shown illustrates the cyclic process of the depth camera in this invention. In this invention, the robot's main controller receives environmental data collected by the depth camera, processes and analyzes the data, and determines whether the delivery robot is in a fall environment. Specifically:

[0054] The depth camera acquires image frames and reads depth information, then sends the depth information to the robot's main controller;

[0055] The robot's main controller converts depth information into three-dimensional data information relative to the center of the delivery robot, that is:

[0056] x=Contour[i].fx+CamParam.HDis

[0057] y=Contour[i].fy+CamParam.VDis

[0058] z=Contour[i].fz+CamParam.ZDis

[0059] Where x, y, and z are coordinates relative to the robot's center, Contour[i].fx, Contour[i].fy, and Contour[i].fz are the coordinate values ​​of the i-th visual data, CamParam.HDis is the lateral offset of the depth camera's mounting position, CamParam.VDis is the longitudinal offset of the depth camera's mounting position, and CamParam.ZDis is the protrusion offset of the depth camera's mounting position.

[0060] The robot's main controller determines the y-coordinate data based on the 3D data information at the robot's center. If the y-coordinate data is greater than the set height threshold, it determines that the environment in front is a fall environment. Otherwise, the robot's main controller continues to control the depth camera to collect image frames in real time.

[0061] like Figure 3 The diagram shown illustrates the cyclic process of the laser sensor in this invention. In this invention, the robot's main controller receives environmental data collected by the laser sensor, processes and analyzes the data, and determines whether the environment in front of the delivery robot is a fall-prone environment. Specifically:

[0062] The laser sensor collects laser detection distance data and reflectivity data of the environment in front of the delivery robot and sends them to the robot's main controller;

[0063] The robot's main controller selects and calculates the difference between laser detection distance data from two adjacent cycles that fall within a preset reflectivity range, i.e.:

[0064] temp = dis[i] - dis[i-1]

[0065] Where i is the number of cycles, and dis[i] is the laser detection distance data of the i-th cycle;

[0066] If the difference in laser detection distance data temp is greater than 100mm for n consecutive cycles, it is determined that the area in front of the delivery robot is a fall environment; otherwise, it is determined that the area in front is not a fall environment, and the robot's main controller continues to control the laser sensor to collect environmental data in a loop.

[0067] In this embodiment, the laser sensor cyclically collects distance data and reflectivity data. Based on the reflectivity, some reflective and transparent objects in the environment can be filtered out. The difference between the distance data of two adjacent cycles within the acceptable reflectivity range is taken. If the difference is greater than the set distance threshold of 100mm and this condition is met for 5 consecutive cycles, it is considered that the environment ahead is a fall environment. The robot stops working and reports the danger status to the host computer. If the difference between the distance data of five consecutive adjacent cycles is not greater than 100mm, it is considered that the environment ahead is not a fall environment and the robot continues to work.

[0068] In addition, to improve the accuracy of detecting fall environments and prevent potential problems where neither depth information nor laser sensors can detect the fall environment, this invention also includes the following steps for situations where the fall environment is a staircase, and the depth camera cannot independently determine the environment in front of the delivery robot:

[0069] Information about the staircase environment is obtained by combining grayscale images and depth images. When there is a value less than 0 in the depth information, the grayscale image of the staircase in the current frame is acquired. First, the grayscale image is binarized. After blurring the image by Gaussian filtering, the grayscale image is subjected to Hough feature transformation to extract lines and edges in the environment.

[0070] After Hough feature transformation, multiple repeating lines are obtained. Based on the parallel line characteristics of the staircase and the distance standard between them, the coordinates of the lines are compared and judged, and then the extracted lines are screened and filtered to extract the line outline information of the staircase.

[0071] Based on the judgment of grayscale and depth image information, when the depth value is greater than the height threshold and the grayscale image has an outline that matches the characteristics of stairs, it is considered that the area in front is a fall environment.

[0072] In summary, this invention prevents robots from tipping over and causing damage in complex application scenarios, such as on stairs, where heights change rapidly. It utilizes a method of cyclically collecting and processing depth information and laser sensors, employing either one to determine the fall environment, thus preventing situations where a single method cannot identify the issue.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for preventing robot falls based on a depth camera and a single-point laser, characterized in that, Includes the following steps: 1) Install depth cameras and laser sensors on the delivery robot and connect them to the robot's main controller so that the depth cameras and laser sensors can periodically collect environmental data and send it to the robot's main controller during the delivery robot's operation. 2) The robot's main controller receives environmental data collected by the depth camera and laser sensor respectively, and performs data processing and analysis on the environmental data to determine whether the environment in front of the delivery robot is a fall environment; In step 2), the robot's main controller receives environmental data collected by the laser sensor, processes and analyzes the data, and determines whether the environment in front of the delivery robot is a fall-prone environment. Specifically: The laser sensor collects laser detection distance data and reflectivity data of the environment in front of the delivery robot and sends them to the robot's main controller; The robot's main controller selects and calculates the difference between laser detection distance data from two adjacent cycles that fall within a preset reflectivity range, i.e.: ； in, The number of cycles, For the first Laser detection distance data for each cycle; When the laser detection distance data difference is set continuously for n periods, If all values ​​are greater than 100mm, the area in front of the delivery robot is determined to be a fall environment; otherwise, the area in front is determined not to be a fall environment, and the robot's main controller continues to control the laser sensor to collect environmental data in a loop. 3) If any of the processed depth camera data or laser sensor data is identified as a fall environment, then the delivery robot is identified as being in a fall environment, and the current dangerous state of the robot is reported to the backend server, which then stops the delivery robot from operating. 4) If the processed data from the depth camera or the laser sensor is determined not to be a fall environment, the delivery robot continues to operate.

2. The method for robot fall prevention detection based on a depth camera and single-point laser according to claim 1, characterized in that, The depth camera and laser sensor are both mounted on the same horizontal mounting surface on top of the delivery robot, and at the same mounting height. The field of view of the depth camera and the laser emission direction of the laser sensor are set in the same direction as the direction of travel of the delivery robot; The laser sensor is irradiated obliquely downwards, and the angle between the laser emission direction of the laser sensor and the horizontal plane is 45°. The depth camera is installed at a 45° angle and shines obliquely downwards, and its field of view covers an area 500mm in front of the delivery robot.

3. The method for robot fall prevention detection based on a depth camera and single-point laser according to claim 1, characterized in that, The laser sensor is a lidar detection module used to collect laser data of the environment in front of the delivery robot.

4. The method for robot fall prevention detection based on a depth camera and single-point laser according to claim 3, characterized in that, The laser data includes: laser detection distance data and reflectivity data.

5. The method for robot fall prevention detection based on a depth camera and single-point laser according to claim 1, characterized in that, In step 2), the robot's main controller receives environmental data collected by the depth camera, processes and analyzes the data, and determines whether the delivery robot is in a fall environment. Specifically: The depth camera acquires image frames and reads depth information, then sends the depth information to the robot's main controller; The robot's main controller converts depth information into three-dimensional data information relative to the center of the delivery robot, that is: ； ； ； in, These are coordinates relative to the robot's center. , , For the first The coordinates and depth values ​​of each visual data point. The lateral offset of the depth camera mounting position. The vertical offset of the depth camera's mounting position. The protrusion offset of the depth camera mounting position; The robot's main controller determines the y-coordinate data based on the 3D data information at the robot's center. If the y-coordinate data is greater than the set height threshold, it determines that the environment in front is a fall environment. Otherwise, the robot's main controller continues to control the depth camera to collect image frames in real time.

6. The method for robot fall prevention detection based on a depth camera and single-point laser according to claim 1, characterized in that, Also includes: When the fall occurs on a staircase, the depth camera cannot accurately determine the environment in front of the delivery robot on its own. In this case, the robot's main controller executes the following steps: Information about the staircase environment is obtained by combining grayscale images and depth images. When there is a value less than 0 in the depth information, the grayscale image of the staircase in the current frame is acquired. First, the grayscale image is binarized. Then, the image is blurred by Gaussian filtering. Finally, the grayscale image is subjected to Hough feature transformation to extract lines and edges in the environment. After Hough feature transformation, multiple repeating lines are obtained. Based on the parallel line characteristics of the staircase and the distance standard between them, the coordinates of the lines are compared and judged, and then the extracted lines are screened and filtered to extract the line outline information of the staircase. Based on the judgment of grayscale and depth image information, when the depth value is greater than the height threshold and the grayscale image has an outline that matches the characteristics of stairs, it is considered that the area in front is a fall environment.

Images