A safety protection method and device for a mobile robot

Abstract

The present invention discloses a safety protection method and device for a mobile robot. In order to overcome the problem that a single sensor in the prior art is difficult to solve the perception of a glass wall, etc.; the method includes the following steps: S1: The camera acquires a real-time image of the detection area with respect to the projected image, and compares it with the reference image of the preset plane area with respect to the projected image to obtain an image difference area; S2: Determine the position of the obstacle according to the image difference area, and calculate the starting point of the line segment difference in the image difference area; S3: Determine the position information of the starting point of the line segment difference in the vehicle body coordinate system through the mapping relationship of the coordinate system; S4: Perform real-time safety protection for the robot according to the position information in the vehicle body coordinate system. By means of the structured information of the projected pattern, obstacles in the detection scene can be detected, effectively solving the perception pain points of other sensors such as glass walls, and the cost is relatively low, which can better meet the protection requirements of mobile robots.

Description

Technical Field

[0001] The present invention relates to the field of robot safety, and particularly to a safety protection method and device for a mobile robot. Background Art

[0002] Scene perception is a very crucial link in the operation of a mobile robot, which directly determines whether the mobile robot can operate stably, reliably and safely in the scene. The sensors used for robot scene perception can be divided into contact type and non-contact type according to the contact method. Among them, the contact type is widely used in household sweeping robots, and other types of mobile robots generally use non-contact type due to the limitations of the application scenarios.

[0003] The mainstream non-contact sensors include 2D lasers, 3D lasers, depth cameras and radars.

[0004] A 2D laser can only sense the distance information of objects under a certain plane, while the objects exist in the entire three-dimensional space, so its scene perception ability is weak.

[0005] A 3D laser can sense the 3D information of the entire scene, but its cost is high, and there will also be a problem of missed detection due to the laser passing through transparent glass.

[0006] The depth camera has a very good performance in the perception ability of the short-range three-dimensional space, and its cost is also much lower than that of the 3D laser. However, whether it is a TOF, structured light or binocular depth camera, it is also difficult to solve the perception problems such as glass walls.

[0007] The radar can effectively detect glass walls, but it can only detect whether there are obstacles in a certain area and cannot give spatial position information, generally playing a role of safety warning assistance. For example, a "robot obstacle avoidance system" disclosed in a Chinese patent document, with the publication number CN108527364A, includes: an acquisition module for acquiring the distance information between the robot and the obstacle; a control module communicatively connected to the acquisition module to obtain the distance information and generate a control signal; and an execution module communicatively connected to the control module to control the movement of the robot according to the control signal. This solution realizes the active obstacle avoidance of the robot by setting two types of sensors, namely a lidar and an ultrasonic array. This solution uses a combination of two sensors to achieve the obstacle avoidance function for glass, with high cost. Summary of the Invention

[0008] The present invention mainly solves the problem that a single sensor in the prior art is difficult to solve the perception problems such as glass walls; provides a safety protection method and device for a mobile robot, which can detect obstacles in the scene by means of the structured information of the projected pattern, can effectively solve the perception pain points of other sensors such as glass walls, and has a relatively low cost, and can better meet the protection requirements of mobile robots.

[0009] The above technical problems of the present invention are mainly solved by the following technical solutions:

[0010] A safety protection method for a mobile robot, comprising the following steps:

[0011] S1: A camera collects a real-time image of the detection area with respect to the projected image, compares it with a reference image of the preset plane area with respect to the projected image, and obtains an image difference area;

[0012] S2: Determine the position of the obstacle according to the image difference area, and calculate the starting point of the line segment difference in the image difference area;

[0013] S3: Determine the position information of the starting point of the line segment difference in the vehicle body coordinate system through the mapping relationship of the coordinate system;

[0014] S4: Perform robot safety protection in real time according to the position information in the vehicle body coordinate system.

[0015] This solution compares the real-time image including the projected image with the reference image, determines the starting point position of the obstacle through image difference, obtains the obstacle starting point coordinates in the vehicle body coordinate system through the coordinate system mapping relationship, and controls the start of the robot's protection by comparing with the distance threshold. With the help of the structured information of the projected pattern, obstacles in the detection scene can be detected, effectively solving the perception pain points of other sensors such as glass walls, and the cost is relatively low, which can better meet the protection requirements of mobile robots.

[0016] Preferably, the preset process of the reference image is as follows:

[0017] The pattern projector projects the projected image into the scene. When the projected image is in the plane area, a reference frame image is captured and used as the preset reference image.

[0018] The reference image is the reference frame image captured when the projected image is in the plane area.

[0019] Preferably, the real-time image and the reference image are used to obtain the image difference area through image difference, and whether there is a non-planar object in the scene is judged according to whether there is a difference area after image difference.

[0020] Preferably, the expression of the image difference is:

[0021]

[0022] Wherein, I(x, y) is the pixel value at the coordinate (x, y) in the real-time image;

[0023] I ref (x, y) is the pixel value at the coordinate (x, y) in the reference image;

[0024] thre is the threshold for whether there is a change.

[0025] Preferably, when it is determined that there are non-planar objects in the scene, determine the starting point of the difference for each line segment; search for the difference points along the upward direction of each line segment in the reference image, and the first difference point is the starting point of the difference for each line segment.

[0026] Search for the difference points in the upward direction, and use the first difference point as the starting point of the difference for each line segment. Due to the installation position relationship of the pattern projector, the lines in the image are getting farther and farther away from the robot body from bottom to top. Select the point closest to the robot as the starting point of the obstacle.

[0027] Preferably, the process of establishing the mapping relationship from the points on the image plane to the vehicle body coordinate system is as follows:

[0028] A1: Fix the calibration board on the ground plane;

[0029] A2: Map the points on the image to the ground plane coordinate system;

[0030] Establish a ground plane coordinate system on the calibration board (so as to determine the coordinate points of each corner of the checkerboard in the ground plane coordinate system), and at the same time perform checkerboard corner detection on the image data collected by the camera (so as to determine the coordinate points of each corner of the checkerboard in the image plane coordinate system). Solve the homography matrix based on the corresponding relationship between the coordinate points in the ground plane coordinate system and the coordinate points in the image plane coordinate system;

[0031] A3: Map from the ground plane coordinate system to the camera coordinate system;

[0032] Collect an image data covering the entire calibration board and perform external parameter calibration (assuming that the camera internal parameter calibration has been completed in advance) to obtain the mapping relationship from the ground plane coordinate system to the camera coordinate system;

[0033] A4: Complete the mapping from the camera coordinate system to the vehicle body coordinate system through hand-eye calibration.

[0034] Establish the mapping relationship from the points on the image plane to the vehicle body coordinate system in advance. Through this mapping relationship, the position information of the starting point of the obstacle in the vehicle body coordinate system can be determined.

[0035] Preferably, calculate the distance from the current coordinate of the robot to the starting point of the obstacle. When the distance is less than or equal to the set safety threshold, start the protection and brake or control the robot to turn.

[0036] Perform real-time robot safety protection according to the position information of the starting point of the non-planar object in the vehicle body coordinate system.

[0037] A mobile robot safety protection device, comprising:

[0038] A camera that acquires a real-time image including a projected image in a detection area;

[0039] A pattern projector that projects a projected image into a detection area in the forward direction of the robot;

[0040] A calculation and analysis unit that compares the real-time image including the projected image with a reference image, determines the starting position of an obstacle through image difference, obtains the starting coordinates of the obstacle in the vehicle body coordinate system through the coordinate system mapping relationship, and controls the activation of the robot's protection according to the comparison of distance thresholds.

[0041] By means of the structured information of the projected pattern, obstacles in the detection scene can be detected, effectively solving the perception pain points of other sensors such as glass walls, and the cost is relatively low, which can better meet the protection requirements of mobile robots.

[0042] Preferably, the projected image is a number of parallel lines, and the line direction extends along the forward direction of the robot. With this setting, the lines in the image are farther and farther away from the robot body from bottom to top, and the point closest to the robot is selected as the starting point of the obstacle.

[0043] Preferably, the line spacing is equal. According to the measurement accuracy requirements, the line density can be adjusted.

[0044] The beneficial effects of the present invention are:

[0045] Compare the real-time image including the projected image with the reference image, determine the starting position of the obstacle through image difference, obtain the starting coordinates of the obstacle in the vehicle body coordinate system through the coordinate system mapping relationship, and control the activation of the robot's protection according to the comparison of distance thresholds. By means of the structured information of the projected pattern, obstacles in the detection scene can be detected, effectively solving the perception pain points of other sensors such as glass walls, and the cost is relatively low, which can better meet the protection requirements of mobile robots. Description of the Drawings

[0046] Figure 1 It is a schematic diagram of the mobile robot safety protection device of the present invention.

[0047] Figure 2 It is a flowchart of the mobile robot safety protection method of the present invention.

[0048] Figure 3 It is a planar scene image of the present invention.

[0049] Figure 4 It is a non-planar scene image of the present invention.

[0050] Figure 5 It is a differential result diagram of the planar scene image of the present invention.

[0051] Figure 6 It is the differential result diagram of the non-planar scene image of the present invention.

[0052] In the figure: 1. Robot body; 2. Pattern projector; 3. Camera; 4. Projected image. Specific implementation mode

[0053] The technical solution of the present invention will be further specifically described below through embodiments in conjunction with the accompanying drawings.

[0054] Embodiment:

[0055] A safety protection device for a mobile robot in this embodiment, as Figure 1 shown, includes a robot body 1, a pattern projector 2 and a camera 3 provided on the robot body 1.

[0056] In this embodiment, the pattern projector 2 and the camera 3 are provided on the top of the robot body 1 and face the advancing direction of the robot.

[0057] The pattern projector 2 projects a projected image 4 into the detection area in the advancing direction of the robot, and the camera 3 acquires a real-time image including the projected image in the detection area. In this embodiment, the camera 3 uses an RGB camera.

[0058] In this embodiment, the projected image 4 is a number of parallel lines, and the line direction extends along the advancing direction of the robot. So that the lines in the projected image 4 are farther and farther away from the robot body 1 from bottom to top, and a point closest to the robot is selected as the starting point of the obstacle. In this embodiment, the line spacing in the projected image 4 is equal, and the line density can be adaptively adjusted according to the measurement accuracy requirements.

[0059] The safety protection device for the mobile robot further includes a calculation and analysis unit. The calculation and analysis unit compares the real-time image including the projected image 4 with a reference image, determines the starting position of the obstacle through image difference, obtains the obstacle starting coordinates in the vehicle body coordinate system through the coordinate system mapping relationship, and controls the protection of the robot to start according to the comparison of the distance threshold.

[0060] By means of the structured information of the projection pattern 4, obstacles in the detection scene can be detected, effectively solving the perception pain points of other sensors such as glass walls, and the cost is relatively low, which can better meet the protection requirements of mobile robots.

[0061] A safety protection method for a mobile robot in this embodiment, as Figure 2 shown, includes the following steps:

[0062] S1: The camera acquires a real-time image of the detection area with respect to the projected image, compares it with the reference image of the preset plane area with respect to the projected image, and obtains the image difference area.

[0063] The preset process of the reference image is as follows:

[0064] The pattern projector 2 projects the projection image 4 into the scene. As Figure 3 shown, when the projection image is in the planar region, the reference frame image is captured and used as the preset reference image.

[0065] The reference image is the reference frame image captured when the projection image 4 is in the planar region, and is used to compare with the real-time image acquired by the camera 3 to obtain the image difference region.

[0066] When the area detected by the robot is planar, as Figure 3 shown, the straight line of the projection image 4 remains a straight line on the image plane. When there are non-planar objects in the area detected by the robot, as Figure 4 shown, the straight line of the projection image 4 will be deformed on the image plane. When the laser line penetrates the glass, due to the refraction of light, the straight line of the projection image 4 will be deformed.

[0067] The real-time image and the reference image are used to obtain the image difference region through image difference, and it is judged whether there are non-planar objects in the scene according to whether there is a difference region after image difference.

[0068] The expression of image difference is:

[0069]

[0070] where I(x, y) is the pixel value at the coordinate (x, y) in the real-time image;

[0071] I ref (x, y) is the pixel value at the coordinate (x, y) in the reference image;

[0072] thre is the threshold for whether a change occurs.

[0073] When there is no difference region after image difference, and the image difference result is as Figure 5 shown, it is judged that all the objects in the scene are planar objects.

[0074] When there is a difference region after image difference, and the image difference result is as Figure 6 shown, it is judged that there are non-planar objects or glass in the scene.

[0075] S2: Determine the obstacle position according to the image difference region, and calculate the starting point of the line segment difference in the image difference region.

[0076] Determine the obstacle position according to the image difference region of image difference. A larger difference indicates the appearance of a non-planar object (obstacle).

[0077] When it is determined that there are non-planar objects in the scene, determine the starting point of the difference for each line segment. Search for the difference points along the upward direction of each line segment in the reference image, and the first difference point is the starting point of the difference for each line segment.

[0078] Search for the difference points in the upward direction. The first difference point is used as the starting point of the difference for each line segment. Due to the installation position relationship of the pattern projector, the lines in the image are getting farther and farther away from the robot body from bottom to top. Select the point closest to the robot as the starting point of the obstacle.

[0079] S3: Determine the position information of the starting point of the line segment difference in the vehicle body coordinate system through the mapping relationship of the coordinate system.

[0080] Establish the mapping relationship from the points on the image plane to the vehicle body coordinate system in advance. Through this mapping relationship, the position information of the starting point of the obstacle in the vehicle body coordinate system can be determined.

[0081] The process of establishing the mapping relationship from the points on the image plane to the vehicle body coordinate system is as follows:

[0082] A1: Fix the calibration board on the ground plane;

[0083] A2: Map the points on the image to the ground plane coordinate system;

[0084] Establish the ground plane coordinate system on the calibration board (so that the coordinate points of each corner of the checkerboard in the ground plane coordinate system can be determined), and at the same time, perform checkerboard corner detection on the image data collected by the camera (so that the coordinate points of each corner of the checkerboard in the image plane coordinate system can be determined). Solve the homography matrix based on the corresponding relationship between the coordinate points in the ground plane coordinate system and the coordinate points in the image plane coordinate system;

[0085] A3: Map from the ground plane coordinate system to the camera coordinate system;

[0086] Collect a piece of image data covering the entire calibration board and perform external parameter calibration (assuming that the camera internal parameter calibration has been completed in advance) to obtain the mapping relationship from the ground plane coordinate system to the camera coordinate system;

[0087] A4: Complete the mapping from the camera coordinate system to the vehicle body coordinate system through hand-eye calibration.

[0088] S4: Perform real-time robot safety protection according to the position information in the vehicle body coordinate system.

[0089] Calculate the distance from the current coordinate of the robot to the starting point of the obstacle. When the distance is less than or equal to the set safety threshold, start protection and brake or control the robot to turn.

[0090] Perform real-time robot safety protection according to the position information of the starting point of the non-planar object in the vehicle body coordinate system.

[0091] The solution comparison in this embodiment includes the real-time image and the reference image of the projected image. The starting position of the obstacle is determined by image difference, and the starting coordinates of the obstacle in the vehicle body coordinate system are obtained through the coordinate system mapping relationship. The protection of the robot is controlled by comparing with the distance threshold. By means of the structured information of the projected pattern, obstacles in the scene can be detected, effectively solving the perception pain points of other sensors such as glass walls, and the cost is relatively low, which can better meet the protection requirements of mobile robots.

[0092] It should be understood that the embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

Claims

1. A safety protection method for a mobile robot, characterized in that, It includes the following steps: S1: The camera acquires a real-time image of the detection area with respect to the projected image, and compares it with the reference image of the preset plane area with respect to the projected image to obtain an image difference area; the projected image is a number of parallel lines with equal spacing, and the line direction extends along the forward direction of the robot; when the area detected by the robot is a plane, the straight lines of the projected image are still straight lines on the image plane. If the area detected by the robot is a non-plane or the laser line penetrates the glass, the straight lines of the projected image will be deformed on the image plane; S2: Determine the position of the obstacle according to the image difference area, and calculate the starting point of the difference of each line segment in the image difference area; Search for the difference points along the upward direction of each line segment in the reference image. The first difference point is the starting point of the difference of each line segment, and select the point closest to the robot as the starting point of the obstacle; S3: Determine the position information of the starting point of the line segment difference in the vehicle body coordinate system through the mapping relationship of the coordinate system; S4: Perform real-time robot safety protection according to the position information in the vehicle body coordinate system.

2. The mobile robot safety protection method according to claim 1, characterized in that The preset process of the reference image is as follows: The pattern projector projects the projected image into the scene. When the projected image is in the plane area, a reference frame image is captured and used as the preset reference image.

3. A mobile robot safety protection method according to claim 1 or 2, characterized in that, The real-time image and the reference image obtain the image difference area through image difference, and judge whether there are non-planar objects in the scene according to whether there is a difference area after image difference.

4. A mobile robot safety protection method according to claim 3, characterized in that, The expression of the image difference is: where I(x, y) is the pixel value at the coordinate (x, y) in the real-time image; I ref (x, y) is the pixel value at the coordinates (x, y) in the reference image; thre is the threshold for whether a change occurs.

5. A safety protection method for a mobile robot according to claim 1, characterized in that, The process of establishing the mapping relationship from the points on the image plane to the vehicle body coordinate system is as follows: A1: Fix the calibration board on the ground plane; A2: Map the points on the image to the ground plane coordinate system; Establish a ground plane coordinate system on the calibration board, and at the same time perform checkerboard corner detection on the image data collected by the camera. Solve the homography matrix based on the corresponding relationship between the coordinate points in the ground plane coordinate system and the coordinate points in the image plane coordinate system; A3: Map from the ground plane coordinate system to the camera coordinate system; Collect an image data covering the entire calibration board and perform external parameter calibration to obtain the mapping relationship from the ground plane coordinate system to the camera coordinate system; A4: Complete the mapping from the camera coordinate system to the vehicle body coordinate system through hand-eye calibration.

6. A mobile robot safety protection method according to claim 1 or 5, characterized in that, Calculate the distance from the current coordinate of the robot to the starting point of the obstacle. When the distance is less than or equal to the set safety threshold, start the protection and brake or control the robot to turn.

7. A safety protection device for a mobile robot, which adopts a safety protection method for a mobile robot as described in any one of claims 1-6, characterized in that, It includes: A camera that acquires a real-time image including the projected image in the detection area; A pattern projector that projects the projected image into the detection area in the forward direction of the robot; A calculation and analysis unit that compares the real-time image including the projected image with the reference image, determines the starting point position of the obstacle through image difference, obtains the coordinates of the starting point of the obstacle in the vehicle body coordinate system through the coordinate system mapping relationship, and controls the start of the robot's protection according to the comparison of the distance threshold.

8. The safety protection device for a mobile robot according to claim 7, wherein, The projected image is a number of parallel lines, and the line direction extends along the forward direction of the robot.

9. The safety protection device for a mobile robot according to claim 8, characterized in that, The line spacing is equal.

Images