Obstacle detection method, apparatus and electronic device

Abstract

The application provides an obstacle detection method and device and electronic equipment. In the application, when a mobile robot loaded with a laser radar and a camera needs to detect an obstacle, the camera is used to determine a target object existing in a target scene, and then a candidate object determined based on point cloud information of the scene by the laser radar is combined, and a target object corresponding to the same physical position of the candidate object is marked as an obstacle, so that the obstacle in the target scene can be accurately identified, and a three-dimensional continuous obstacle avoidance effect of the mobile robot is realized. In the above scheme, compared with the traditional detection method of using the camera or the laser radar alone, the obstacle detection method based on the cooperation of the camera and the laser radar has the advantages of lower requirement for sensor performance and higher detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to sensor detection technology, in particular to an obstacle detection method and device and electronic equipment. BACKGROUND

[0002] Obstacle detection is a key technology in mobile robots. In order to avoid collision between mobile robots and people or objects during driving, it is necessary to perform three-dimensional and continuous obstacle detection on the forward direction or the surroundings of the mobile robot, so as to avoid obstacles according to the detected obstacle positions. With the continuous increase of the vehicle body of the existing mobile robot, the demand for obstacle avoidance is also increasing.

[0003] At present, there are various obstacle avoidance schemes in the field of mobile robots, but each has limitations. For example, the cost of installing a 3D laser radar capable of independently identifying obstacles is high. The obstacle detection by a stereo camera has the problem of limited field of view, and multiple stereo cameras need to be installed to realize stereo protection, which is costly. The relatively more cost-effective 2D laser radar can only detect planar obstacles in the installation direction and cannot detect low or suspended obstacles, with a large blind area. SUMMARY

[0004] The embodiments of the present application provide an obstacle detection method, device and electronic equipment, which perform obstacle detection by cooperating a laser radar and a camera, so as to realize three-dimensional and continuous obstacle avoidance of the mobile robot.

[0005] In a first aspect, the embodiments of the present application provide an obstacle detection method applied to a mobile robot including a laser radar and a camera, the laser radar is used to obtain point cloud information of a target scene, the camera is used to obtain image information of the target scene, and the method comprises:

[0006] obtaining point cloud information and image information at the same time;

[0007] if it is determined based on the image information that the target scene has a target object, determining whether the target object has been marked as an obstacle type;

[0008] if yes, identifying the target object as an obstacle;

[0009] if no, determining whether the target scene has a candidate object based on the point cloud information;

[0010] if the candidate object exists and the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object, identifying the target object as an obstacle and marking the target object as an obstacle type.

[0011] In a possible implementation, after determining whether the target scene contains a candidate object based on the point cloud information, the method further includes:

[0012] If there is no candidate object, or the physical position corresponding to the candidate object is different from the physical position corresponding to the target object, the target object is identified as a non-obstacle.

[0013] In a possible implementation, after identifying the target object as an obstacle, the method further includes:

[0014] Performing an obstacle avoidance operation on the target object.

[0015] In a possible implementation, whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object is determined in the following manner:

[0016] Determining a conversion matrix based on the extrinsic parameters between the camera and the lidar, where the conversion matrix is used to represent the coordinate conversion relationship between the camera coordinate system of the camera and the radar coordinate system of the lidar.

[0017] Determining whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the radar coordinate system, and the conversion matrix.

[0018] In a possible implementation, determining whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the radar coordinate system, and the conversion matrix includes:

[0019] Determining a first coordinate position of the target object in the camera coordinate system based on the image information.

[0020] Converting the first coordinate position to a second coordinate position in the radar coordinate system based on the conversion matrix.

[0021] Determining a third coordinate position of the candidate object in the radar coordinate system based on the point cloud information.

[0022] If the second coordinate position matches the third coordinate position, it is determined that the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object.

[0023] If the second coordinate position does not match the third coordinate position, it is determined that the physical position corresponding to the candidate object is different from the physical position corresponding to the target object.

[0024] In a possible implementation, the laser radar is a 2D laser radar, and the camera is a monocular camera.

[0025] The 2D laser radar is installed on the mobile robot in a downward tilt manner, and a detection area of the 2D laser radar overlaps a detection area of the monocular camera.

[0026] The extrinsic parameter is determined based on relative installation positions of the 2D laser radar and the monocular camera.

[0027] In a second aspect, an obstacle detection apparatus is provided, which is applied to a mobile robot including a laser radar and a camera, the laser radar is configured to acquire point cloud information of a target scene, and the camera is configured to acquire image information of the target scene, and the apparatus includes:

[0028] An information acquisition unit is configured to acquire the point cloud information and the image information at the same time.

[0029] A label detection unit is configured to determine whether a target object in the target scene has been labeled as an obstacle type based on the image information.

[0030] A label confirmation unit is configured to identify the target object as an obstacle if the target object has been labeled as the obstacle type.

[0031] An obstacle identification unit is configured to determine whether a candidate object exists in the target scene based on the point cloud information if the target object has not been labeled as the obstacle type.

[0032] The obstacle identification unit is further configured to identify the target object as the obstacle and label the target object as the obstacle type if the candidate object exists and a physical position corresponding to the candidate object is the same as a physical position corresponding to the target object.

[0033] In a possible implementation, after the obstacle identification unit determines whether the candidate object exists in the target scene based on the point cloud information, the obstacle identification unit is further configured to identify the target object as a non-obstacle if the candidate object does not exist or the physical position of the candidate object is different from the physical position corresponding to the target object.

[0034] In a possible implementation, after the obstacle identification unit identifies the target object as the obstacle, the obstacle identification unit is further configured to perform an obstacle avoidance operation on the target object.

[0035] In a possible implementation, the obstacle identification unit determines whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object by: determining a conversion matrix based on the extrinsic parameter between the camera and the lidar, where the conversion matrix is used to represent a coordinate conversion relationship between a camera coordinate system of the camera and a lidar coordinate system of the lidar; and determining whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the lidar coordinate system, and the conversion matrix.

[0036] In a possible implementation, the obstacle identification unit determines whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the lidar coordinate system, and the conversion matrix by: determining a first coordinate position of the target object in the camera coordinate system based on the image information; converting the first coordinate position into a second coordinate position in the lidar coordinate system based on the conversion matrix; determining a third coordinate position of the candidate object in the lidar coordinate system based on the point cloud information; determining that the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object if the second coordinate position matches the third coordinate position; and determining that the physical position corresponding to the candidate object is different from the physical position corresponding to the target object if the second coordinate position does not match the third coordinate position.

[0037] In a possible implementation, the lidar is a 2D lidar, and the camera is a monocular camera; the 2D lidar is installed on the mobile robot in a downward tilt manner, and a detection region of the 2D lidar overlaps with a detection region of the monocular camera; and the extrinsic parameter in the obstacle identification unit is determined based on a relative installation position of the 2D lidar and the monocular camera.

[0038] In a third aspect, an electronic device is provided, which includes: a processor and a machine readable storage medium.

[0039] The machine readable storage medium stores machine executable instructions which can be executed by the processor.

[0040] The processor is configured to execute the machine executable instructions to implement the obstacle detection method.

[0041] In a fourth aspect, the embodiments of the present application further provide a machine readable storage medium, the machine readable storage medium stores machine readable instructions, when the machine readable instructions are called and executed by a processor, the machine readable instructions cause the processor to implement the above obstacle detection method.

[0042] It can be seen from the above technical solutions that when a mobile robot loaded with a laser radar and a camera needs to detect obstacles, the camera is used to determine target objects existing in a target scene, and then candidate objects determined based on point cloud information of the scene by the laser radar are combined to mark target objects with the same physical position as the candidate objects as obstacles, so that the obstacles in the target scene can be accurately identified, and the three-dimensional continuous obstacle avoidance effect of the mobile robot is realized. Compared with the traditional detection method of using a camera or a laser radar alone, the obstacle detection method based on the cooperation of the camera and the laser radar in the above scheme has the advantages of lower requirement for sensor performance and higher detection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0044] Figure 1 A method flowchart provided for the embodiments of the present application;

[0045] Figure 2 Another flowchart provided for the embodiments of the present application;

[0046] Figure 3 A mobile robot laser radar installation schematic diagram;

[0047] Figure 4 A mobile robot laser radar installation schematic diagram provided for the embodiments of the present application;

[0048] Figure 5 An apparatus structure diagram provided for the embodiments of the present application;

[0049] Figure 6 An electronic device structure diagram provided for the embodiments of the present application. DETAILED DESCRIPTION

[0050] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is with reference to the drawings, in which like numerals represent like elements, throughout the several views. The following exemplary embodiments described herein represent implementations consistent with the present disclosure. The following description of the exemplary embodiments is not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0051] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0052] In order to better understand the technical solutions provided by the embodiments of the application, and to make the above-mentioned purposes, characteristics and advantages of the embodiments of the application more apparent and easy to understand, the technical solutions in the embodiments of the application will be further described in detail below with reference to the drawings.

[0053] Referring to Figure 1 , Figure 1 The method flowchart provided by the embodiments of the application. The flowchart can be applied to any mobile robot loaded with a laser radar and a camera, such as a logistics robot, a cleaning robot, etc.; the embodiments of the application are not limited to the types of laser radars and cameras, as long as the laser radar has the function of judging obstacles by collecting point clouds, and the camera has the function of shooting images, and the resolution of the images supports object recognition.

[0054] As a preferred embodiment, the laser radar herein can adopt a 2D laser radar, and the camera can adopt a monocular camera: in a conventional obstacle detection method, since the 2D laser radar can only detect a planar obstacle, and the monocular camera cannot judge a three-dimensional object that may be an obstacle from the image, these two low-cost sensors are difficult to be used for three-dimensional continuous obstacle detection of mobile robots; and in the technical solution provided by the embodiments of the application, the above defects can be overcome by the mutual cooperation of the camera and the laser radar, so as to achieve the effect of accurately detecting obstacles.

[0055] As Figure 1 shown, the flowchart can include the following steps:

[0056] Step 101, acquiring point cloud information and image information at the same time.

[0057] In the embodiment, the point cloud information is the information of the target scene collected by the laser radar installed on the mobile robot, and the image information is the information of the target scene collected by the camera installed on the mobile robot. The range of the target scene is determined relative to the mobile robot. For example, when the laser radar and the camera are installed on the front side of the mobile robot body and used to detect obstacles in front of the mobile robot, the range of the target scene is a certain range in front of the mobile robot and moves with the position of the mobile robot, so as to realize continuous obstacle detection of the robot during movement. The certain range can be set according to actual requirements and performance parameters such as the detection distance of the laser radar and the camera. For example, when the camera can collect images within a range of 5 meters*10 meters in front, the range can be set to 4 meters*6 meters or other ranges according to the robot speed or brake distance, etc. The embodiment is not limited in this regard.

[0058] In the embodiment, it is necessary to ensure that the obtained point cloud information and image information correspond to the same time, so as to facilitate subsequent obstacle judgment and calibration operation, avoid the same obstacle being misjudged as being at different positions in the laser radar and the camera due to continuous movement of the robot, and affect the obstacle judgment accuracy or accuracy. It should be noted that the same time herein refers to the same time corresponding to the two kinds of information used for subsequent judgment of obstacles. Whether the start time or the end time of collection of the two sensors is the same, whether the collection is performed simultaneously, whether the collection is performed in series or in parallel, etc. are not limited.

[0059] In step 102, if it is determined based on the image information that the target scene has a target object, it is determined whether the target object has been marked as an obstacle type. If yes, subsequent step 103 is performed, and if no, subsequent step 104 is performed.

[0060] In the embodiment, after the camera collects the image information of the target scene, object recognition can be performed on the image information to mark the target object. The process of object recognition on the image information can use object edge segmentation and other object recognition methods, and the embodiment is not limited in this regard. The target object includes ground patterns, paper, and other planar objects that do not affect the movement of the robot, and also includes steps, packages, furniture, and other three-dimensional objects that may affect the movement of the robot.

[0061] In the embodiment, when the camera is a monocular camera, it cannot independently determine whether the target object is an obstacle affecting the movement of the mobile robot due to its lack of ability to obtain depth, distance, and other information, and thus needs to be combined with point cloud information at the same time for determination. When the camera is a binocular camera or other device capable of distinguishing whether each target object is a planar or three-dimensional object, it can also be combined with point cloud information at the same time for verification to improve obstacle detection accuracy and accuracy, which is not limited in the embodiment.

[0062] In the embodiment, each target object can be labeled by the method in step 105 (see the subsequent embodiment for the specific labeling method, which is not described here). Since obstacle detection is a process that needs to be repeated continuously during the movement of the robot, the target object determined in the image information may have been labeled as an obstacle type in the previous round of obstacle detection, and thus needs to be confirmed before step 103 or 104 is executed.

[0063] Step 103, identifying the target object as an obstacle.

[0064] In the embodiment, if it is determined in step 102 that the target object has been labeled as an obstacle type, the target object can be identified as an obstacle.

[0065] As an optional embodiment, after determining that the target object in front of or in the moving direction of the mobile robot is an obstacle, an obstacle avoidance operation can be performed on the target object, such as re-planning the travel path to bypass the obstacle, controlling the robot to stop to avoid collision with the obstacle, etc. There are many specific measures after detecting the obstacle, and the processing method after detecting the obstacle is not limited in the embodiment.

[0066] Step 104, determining whether the target scene has a candidate object based on the point cloud information.

[0067] In the embodiment, if it is determined in step 102 that the target object has not been labeled as an obstacle type, subsequent steps 104 and 105 need to be executed to determine whether the target object is an obstacle in combination with the point cloud information collected by the laser radar.

[0068] In the embodiment, the laser radar collects data of the target scene to obtain point cloud information, and identifies all or part of the solid objects in the target scene as candidate objects based on the point cloud information. Optionally, any solid object higher than the driving plane can be regarded as an obstacle, and in this case, the candidate objects correspond to all solid objects in the target scene. Alternatively, only solid objects that can hinder the normal driving of the mobile robot can be regarded as obstacles, and in this case, the candidate objects can correspond to solid objects that exceed the chassis height of the mobile robot, solid objects that exceed the obstacle-crossing capability of the mobile robot, or solid objects within a specified range, etc. Objects that are not regarded as obstacles are not used as candidate objects for subsequent obstacle labeling in step 105.

[0069] In addition, all solid objects detected by the laser radar can also be regarded as candidate objects, but at the same time, the height information of each candidate object is labeled, and only candidate objects exceeding a preset height value are used for obstacle labeling in the subsequent step, etc. The effect of obstacle detection according to actual needs can also be achieved, and the specific implementation mode is not limited in the embodiment.

[0070] In step 105, if the candidate object exists and the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object, the target object is identified as an obstacle, and the target object is labeled as an obstacle type.

[0071] In the embodiment, when the point cloud information collected by the laser radar determines that the candidate object exists, it can be further judged whether the candidate object and the target object correspond to the same physical position in the actual scene, so as to judge whether the target object should be labeled as an obstacle type.

[0072] As an optional embodiment, whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object can be judged by the following method:

[0073] Based on the extrinsic parameters between the camera and the laser radar, a conversion matrix is determined.

[0074] Based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the radar coordinate system, and the conversion matrix, it is judged whether the physical positions corresponding to the candidate object and the target object are the same.

[0075] In the embodiment, the extrinsic parameter, i.e., the positional relationship between the camera and the lidar mounted on the same mobile robot, including the relative position, the relative angle, the relationship of the detection range, etc., can be determined by setting objects in the overlapping area of the detection ranges of the two sensors in the test stage for data calibration. The specific process of the extrinsic parameter calibration is not limited in the embodiment. The same object in the real scene can be presented in the corresponding two coordinate systems through the data collected by the camera and the lidar respectively. The conversion matrix is used to express the coordinate conversion relationship or the corresponding relationship between the camera coordinate system corresponding to the camera and the radar coordinate system corresponding to the lidar.

[0076] Further, there are various implementation manners to determine whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the radar coordinate system, and the conversion matrix. Herein, one optional manner is exemplarily given as follows:

[0077] The first coordinate position of the target object in the camera coordinate system is determined based on the image information.

[0078] The first coordinate position is converted into a second coordinate position in the radar coordinate system based on the conversion matrix.

[0079] The third coordinate position of the candidate object in the radar coordinate system is determined based on the point cloud information.

[0080] If the second coordinate position matches the third coordinate position, it is determined that the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object.

[0081] If the second coordinate position does not match the third coordinate position, it is determined that the physical position corresponding to the candidate object is different from the physical position corresponding to the target object.

[0082] In the embodiment, the coordinate position of the target object in the camera coordinate system can be determined according to the image information, and the coordinate position in the radar coordinate system can be converted from the coordinate position in the camera coordinate system according to the conversion matrix, so as to compare the coordinate position of the candidate object in the radar coordinate system. If the two match, it can be determined that the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object. The target object needs to be identified as an obstacle and marked as an obstacle type for subsequent continuous obstacle detection and obstacle avoidance operation. If the coordinate position of the candidate object in the radar coordinate system does not match the coordinate position converted from the target object, it can be determined that the physical position corresponding to the candidate object is different from the physical position corresponding to the target object, and it is not necessary to mark it as an obstacle type.

[0083] Similarly, there are a variety of other optional ways that have the same actual effect as the above judgment method, for example, the coordinate position of the candidate object in the radar coordinate system can be converted to the coordinate position in the camera coordinate system through a conversion matrix, so as to judge whether the converted coordinate position of the candidate object matches the coordinate position of the target object in the camera coordinate system; the coordinates in the camera coordinate system and the radar coordinate system can also be converted to a third-party coordinate system through a conversion matrix respectively, so as to judge whether the coordinates of the target object and the candidate object in the third-party coordinate system match, and the like. The present embodiment is not limited in this regard.

[0084] As an optional embodiment, if multiple candidate objects can be determined from the point cloud information at the same time, at least one candidate object has the same physical position as the target object, that is, the target object can be identified as an obstacle; similarly, if multiple target objects can be determined from the image information at the same time, the target object that matches the candidate object coordinate can be identified as an obstacle, and the target object that does not match the coordinate is not identified as an obstacle, and the like. The present embodiment is not limited in this regard.

[0085] As an optional embodiment, the detection range of the laser radar can be calibrated in the image collected by the camera. When it is determined that the physical position corresponding to the target object is within the detection range of the laser radar, and there is no candidate object at the position, the target object can be marked as a non-obstacle type. For the target object in the image information that has been marked as a non-obstacle type, subsequent judgment of whether it is an obstacle can not be combined with the point cloud information, thereby reducing the consumption of computing resources. If the target object has not been marked as an obstacle, and its corresponding physical position is not within the detection range of the laser radar, the target object will not be marked as a type, so as to avoid the corresponding three-dimensional object being marked as a non-obstacle type when it enters the detection range of the laser radar subsequently, resulting in that the mobile robot cannot correctly identify the obstacle.

[0086] For example, when the above laser radar adopts a 2D laser radar, the detection range thereof is a plane, and when the laser radar is installed obliquely downward, the detection area thereof is a plane with an angle with respect to the ground. During the movement of the mobile robot, if a three-dimensional object has not entered the detection area, the object corresponding candidate object will not exist in the point cloud information, and thus even if the corresponding candidate object does not exist at the physical position of the target object, it cannot be determined that the target object is necessarily a non-obstacle, and the target object should not be marked as a non-obstacle type, so as to avoid affecting the subsequent determination. When the above object is just in the detection area of the laser radar, the object can be identified as an obstacle based on the above determination manner in step 105 and marked as an obstacle type. Thereafter, even if the object leaves the detection area of the laser radar due to movement of the object or movement of the robot, etc., as long as the object is still in the image information of the camera, the object will still exist as a target object marked as an obstacle type, and the existing mark will not change because the object leaves the detection area of the laser radar, so as to avoid obstacle detection failure due to the obstacle leaving the detection range of the laser, and collision.

[0087] At this point, the process shown in Figure 1 is completed.

[0088] As can be seen from the process shown in Figure 1 , when the mobile robot loaded with the laser radar and the camera needs to detect an obstacle in the present embodiment, the target object existing in a target scene is determined by using the camera, and then the target object with the same physical position as the candidate object determined based on the point cloud information of the scene by the laser radar is marked as an obstacle, so as to accurately identify the obstacle in the target scene, and realize the three-dimensional continuous obstacle avoidance effect of the mobile robot. The obstacle detection manner based on cooperation of the camera and the laser radar in the above scheme has the advantages of lower requirement for sensor performance and higher detection efficiency compared with the traditional detection manner using only the camera or the laser radar.

[0089] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the embodiments of the present application also provide a flowchart as shown in Figure 2 to implement the above method disclosed by the embodiments of the present application.

[0090] As an optional embodiment, the laser radar loaded by the mobile robot in the present embodiment is a 2D laser radar, and compared with the conventional installation manner of the 2D laser radar as shown in Figure 3 , the 2D laser radar in the present embodiment is installed in the manner as shown in Figure 4 . As shown in Figure 3The 2D laser radar shown is horizontally mounted on the vehicle body. Since the 2D laser radar can only detect obstacles in one plane, it cannot detect low or suspended obstacles outside the plane, and there is a safety hazard of the mobile robot colliding with the obstacles.

[0091] However, in the installation mode shown in the figure, the 2D laser radar can detect obstacles within the range of the inclined plane in the figure, so that obstacles not higher than the installation position of the mobile robot can be detected during the movement of the mobile robot. The specific installation height or the angle between the detection plane and the ground can be set according to actual needs, and this embodiment does not limit it. Figure 4 Based on the installation mode shown in the figure, since the detection plane of the 2D laser radar has an angle with the movement direction of the mobile robot, there may be a problem that the same obstacle cannot be continuously detected, that is, when the mobile robot approaches the obstacle, the obstacle enters the blind area of the laser radar detection, and it cannot be determined whether the obstacle still exists in front of the robot. In the case of relying solely on the 2D laser radar for obstacle detection and obstacle avoidance, if the obstacle is out of the field of view and continues to move, there is a possibility of collision with the obstacle; if the obstacle is detected and the vehicle is stopped, the obstacle may enter the detection blind area during deceleration or continue to exist in the detection range after stopping, and frequent manual intervention is required to restore the movement of the mobile robot, and the intelligent degree of the obstacle avoidance process is low.

[0092] Figure 4 Therefore, in the flow shown in the figure, the mobile robot also carries a monocular camera in the same direction as the 2D laser radar, and adopts a mode in which the 2D laser radar cooperates with the monocular camera to continuously detect obstacles out of the detection range of the laser radar by using the characteristic that the viewing angle range of the monocular camera is usually larger than that of the 2D laser radar. Specifically, the flow can include the following steps:

[0093] Figure 2

[0094] Step 201, determining whether the laser radar detects an obstacle.

[0095] In this embodiment, the laser radar needs to continuously determine whether there is an obstacle in its detection plane. The obstacle here can refer to any three-dimensional object above the ground, or a three-dimensional object above a certain height can be set to be identified as an obstacle according to actual needs, for example, for a mobile robot with strong obstacle crossing ability, objects such as low thresholds and carpets will not be identified as obstacles.

[0096] When the laser radar detects an obstacle, it will be presented as a candidate object in the point cloud information it collects, and the position of the object in the point cloud will be recorded for subsequent obstacle marking of the objects identified in the image information.​​​

[0097] Optionally, in the present embodiment, it can be provided that the laser radar judges that an obstacle is detected, and then the subsequent marking operation is performed, otherwise step 201 is repeated until an obstacle is detected. Similarly, it can also be provided that whether there is an object in the image captured by the camera is preferentially judged, and the like, and the present embodiment does not limit this.

[0098] Step 202, edge segmentation is performed on the environment object at the same time based on the image.

[0099] As an optional embodiment, when the laser radar detects an obstacle at a certain time, all objects in the image information at the same time captured by the camera are identified and the corresponding regions of the objects in the image are determined by using the edge segmentation technology. Since the image captured by the monocular camera lacks depth or distance parameters, it is impossible to confirm alone whether the objects identified in the image are planar objects or three-dimensional objects, and it is necessary to combine the point cloud information to determine whether each object in the image information is an obstacle.

[0100] Optionally, object identification can be performed on the image information at the corresponding time only after the laser radar detects an obstacle, and the object identification is not performed on the image information when the laser radar does not detect an obstacle. The laser radar obstacle detection and the object identification in the image can also be performed separately to ensure that the object identification result at the same time can be obtained immediately after the laser radar detects an obstacle, and the efficiency and timeliness of obstacle detection are improved.

[0101] Step 203, corresponding to the object detected by the laser radar, the corresponding region in the image is determined, and the corresponding object in the image is marked as an obstacle.

[0102] In the present embodiment, based on the position of the obstacle in the point cloud obtained in step 201 and the position of each object in the image obtained in step 202, the object corresponding to the same physical position as the obstacle is determined from each object determined by image recognition, and is marked as an obstacle. If multiple obstacles are detected in step 201, multiple obstacles are marked in step 203. The specific determination method can refer to the related content in step 105, which is not repeated here.

[0103] Step 204, real-time association and matching are performed on the continuous frames of the image.

[0104] In this embodiment, since the detection range of the 2D laser radar is generally smaller than the monocular camera, during the movement of the mobile robot, there can be a situation that the obstacle enters the blind area of the laser radar detection, but is still within the detection range of the monocular camera. At this time, since the monocular camera continuously performs image acquisition, the label of the object based on steps 201-203 in the subsequent image frame can be matched in real time, so as to continuously determine the position of the obstacle in the image. The specific matching manner is not limited in this embodiment, for example, the position of the obstacle in the image can be continuously determined by referring to the foregoing steps 102 and 103, that is, whether the object in the image information has been labeled as an obstacle type at a past time.

[0105] Step 205: determining whether the above object is continuously detected in the image;

[0106] Step 206: when the above object can be continuously detected, it is determined that the obstacle affects the driving safety of the mobile robot, and an obstacle avoidance operation is performed.

[0107] Step 207: when the above object cannot be continuously detected, it is determined that the obstacle is removed or dynamically away from the field of view of the mobile robot, and no obstacle avoidance operation is performed.

[0108] In this embodiment, after the obstacle is labeled, the mobile robot can perform an obstacle avoidance operation for the obstacle to avoid collision with the obstacle during driving.

[0109] There are various optional embodiments for how to determine whether to perform obstacle avoidance and how to perform obstacle avoidance, for example, when the mobile robot can continuously detect the above object in the image during driving, and the continuous time exceeds a preset time length, it is considered that the object continuously exists on the driving path, and the obstacle avoidance operation needs to be performed for it. Otherwise, it is considered that the object is located on the side of the driving path or has left the driving path, and no obstacle avoidance operation needs to be performed. The above time length can be set based on the driving speed and detection distance of the mobile robot, for example, when the driving speed is faster and the detection distance is closer, a relatively shorter time length is set.

[0110] Similarly, other ways can also be used as the judgment condition of whether to perform the obstacle avoidance operation, for example, a certain area is set in the image shot by the monocular camera, corresponding to the part of the detection range close to the mobile robot, when the obstacle appears in this part of the area, it is determined that the obstacle avoidance operation needs to be performed, and the like. The specific judgment manner is not limited in this embodiment.

[0111] Optionally, when it is determined that the obstacle avoidance operation needs to be performed for the obstacle, there are various obstacle avoidance manners that can be adopted, for example, re-planning a driving path to bypass the obstacle, controlling the robot to slow down or stop to avoid collision with the obstacle, etc., and the embodiments are not limited thereto.

[0112] Thus far, the description of the method flow example shown in Figure 2 is completed.

[0113] The above describes the method provided by the embodiments of the present application, and the following describes the device provided by the embodiments of the present application.

[0114] Referring to Figure 5 , Figure 5 the device structure diagram provided by the embodiments of the present application. The device corresponds to the method flow shown in Figure 1 , and is applied to a mobile robot including a laser radar and a camera, the laser radar is used to acquire point cloud information of a target scene, and the camera is used to acquire image information of the target scene.

[0115] As shown in Figure 5 , the device can include:

[0116] An information acquisition unit 501 is configured to acquire point cloud information and image information at the same time.

[0117] A label detection unit 502 is configured to determine whether a target object in the target scene has been labeled as an obstacle type based on the image information.

[0118] A label confirmation unit 503 is configured to identify the target object as an obstacle if the target object has been labeled as an obstacle type.

[0119] An obstacle identification unit 504 is configured to determine whether there is a candidate object in the target scene based on the point cloud information if the target object has not been labeled as an obstacle type.

[0120] The obstacle identification unit 504 is further configured to identify the target object as an obstacle and label the target object as an obstacle type if there is a candidate object and the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object.

[0121] In a possible implementation, after the obstacle identification unit 504 determines whether there is a candidate object in the target scene based on the point cloud information, the obstacle identification unit 504 is further configured to identify the target object as a non-obstacle if there is no candidate object or the physical position of the existing candidate object is different from the physical position corresponding to the target object.

[0122] In a possible implementation, after the obstacle identification unit 504 identifies the target object as an obstacle, the obstacle identification unit 504 is further configured to perform an obstacle avoidance operation on the target object.

[0123] In a possible implementation, the obstacle identification unit 504 determines whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object by: determining a conversion matrix based on the extrinsic parameters between the camera and the lidar, where the conversion matrix is used to represent a coordinate conversion relationship between a camera coordinate system of the camera and a radar coordinate system of the lidar; and determining whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on a coordinate position of the target object in the camera coordinate system, a coordinate position of the candidate object in the radar coordinate system, and the conversion matrix.

[0124] In a possible implementation, the obstacle identification unit 504 determines whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the radar coordinate system, and the conversion matrix by: determining a first coordinate position of the target object in the camera coordinate system based on the image information; converting the first coordinate position into a second coordinate position in the radar coordinate system based on the conversion matrix; determining a third coordinate position of the candidate object in the radar coordinate system based on the point cloud information; determining that the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object if the second coordinate position matches the third coordinate position; and determining that the physical position corresponding to the candidate object is different from the physical position corresponding to the target object if the second coordinate position does not match the third coordinate position.

[0125] In a possible implementation, the lidar is a 2D lidar, and the camera is a monocular camera; the 2D lidar is installed on the mobile robot in a downward tilt manner, and a detection region of the 2D lidar overlaps a detection region of the monocular camera; and the extrinsic parameters in the obstacle identification unit 504 are determined based on relative installation positions of the 2D lidar and the monocular camera.

[0126] So far, the structure of the apparatus is described. Figure 5

[0127] The embodiments of the present application also provide a hardware structure of the apparatus. Referring to FIG. 2, the apparatus includes a processor 201, a memory 202, and a bus 203. Figure 5 Figure 6 Figure 6 The electronic device structure diagram provided by the embodiments of the present application. As shown in FIG. 2, the electronic device includes a processor 201, a memory 202, and a bus 203. Figure 6 ​​​As shown, the hardware structure can include a processor and a machine readable storage medium storing machine executable instructions capable of being executed by the processor; and the processor is configured to execute the machine executable instructions to implement the method disclosed in the above examples of the present application.

[0128] Based on the same application concept as the above method, the embodiments of the present application further provide a machine readable storage medium, wherein the machine readable storage medium stores a plurality of computer instructions, and the computer instructions can implement the method disclosed in the above examples of the present application when executed by a processor.

[0129] For example, the machine readable storage medium can be a RAM (Random Access Memory), a volatile memory, a non-volatile memory, a flash memory, a storage drive (such as a hard drive), a solid state drive, any type of storage disk (such as an optical disk, a DVD, etc.), or similar storage medium, or a combination thereof.

[0130] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an e-mail device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0131] For the convenience of description, the above device is described as various units by functions respectively in the description. Of course, the functions of each unit can be implemented in one or more software and / or hardware in the implementation of the present application.

[0132] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to a magnetic disk storage, a CD-ROM, an optical storage etc.) containing computer usable program code.

[0133] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0134] Also, these computer program instructions can be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0135] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0136] The above description is only some embodiments of the present application, and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. An obstacle detection method, characterized in that, An application is made to a mobile robot that includes a LiDAR and a camera, wherein the LiDAR is a 2D LiDAR and the camera is a monocular camera; the 2D LiDAR is mounted on the mobile robot at an angle downwards, the direction of the monocular camera is the same as the direction of the 2D LiDAR, and there is an overlap between the detection areas of the 2D LiDAR and the detection areas of the monocular camera; The lidar is used to acquire point cloud information of the target scene, the camera is used to acquire image information of the target scene, and the method includes: Acquire point cloud information and image information at the same time point; If it is determined based on the image information that a target object exists in the target scene, then it is determined whether the target object has been marked as an obstacle type; If so, the target object is identified as an obstacle; If not, then determine whether there are candidate objects in the target scene based on the point cloud information; If a candidate object exists, and the physical location corresponding to the candidate object is the same as the physical location corresponding to the target object, then the target object is identified as an obstacle and marked as an obstacle type. If no candidate object exists, or if the physical location of an existing candidate object is different from the physical location of the target object, then the target object is identified as a non-obstacle.

2. The method according to claim 1, characterized in that, After identifying the target object as an obstacle, the method further includes: Perform obstacle avoidance operations on the target object.

3. The method according to claim 1, characterized in that, The physical location of the candidate object is the same as the physical location of the target object, determined by the following method: Based on the extrinsic parameters between the camera and the lidar, a transformation matrix is ​​determined; wherein, the transformation matrix is ​​used to represent the coordinate transformation relationship between the camera coordinate system of the camera and the lidar coordinate system; Based on the coordinates of the target object in the camera coordinate system, the coordinates of the candidate object in the radar coordinate system, and the transformation matrix, it is determined whether the physical position of the candidate object is the same as the physical position of the target object.

4. The method according to claim 3, characterized in that, The step of determining whether the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the radar coordinate system, and the transformation matrix includes: Based on the image information, determine the first coordinate position of the target object in the camera coordinate system; Based on the transformation matrix, the first coordinate position is converted into a second coordinate position in the radar coordinate system; Based on the point cloud information, determine the third coordinate position of the candidate object in the radar coordinate system; If the second coordinate position matches the third coordinate position, then it is determined that the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object; If the second coordinate position does not match the third coordinate position, then it is determined that the physical position corresponding to the candidate object is different from the physical position corresponding to the target object.

5. The method according to claim 3, characterized in that, The extrinsic parameters are determined based on the relative installation positions of the 2D lidar and the monocular camera.

6. An obstacle detection device, characterized in that, An application is made to a mobile robot that includes a LiDAR and a camera, wherein the LiDAR is a 2D LiDAR and the camera is a monocular camera; the 2D LiDAR is mounted on the mobile robot at an angle downwards, the direction of the monocular camera is the same as the direction of the 2D LiDAR, and there is an overlap between the detection areas of the 2D LiDAR and the detection areas of the monocular camera; The lidar is used to acquire point cloud information of the target scene, and the camera is used to acquire image information of the target scene. The device includes: The information acquisition unit is used to acquire point cloud information and image information at the same time. The marker detection unit is used to determine whether the target object has been marked as an obstacle type if it is determined that there is a target object in the target scene based on the image information; The marking confirmation unit is used to identify the target object as an obstacle if the target object has been marked as an obstacle type. An obstacle recognition unit is used to determine whether a candidate object exists in the target scene based on the point cloud information if the target object is not labeled as an obstacle type. The obstacle recognition unit is further configured to: if there is a candidate object and the physical position corresponding to the candidate object is the same as the physical position corresponding to the target object, then identify the target object as an obstacle and mark the target object as an obstacle type; the obstacle recognition unit is further configured to: if there is no candidate object, or if the physical position of the existing candidate object is different from the physical position corresponding to the target object, then identify the target object as a non-obstacle.

7. The apparatus according to claim 6, Its features are, in, After the obstacle recognition unit identifies the target object as an obstacle, it is also used to: perform obstacle avoidance operation on the target object; Specifically, the obstacle recognition unit determines whether the physical location of the candidate object is the same as the physical location of the target object by: determining a transformation matrix based on the extrinsic parameters between the camera and the lidar; wherein the transformation matrix represents the coordinate transformation relationship between the camera coordinate system and the lidar coordinate system; and determining whether the physical locations of the candidate object and the target object are the same based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the lidar coordinate system, and the transformation matrix. Specifically, when the obstacle recognition unit determines whether the physical position of the candidate object is the same as the physical position of the target object based on the coordinate position of the target object in the camera coordinate system, the coordinate position of the candidate object in the radar coordinate system, and the transformation matrix, it performs the following steps: determining a first coordinate position of the target object in the camera coordinate system based on the image information; converting the first coordinate position to a second coordinate position in the radar coordinate system based on the transformation matrix; determining a third coordinate position of the candidate object in the radar coordinate system based on the point cloud information; if the second coordinate position matches the third coordinate position, then it is determined that the physical position of the candidate object is the same as the physical position of the target object; if the second coordinate position does not match the third coordinate position, then it is determined that the physical position of the candidate object is different from the physical position of the target object. The extrinsic parameters in the obstacle recognition unit are determined based on the relative installation positions of the 2D lidar and the monocular camera.

8. An electronic device, characterized in that, The electronic device includes: a processor and a machine-readable storage medium; The machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1-5.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-readable instructions, which, when invoked and executed by a processor, cause the processor to implement the method described in any one of claims 1-5.

Images