An obstacle detection method, a mobile robot, and a machine-readable storage medium

Abstract

The present application provides an obstacle detection method, a mobile robot, and a machine-readable storage medium. The method includes: collecting laser data of a target scene through a laser sensor, and collecting image data of the target scene through an image sensor; determining first feature information of a target object in the target scene based on the laser data; determining second feature information of a candidate object in the target scene based on the image data; and determining whether the target object is an obstacle based on the first feature information and the second feature information. Through the technical solution of the present application, it is possible to accurately detect obstacles in a target scene based on laser data and image data, better perform obstacle detection and avoidance, greatly improve the accuracy of obstacle detection, and greatly reduce the possibility of collision between the mobile robot and the obstacle on the premise of reducing false alarms, so as to achieve the obstacle avoidance function.

Description

Technical Field

[0001] The present application relates to the field of mobile robots, and in particular, to an obstacle detection method, a mobile robot, and a machine-readable storage medium. Background Art

[0002] In recent years, various types of mobile robots have developed rapidly in terms of technology and market. A mobile robot is a machine device that automatically performs tasks and is a machine that relies on its own power and control capabilities to achieve various functions. A mobile robot can accept human commands, run pre-programmed programs, or act according to strategies formulated by artificial intelligence. For example, a user uses a manual remote control to control a mobile robot to perform related operations, such as sending an operation command to the mobile robot wirelessly through the manual remote control. After receiving the operation command, the mobile robot executes the operation specified by the operation command to complete related functions.

[0003] When a mobile robot moves in a target scenario, it can locate itself during the movement and construct an incremental map based on its own location. Moreover, during the movement of the mobile robot, the mobile robot needs to detect obstacles in the target scenario and perform obstacle avoidance operations based on the positions of the obstacles.

[0004] In order to detect obstacles in a target scenario, a mobile robot needs to deploy an image sensor to collect RGB images of the target scenario and detect obstacles in the target scenario based on the RGB images. However, when detecting obstacles through RGB images, there will be false detections. For example, the influence of light will cause light and shadow to be recognized as obstacles, resulting in the inability to accurately detect obstacles in the target scenario. Summary of the Invention

[0005] The present application provides an obstacle detection method applied to a mobile robot including a laser sensor and an image sensor. The laser sensor is used to collect laser data of a target scenario, and the image sensor is used to collect image data of the target scenario. The method includes:

[0006] Determining first feature information of a target object in the target scenario based on the laser data;

[0007] Determining second feature information of a candidate object in the target scenario based on the image data;

[0008] Determining whether the target object is an obstacle based on the first feature information and the second feature information.

[0009] The present application provides a mobile robot, including:

[0010] A laser sensor for collecting laser data of a target scenario;

[0011] An image sensor for collecting image data of the target scene;

[0012] A processor for determining first feature information of a target object in the target scene based on the laser data; determining second feature information of a candidate object in the target scene based on the image data; and determining whether the target object is an obstacle based on the first feature information and the second feature information.

[0013] The present application provides a mobile robot, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; wherein, the processor is configured to execute the machine-executable instructions to implement the obstacle detection method in the above examples of the present application.

[0014] The present application provides a machine-readable storage medium having a number of computer instructions stored thereon, which, when executed by a processor, are capable of implementing the obstacle detection method in the above examples of the present application.

[0015] As can be seen from the above technical solutions, in the embodiments of the present application, laser data of a target scene can be collected by a laser sensor, and image data of the target scene can be collected by an image sensor, and obstacles in the target scene can be accurately detected based on the laser data and the image data, that is, the fusion of obstacle detection for the laser sensor and the image sensor is performed, so as to better perform obstacle detection and avoidance, greatly improve the accuracy of obstacle detection, greatly reduce the possibility of collision between the mobile robot and an obstacle on the premise of reducing false alarms, and better implement the obstacle avoidance function. Combining the advantages of the laser sensor and the image sensor, dense and accurate obstacle depth estimation can be achieved. The use of image data can overcome the detection defects of laser data, such as the difficulty in detecting low obstacles, black obstacles, etc., and the use of laser data can overcome the defects of image data, such as the problem of identifying light and shadow as obstacles due to the influence of light. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for describing the embodiments of the present application or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application, and those of ordinary skill in the art can also obtain other drawings according to these drawings of the embodiments of the present application.

[0017] Figure 1 It is a flowchart of an obstacle detection method in an embodiment of the present application;

[0018] Figure 2It is a schematic diagram of the positional relationship between a laser sensor and an image sensor;

[0019] Figure 3 It is a schematic flowchart of an obstacle detection method in an embodiment of the present application;

[0020] Figure 4A It is a schematic diagram of the position coordinate frame of a candidate object;

[0021] Figure 4B It is a schematic diagram of the position coordinate frames of a candidate object in two frames of RGB images;

[0022] Figure 4C It is a schematic diagram of obtaining three-dimensional coordinates by using the triangulation method;

[0023] Figure 5 It is a view of the XOY plane;

[0024] Figure 6 It is a schematic flowchart of an obstacle detection method in an embodiment of the present application;

[0025] Figure 7 It is a schematic flowchart of an obstacle detection method in an embodiment of the present application;

[0026] Figure 8 It is a schematic structural diagram of an obstacle detection device in an embodiment of the present application;

[0027] Figure 9 It is a hardware structure diagram of a mobile robot in an embodiment of the present application. Specific embodiments

[0028] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and do not limit the present application. The singular forms of "a", "the" and "said" used in the present application and the claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "when" or "while" or "in response to determining".

[0030] In an embodiment of the present application, an obstacle detection method is proposed, which is used to detect whether there are obstacles in a target scene and can be applied to a mobile robot including a laser sensor and an image sensor. The laser sensor is used to collect laser data of the target scene, and the image sensor is used to collect image data of the target scene. Refer to Figure 1 As shown in

[0031] Step 101: Determine first feature information of a target object in the target scene based on the laser data.

[0032] Step 102: Determine second feature information of candidate objects in the target scene based on the image data.

[0033] Step 103: Determine whether the target object is an obstacle based on the first feature information and the second feature information.

[0034] In a possible implementation manner, the first feature information may include the initial height and three-dimensional coordinates of the target object, and the second feature information may include the suspicious area and object type of the candidate object. On this basis, if the initial height is greater than a first preset threshold and less than a second preset threshold, and the three-dimensional coordinates are within the suspicious area, the target height corresponding to the object type can be determined. Further, if the initial height matches the target height, it can be determined that the target object is an obstacle; or, if the initial height does not match the target height, it can be determined that the target object is not an obstacle.

[0035] In another possible implementation manner, the first feature information may include the number of echo signals and three-dimensional coordinates of the target object, and the second feature information may include the suspicious area and object color of the candidate object. On this basis, if it is determined that the area where the target object is located is a void area based on the number of echo signals, and the three-dimensional coordinates are within the suspicious area, it can be determined whether the object color is black. Further, if the object color is black, it can be determined that the target object is an obstacle; or, if the object color is not black, it can be determined that the target object is not an obstacle.

[0036] In another possible implementation manner, the first feature information may include the initial height. On this basis, after determining the first feature information of the target object in the target scene based on the laser data, if the initial height is not less than the second preset threshold, it can be determined that the target object is an obstacle.

[0037] In the above embodiments, determining the first feature information of the target object in the target scene based on the laser data may include, but is not limited to, the following methods: determining the two-dimensional coordinates and the number of echo signals of the target object based on the laser data; determining the three-dimensional coordinates of the target object based on the two-dimensional coordinates, the pose of the laser sensor, and the pose of the mobile robot, and determining the initial height of the target object based on the three-dimensional coordinates. On this basis, the first feature information of the target object may be determined based on the number of echo signals, the three-dimensional coordinates, and the initial height.

[0038] In the above embodiments, determining the second feature information of the candidate object in the target scene based on the image data may include, but is not limited to, the following methods: the image data may be input into a trained target network model to obtain the position coordinate frame of the candidate object, the object type of the candidate object, and the object color in the image data; then, the candidate feature points within the position coordinate frame are selected, and the target feature points corresponding to the candidate feature points are selected from the image data of the previous frame of the image data; then, the three-dimensional coordinates corresponding to the candidate feature points are determined based on the pixel coordinates of the candidate feature points and the pixel coordinates of the target feature points, and the suspicious area corresponding to the candidate object is determined based on the three-dimensional coordinates. On this basis, the second feature information of the candidate object is determined based on the suspicious area, the object type, and the object color.

[0039] Among them, the position coordinate frame may include multiple candidate feature points, and the three-dimensional coordinates corresponding to each candidate feature point include the horizontal axis coordinate, the vertical axis coordinate, and the vertical axis coordinate. Determining the suspicious area corresponding to the candidate object based on the three-dimensional coordinates may include, but is not limited to: selecting the minimum horizontal axis coordinate and the maximum horizontal axis coordinate from the horizontal axis coordinates corresponding to all candidate feature points, selecting the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points, and selecting the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points; based on the minimum horizontal axis coordinate, the maximum horizontal axis coordinate, the minimum vertical axis coordinate, the maximum vertical axis coordinate, the minimum vertical axis coordinate, and the maximum vertical axis coordinate, the suspicious area corresponding to the candidate object is determined.

[0040] As can be seen from the above technical solutions, in the embodiments of the present application, laser data of a target scene can be collected by a laser sensor, and image data of the target scene can be collected by an image sensor. Based on the laser data and the image data, obstacles in the target scene can be accurately detected, that is, the fusion of obstacle detection for the laser sensor and the image sensor is performed, so as to better perform obstacle detection and avoidance, greatly improving the accuracy of obstacle detection. On the premise of reducing false alarms, the possibility of collision between the mobile robot and the obstacle is greatly reduced, and the obstacle avoidance function is better realized. By combining the advantages of the laser sensor and the image sensor, dense and accurate obstacle depth estimation can be achieved. The use of image data can overcome the detection defects of laser data, such as the difficulty in detecting low obstacles, black obstacles, etc. The use of laser data can overcome the defects of image data, such as the problem that light influence causes the recognition of light and shadow as obstacles.

[0041] The above technical solutions of the embodiments of the present application will be described below in conjunction with specific application scenarios.

[0042] During the movement of the mobile robot, the mobile robot needs to detect obstacles in the target scene (that is, the scene passed by the mobile robot. For example, when the mobile robot moves indoors, the indoor scene is used as the target scene), and perform obstacle avoidance operations based on the positions of the obstacles. In order to detect obstacles in the target scene, the mobile robot can deploy an image sensor, collect the RGB image of the target scene through the image sensor, and detect obstacles in the target scene based on the RGB image. However, when detecting obstacles through the RGB image, there will be misdetection. For example, light influence will cause the recognition of light and shadow as obstacles, resulting in the inability to accurately detect obstacles in the target scene, that is, the accuracy of obstacle detection is relatively poor.

[0043] For another example, the mobile robot can deploy a laser sensor, collect the laser data of the target scene through the laser sensor, and detect obstacles in the target scene based on the laser data. However, since the laser sensor can only detect the height information of an object and lacks object attributes, and for objects of the same height, some can be crossed by the mobile robot, such as thresholds, and some cannot be crossed by the mobile robot, such as wires, and the characteristics of the laser data of the two are basically the same, it is impossible to distinguish these two types of objects based on the laser data. In some scenarios, the laser sensor has inaccurate ranging, resulting in misdetection of obstacles, such as inaccurate ranging for marble surfaces. In some scenarios, the laser sensor cannot detect black obstacles.

[0044] In view of the above findings, in the embodiments of the present application, a laser sensor and an image sensor are deployed on the mobile robot simultaneously. The laser sensor can be a radar sensor or other types of sensors, as long as it has a ranging function. The laser sensor can be a single-line laser sensor or a multi-line laser sensor, and there is no limitation in this regard. In this embodiment, a single-line laser sensor is taken as an example. The image sensor can be an RGB camera or other types of sensors, as long as it has an image acquisition function. The image sensor can be a monocular sensor or a multiocular sensor, and there is no limitation in this regard. In this embodiment, a monocular sensor is taken as an example.

[0045] Referring to Figure 2 As shown, it is a schematic diagram of the positional relationship between the laser sensor and the image sensor. The image sensor can be located above the laser sensor. Of course, the laser sensor can also be located above the image sensor. Among them, the field of view range of the laser sensor and the field of view range of the image sensor can completely overlap or partially overlap, and there is no limitation in this regard, as long as there is an overlapping area between the field of view ranges of the two.

[0046] In practical applications, the number of laser sensors can be one or at least two, and the number of image sensors can be one or at least two, and there is no limitation in this regard. For the convenience of description, in this embodiment, one laser sensor and one image sensor are taken as examples for illustration.

[0047] In the embodiments of the present application, the laser data of the target scene can be collected by the laser sensor, and the image data of the target scene can be collected by the image sensor, and the laser data and the image data are fused, so as to accurately detect the obstacles in the target scene and better perform obstacle detection and avoidance.

[0048] In the embodiments of the present application, it involves an analysis process based on image data, an analysis process based on laser data, and an obstacle fusion detection process. In the analysis process based on image data, it is necessary to determine the feature information of each object in the target scene (that is, the feature information obtained based on the image data). For the convenience of distinction, the objects in the target scene are called candidate objects, and the feature information of the candidate objects is called the second feature information. In the analysis process based on laser data, it is necessary to determine the feature information of each object in the target scene (that is, the feature information obtained based on the laser data). For the convenience of distinction, the objects in the target scene are called target objects, and the feature information of the target objects is called the first feature information. In the obstacle fusion detection process, obstacle detection can be performed based on the second feature information of the candidate objects and the first feature information of the target objects.

[0049] First, the analysis process based on image data, referring to Figure 3 As shown, this process may include:

[0050] Step 301: Collect the image data of the target scene through an image sensor. This image data can be an RGB image or other types of images, and there is no restriction on the type of this image data.

[0051] Step 302: Input the RGB image into the trained target network model to obtain the position coordinate box of the candidate object in the RGB image, the object type of the candidate object, and the object color of the candidate object.

[0052] Exemplarily, in order to detect the position coordinate box, object type, and object color of the candidate object, it involves the training process and the detection process of the network model. During the training process of the network model, it is necessary to pre-train the target network model in advance. This training process is executed before step 302 and is implemented on a training device (or training platform), and there is no restriction on the type of this training device as long as it can train the target network model. After training the target network model, the target network model can be deployed to the mobile robot. During the detection process of the network model, the mobile robot can use the target network model to detect the position coordinate box, object type, and object color of the candidate object.

[0053] For the training process of the network model, an initial network model can be configured, and there is no restriction on the structure and function of this initial network model as long as the initial network model is used to detect the position coordinate box, object type, and object color of the object. For example, the initial network model can be a network model based on a deep learning algorithm, a network model based on a neural network, or other types of network models. As an example, the initial network model can be the YOLOV3 (You Only Look Once) model. Of course, the YOLOV3 model is just an example of the initial network model.

[0054] In order to train the network model, a training dataset can be obtained. This training dataset includes a large number of sample images (such as RGB-type sample images). For each sample image, the label information corresponding to the sample image includes: the position coordinate box of the object in the sample image, the object type, and the object color.

[0055] Based on the training dataset and the initial network model, the initial network model can be trained with the training dataset. There is no restriction on this training process, and the trained network model is obtained. For the convenience of distinction, the trained network model can be denoted as the target network model. Obviously, since the initial network model is used to detect the position coordinate box, object type, and object color of the candidate object, the target network model is used to detect the position coordinate box, object type, and object color of the candidate object. In practical applications, if the initial network model is the YOLOV3 model, then the target network model is the YOLOV3 model.

[0056] For the detection process of the network model, in step 302, after obtaining the RGB image, the RGB image can be input into the target network model. Since the target network model is used to detect the position coordinate frame, object type, and object color of the candidate objects in the RGB image, the target network model can output features such as the position coordinate frame, object type, and object color of the candidate objects in the RGB image.

[0057] Exemplarily, the position coordinate frame is used to represent the position of the candidate object in the RGB image. The position coordinate frame can be a rectangular coordinate frame, which can be represented by the coordinates of any corner point of the rectangular coordinate frame (such as the upper left corner point, or the upper right corner point, or the lower left corner point, or the lower right corner point), and the length and height of the rectangular coordinate frame, or can be represented by the coordinates of the four corner points of the rectangular coordinate frame (such as the upper left corner point, the upper right corner point, the lower left corner point, and the lower right corner point). Of course, the above are only two examples, and there is no limitation on this.

[0058] Exemplarily, the object type is used to represent the type of the candidate object in the RGB image, such as wire, sock, shoe, trash can, etc. There is no limitation on the type of the candidate object in this embodiment.

[0059] Exemplarily, the object color is used to represent the color of the candidate object in the RGB image. In one example, the object color is divided into black and non - black (i.e., all colors other than black). In another example, the object color is divided into black, red, blue, green, etc., that is, including various colors.

[0060] See Figure 4A As shown, it is an example of the position coordinate frame of the candidate object detected by the target network model. Obviously, through this position coordinate frame, the candidate object can be found from the RGB image.

[0061] Step 303: Select the candidate feature points within the position coordinate frame from the RGB image, and select the target feature points corresponding to the candidate feature points from the previous frame of RGB image of this RGB image.

[0062] Exemplarily, in the image processing process, the feature point can be a point where the image gray value changes violently, or a point with a large curvature on the image edge (i.e., the intersection of two edges). After obtaining the position coordinate frame of the RGB image, candidate feature points can be selected from within the position coordinate frame of the RGB image, that is, the candidate feature points are within the position coordinate frame of the RGB image. For example, the Fast corner point method is used to select candidate feature points from within the position coordinate frame of the RGB image. Of course, the Fast corner point method is only an example, and there is no limitation on the selection method of this feature point, as long as the candidate feature points can be selected from within the position coordinate frame of the RGB image.

[0063] Among them, when selecting candidate feature points from within the position coordinate frame of the RGB image, multiple candidate feature points can be selected from within this position coordinate frame, and the number of such candidate feature points is not restricted.

[0064] Exemplarily, for each candidate feature point in the RGB image, a target feature point corresponding to this candidate feature point can be selected from another RGB image in front of this RGB image (for example, the previous frame RGB image of this RGB image), that is, the feature points in two frames of RGB images are matched to find the matching relationship between the candidate feature point and the target feature point, denoted as the matching relationship between Pi and Sj, indicating that the i-th feature point in RGB image P matches the j-th feature point in RGB image S.

[0065] When matching the feature points in two frames of RGB images, the similarity between the descriptors of the feature points can be calculated, and based on the similarity between the descriptors of the feature points, it is determined whether two feature points have a matching relationship. For example, when the similarity between the descriptors of two feature points is greater than a preset threshold, it is determined that there is a matching relationship between these two feature points, otherwise it is determined that there is no matching relationship between these two feature points. In this case, the similarity between the descriptor of the candidate feature point and the descriptor of the target feature point is greater than the preset threshold. Another example is to calculate the similarity between the descriptor of the candidate feature point and each feature point in another RGB image, and take the feature point corresponding to the maximum similarity as the target feature point corresponding to this candidate feature point.

[0066] In the above process, the descriptor can be, such as, a Brief descriptor, a sift descriptor, etc., and the type of this descriptor is not restricted. The descriptor is a method for extracting features from image data.

[0067] In the above process, in order to be able to quickly match the feature points in two frames of RGB images, the position coordinate frame of the candidate object in another frame of RGB image can also be found based on the pose relationship between the two frames of RGB images and the position coordinate frame of the candidate object in one frame of RGB image. Then, the feature points in the two position coordinate frames are matched, so as to quickly complete the feature point matching. Among them, the pose relationship between the two frames of RGB images can be given by the positioning system of the mobile robot, and this positioning system can be laser slam, visual slam, or inertial navigation unit, and the type of this positioning system is not restricted.

[0068] See Figure 4B As shown, the position relationship between two frames of RGB images under a certain pose transformation before and after is given, Figure 4B which shows an example of the position coordinate frames of the candidate object in two frames of RGB images.

[0069] Step 304: For each candidate feature point, based on the pixel coordinates of the candidate feature point and the pixel coordinates of the target feature point, determine the three-dimensional coordinates corresponding to the candidate feature point, that is, the three-dimensional physical coordinates.

[0070] For example, based on the feature point matching result, find the candidate feature point and the target feature point corresponding to the candidate feature point. Combining the pixel coordinates (pixel positions) of the candidate feature point and the pixel coordinates of the target feature point, the three-dimensional coordinates corresponding to the candidate feature point can be obtained. For example, the three-dimensional coordinates corresponding to the candidate feature point are obtained by using the triangulation method. See Figure 4C As shown, it is an example of obtaining three-dimensional coordinates by using the triangulation method. p1 represents the pixel coordinates of the target feature point, p2 represents the pixel coordinates of the candidate feature point, O1 represents the position of the image sensor at the previous frame of RGB image, O2 represents the position of the image sensor at the current frame of RGB image, t represents the time interval from the previous frame of RGB image to the current frame of RGB image, and P represents the three-dimensional coordinates.

[0071] Based on the pixel coordinates of the candidate feature point and the pixel coordinates of the target feature point, the specific implementation manner of obtaining the three-dimensional coordinates by using the triangulation method is not limited in this embodiment, as long as the three-dimensional coordinates can be obtained. The following combines a specific example to illustrate the determination process of the three-dimensional coordinates.

[0072] According to the camera imaging principle, it is known that K * x * 1 / Z = p, and x = [X Y Z], where p is the pixel coordinate, that is, p1 is the pixel coordinate of the target feature point, p2 is the pixel coordinate of the candidate feature point, and x is the three-dimensional coordinate corresponding to the candidate feature point in the world coordinate system, that is, Figure 4C the three-dimensional coordinates of point P in, and K is the camera internal parameter.

[0073] For the successfully matched candidate feature point and target feature point, the three-dimensional coordinate corresponding to the candidate feature point in the world coordinate system should be the same as the three-dimensional coordinate corresponding to the target feature point in the world coordinate system. On this basis, based on the above relationship, the following formulas (1) and (2) can be satisfied:

[0074]

[0075]

[0076] In formulas (1) and (2), represents the rotation matrix of the image sensor at the previous frame of RGB image, represents the translation matrix of the image sensor at the previous frame of RGB image, represents the rotation matrix of the image sensor at the current frame of RGB image, represents the translation matrix of the image sensor at the current frame of RGB image, while and is the pose corresponding to the previous frame RGB image, and is the pose corresponding to the current frame RGB image. Obviously, and and and are all known quantities. In addition, K is the camera internal parameter, p1 is the pixel coordinate of the target feature point, and p2 is the pixel coordinate of the candidate feature point. These parameters are also known quantities. It can be seen from Formula (1) and Formula (2) that the variables to be solved are Z1 and Z2, and the rest are all known quantities.

[0077] Assume x' = K -1 p. Then, based on Formula (1) and Formula (2), the equation can be changed to Formula (3):

[0078]

[0079] Utilizing the particularity of the cross product, where x represents the cross product, the equation can be changed to Formula (4):

[0080]

[0081] From Formula (4), Z2 can be obtained, and then substituting it into Formula (3), Z1 can be obtained.

[0082] Substituting Z1 into K * x * 1 / Z = p, X1 and Y1 can be obtained, and then x1 = [X1 Y1 Z1] can be obtained. And x1 represents the three-dimensional coordinates corresponding to the target feature point. Similarly, substituting Z2 into K * x * 1 / Z = p, X2 and Y2 can be obtained, and then x2 = [X2 Y2 Z2] can be obtained. And x2 represents the three-dimensional coordinates corresponding to the candidate feature point.

[0083] In summary, the three-dimensional coordinates corresponding to all candidate feature points in the world coordinate system can be obtained.

[0084] Step 305: Determine the suspicious area corresponding to the candidate object based on the three-dimensional coordinates.

[0085] Exemplarily, the position coordinate frame of the RGB image may include multiple candidate feature points. Each candidate feature point corresponds to a three-dimensional coordinate, and the three-dimensional coordinate may include a horizontal axis coordinate, a vertical axis coordinate, and a vertical axis coordinate. On this basis, the minimum horizontal axis coordinate and the maximum horizontal axis coordinate may be selected from the horizontal axis coordinates corresponding to all candidate feature points, the minimum vertical axis coordinate and the maximum vertical axis coordinate may be selected from the vertical axis coordinates corresponding to all candidate feature points, and the minimum vertical axis coordinate and the maximum vertical axis coordinate may be selected from the vertical axis coordinates corresponding to all candidate feature points. Based on the minimum horizontal axis coordinate, the maximum horizontal axis coordinate, the minimum vertical axis coordinate, the maximum vertical axis coordinate, the minimum vertical axis coordinate, and the maximum vertical axis coordinate, the suspicious area corresponding to the candidate object can be determined.

[0086] See Figure 5 As shown, it is a view of the XOY plane, showing the mobile robot and the candidate feature points. The arrow indicates the direction, and the black box represents the suspicious area corresponding to the candidate object. Since the candidate feature points extracted from the RGB image are sparse, the suspicious area corresponding to the candidate object should be slightly larger than the distribution of the candidate feature points.

[0087] Among them, the minimum horizontal axis coordinate is denoted as x min , the maximum horizontal axis coordinate is denoted as x max , the minimum vertical axis coordinate is denoted as y min , the maximum vertical axis coordinate is denoted as y max , then the four vertices of the suspicious area corresponding to the candidate object are B1, B2, B3, and B4. Then, the coordinates of B1, B2, B3, and B4 can be as follows:

[0088]

[0089] In addition, the minimum vertical axis coordinate is denoted as z min , the maximum vertical axis coordinate is denoted as z max , then, on the Z axis, there are also upper and lower limits of the obstacle height, that is, H max = z max +Δz and H min = z min -Δz.

[0090] Exemplarily, Δx, Δy, and Δz are all empirical values and are related to the types of recognized objects. There is no limitation on this. For example, for a line, Δx takes 0.05 - 0.1 m, Δy takes 0.02 - 0.05 m, and Δz takes 0.02 - 0.05 m.

[0091] Step 306: Determine the second feature information of the candidate object based on the suspicious area, the object type, and the object color, that is, the second feature information includes the suspicious area, the object type, and the object color.

[0092] Exemplarily, the suspicious area can be projected onto the map, that is, the suspicious area is marked on the map to indicate the suspicious position of the obstacle, and the object type and the object color are marked for the suspicious area.

[0093] In summary, the analysis process based on the image data is completed, and the second feature information of the candidate object is obtained.

[0094] Second, for the analysis process based on the laser data, refer to Figure 6 As shown, this process may include:

[0095] Step 601: Collect the laser data of the target scene through the laser sensor, and determine the two-dimensional coordinates and the number of echo signals of the target object based on the laser data. Exemplarily, the laser sensor can emit line laser and receive the echo signals of the line laser. The laser sensor collects the laser data of the target scene based on the echo signals. The laser data may include, but is not limited to, the two-dimensional coordinates and the number of echo signals of the target object in the target scene. The number of echo signals represents the number of echo signals received by the laser sensor. In this way, the two-dimensional coordinates and the number of echo signals of the target object can be determined based on the laser data.

[0096] Step 602: Determine the three-dimensional coordinates of the target object based on the two-dimensional coordinates, the pose of the laser sensor, and the pose of the mobile robot, that is, the three-dimensional physical coordinates corresponding to the target object in the world coordinate system.

[0097] Exemplarily, for the line laser, the laser sensor detects points on a single plane, and the coordinates are only (x, y). However, through the design of the placement position, according to the external parameters of the laser sensor, the pose of the mobile robot, and the pose of the laser sensor, the three-dimensional coordinates of the target object can be obtained. Refer to the formula (5) as follows:

[0098]

[0099] In formula (5), represents the pose of the mobile robot, T l represents the pose of the laser sensor, (x l , y l ) represents the two-dimensional coordinates of the target object, (x w , y w , z w ) represents the three-dimensional coordinates of the target object.

[0100] Step 603: Determine the initial height of the target object based on the three-dimensional coordinates.

[0101] Exemplarily, for the three-dimensional coordinates (x w , y w , z w ) of the target object, zw It represents the height of the target object. For the convenience of distinction, the height of the target object is called the initial height of the target object.

[0102] Step 604: Determine the first feature information of the target object based on the number of echo signals, the three-dimensional coordinates, and the initial height, that is, the first feature information includes the number of echo signals, the three-dimensional coordinates, and the initial height.

[0103] In summary, the analysis process based on the laser data is completed, and the first feature information of the target object is obtained.

[0104] Third, the obstacle fusion detection process. For the target object with the initial height z w equal to 0, it indicates that the target object is the ground and not an obstacle. For the target object with the initial height z w greater than 0, it indicates that the target object is an obstacle on the ground. However, due to the inaccurate estimation of the pose of the mobile robot and the line laser only having two-dimensional information, during the movement of the mobile robot, there are situations such as bumps and jitters, which will affect the three-dimensional coordinates of the finally detected obstacles. In addition, due to the characteristics of the sensor data of the line laser, there are also situations such as inaccurate ranging on the marble floor, which will also cause errors in the calculation of the three-dimensional coordinates of the obstacles. In order to avoid false alarms for low obstacles, in this embodiment, it is necessary to combine the laser data and the image data. The obstacle fusion detection process can be referred to Figure 7 as shown, and this process may include:

[0105] Step 701: Determine whether the initial height of the target object is not less than the second preset threshold.

[0106] If so, step 702 can be executed; if not, step 703 can be executed.

[0107] Exemplarily, the first feature information of the target object may include the initial height of the target object, and it can be determined whether the initial height is not less than the second preset threshold. The second preset threshold can be configured according to experience. If the initial height is not less than the second preset threshold, it indicates that the target object is a high obstacle; if the initial height is less than the second preset threshold, it indicates that the target object is not a high obstacle.

[0108] Step 702: If the initial height is not less than the second preset threshold, determine that the target object is an obstacle.

[0109] In a possible implementation manner, after determining that the target object is an obstacle, obstacle avoidance operations can also be performed on the obstacle (i.e., the target object). In this embodiment, the manner of this obstacle avoidance operation is not limited.

[0110] Step 703: Determine whether the initial height of the target object is greater than the first preset threshold.

[0111] If it is, step 704 can be executed; if not, step 707 can be executed.

[0112] Exemplarily, it can be determined whether the initial height is greater than the first preset threshold, and the first preset threshold can be configured according to experience. If the initial height is greater than the first preset threshold, that is, greater than the first preset threshold and less than the second preset threshold, it indicates that the target object is a low obstacle. If the initial height is not greater than the first preset threshold, it indicates that the target object is not a low obstacle and may be the ground.

[0113] In the above embodiment, the first preset threshold can be less than the second preset threshold.

[0114] Step 704: If the initial height is greater than the first preset threshold and less than the second preset threshold, determine whether the three-dimensional coordinates of the target object are within the suspicious area, that is, the suspicious area of the candidate object.

[0115] If not, step 705 can be executed; if so, step 706 can be executed.

[0116] Exemplarily, the first feature information of the target object includes the three-dimensional coordinates of the target object, and the second feature information of the candidate object includes the suspicious area of the candidate object. Therefore, it can be determined whether the three-dimensional coordinates are within the suspicious area. For example, if the suspicious area is a two-dimensional suspicious area, when the abscissa value x of the three-dimensional coordinates w is between the minimum abscissa value and the maximum abscissa value of the suspicious area, and the ordinate value y of the three-dimensional coordinates w is between the minimum ordinate value and the maximum ordinate value of the suspicious area, it is determined that the three-dimensional coordinates are within the suspicious area; otherwise, it is determined that the three-dimensional coordinates are not within the suspicious area. If the suspicious area is a three-dimensional suspicious area, when the abscissa value x of the three-dimensional coordinates w is between the minimum abscissa value and the maximum abscissa value of the suspicious area, and the ordinate value y of the three-dimensional coordinates w is between the minimum ordinate value and the maximum ordinate value of the suspicious area, and the vertical coordinate value z of the three-dimensional coordinates w is between the minimum vertical coordinate value and the maximum vertical coordinate value of the suspicious area, it is determined that the three-dimensional coordinates are within the suspicious area; otherwise, it is determined that the three-dimensional coordinates are not within the suspicious area.

[0117] Step 705: Determine that the target object is not an obstacle, that is, it is detected that it is not an obstacle.

[0118] Step 706: Determine the target height corresponding to the object type of the candidate object, and determine whether the initial height matches the target height. If so, execute step 702; if not, execute step 705.

[0119] Exemplarily, for each type of obstacle, a mapping relationship between the object type of the obstacle and the target height of the obstacle (i.e., the actual height of the obstacle) can be pre-configured. Based on this, for step 706, if the three-dimensional coordinates of the target object are within the suspicious area of the candidate object, since the second feature information of the candidate object can also include the object type of the candidate object, the mapping relationship can be queried through the object type, so as to obtain the target height corresponding to the object type.

[0120] Then, if the initial height matches the target height, such as the absolute value of the difference between the initial height and the target height is less than a preset height threshold, it can be determined that the target object is an obstacle.

[0121] If the initial height does not match the target height, such as the absolute value of the difference between the initial height and the target height is not less than the preset height threshold, it can be determined that the target object is not an obstacle.

[0122] Step 707: If the initial height is not greater than the first preset threshold, determine whether the area where the target object is located is a void area based on the number of echo signals corresponding to the target object.

[0123] If not, step 705 can be executed; if so, step 708 can be executed.

[0124] Exemplarily, for black objects, the characteristic of the laser sensor is that there will be a void phenomenon during detection. That is to say, the laser emitted by the laser sensor is absorbed by the black absorption line, resulting in the laser sensor being unable to receive echo signals, that is, there is a void area. Therefore, when the laser sensor detects that the area where the target object is located is a void area, it can be further analyzed in combination with the suspicious area of the image sensor.

[0125] Exemplarily, the first feature information of the target object can include the number of echo signals of the target object, and it can be determined whether the area where the target object is located is a void area based on the number of echo signals. For example, if the number of echo signals is greater than the first threshold, it can be determined that the area where the target object is located is not a void area; if the number of echo signals is not greater than the first threshold, it can be determined that the area where the target object is located is a void area. Another example is that if the ratio of the number of echo signals to the number of laser emissions is greater than the second threshold, it can be determined that the area where the target object is located is not a void area; if the ratio of the number of echo signals to the number of laser emissions is not greater than the second threshold, it can be determined that the area where the target object is located is a void area.

[0126] Exemplarily, if the area where the target object is located is not a void area, when the initial height is not greater than the first preset threshold, it can be directly determined that the target object is not an obstacle, and this process will not be elaborated here.

[0127] Exemplarily, if the area where the target object is located is a void area, when the initial height is not greater than the first preset threshold, the target object may or may not be an obstacle, and step 708 is executed.

[0128] Step 708: Determine whether the three-dimensional coordinates of the target object are within the suspicious area. Exemplarily, the first feature information of the target object includes the three-dimensional coordinates of the target object, and the second feature information of the candidate object includes the suspicious area of the candidate object. Therefore, it can be determined whether the three-dimensional coordinates are within the suspicious area.

[0129] If not, step 705 can be executed; if so, step 709 can be executed.

[0130] Step 709: Determine the object color of the candidate object and determine whether the object color is black.

[0131] If so, step 702 can be executed; if not, step 705 can be executed.

[0132] Exemplarily, if the object color of the candidate object is black and the line laser detects a void (i.e., the area where the target object is located is a void area), it indicates that the target object is a black obstacle, that is, it is determined that the target object is an obstacle. If the object color of the candidate object is not black, even if the area where the target object is located is a void area, it indicates that the target object is not a black obstacle, that is, it is determined that the target object is not an obstacle.

[0133] Exemplarily, since the second feature information of the candidate object can also include the object color of the candidate object, the object color of the candidate object can be determined, and the object color is black or not black.

[0134] As can be seen from the above technical solutions, in the embodiments of the present application, the fusion of obstacle detection can be performed for the laser sensor and the image sensor, so as to better perform obstacle detection and avoidance, greatly improve the accuracy of obstacle detection, and on the premise of reducing false alarms, greatly reduce the possibility of collision between the mobile robot and the obstacle, and better realize the obstacle avoidance function. By combining the advantages of the laser sensor and the image sensor, dense and accurate obstacle depth estimation can be achieved. The detection defects of the laser data can be overcome by using the image data, such as the problems of difficult detection of low obstacles and black obstacles, etc. The defects of the image data can be overcome by using the laser data, such as the problem of identifying light and shadow as obstacles due to the influence of light, etc., so as to realize the recognition and avoidance of various three-dimensional complex obstacles. By fusing the laser data with the image data of the image sensor, dense obstacle position estimation is achieved. By fusing the image data with the laser data of the laser sensor, the problem that the line laser has poor detection effect on low obstacles and black obstacles is compensated.

[0135] Based on the same application concept as the above method, in the embodiments of the present application, an obstacle detection device is proposed, which is applied to a mobile robot including a laser sensor and an image sensor. The laser sensor is used to collect laser data of the target scene, and the image sensor is used to collect image data of the target scene. Refer to Figure 8 As shown, it is a schematic structural diagram of the obstacle detection device. The device includes: a first determination module 81, configured to determine first feature information of a target object in the target scene based on the laser data collected by the laser sensor; a second determination module 82, configured to determine second feature information of a candidate object in the target scene based on the image data collected by the image sensor; a third determination module 83, configured to determine whether the target object is an obstacle based on the first feature information and the second feature information.

[0136] Exemplarily, the first feature information includes an initial height and three-dimensional coordinates, and the second feature information includes a suspicious area and an object type. When the third determination module 83 determines whether the target object is an obstacle based on the first feature information and the second feature information, it specifically is used for: if the initial height is greater than a first preset threshold and less than a second preset threshold, and the three-dimensional coordinates are within the suspicious area, then determine a target height corresponding to the object type; if the initial height matches the target height, then determine that the target object is an obstacle; if the initial height does not match the target height, then determine that the target object is not an obstacle.

[0137] Exemplarily, the first feature information includes the number of echo signals and three-dimensional coordinates, and the second feature information includes a suspicious area and object color. When the third determination module 83 determines whether the target object is an obstacle based on the first feature information and the second feature information, it specifically is configured to: if it is determined based on the number of echo signals that the area where the target object is located is a hollow area and the three-dimensional coordinates are within the suspicious area, then determine whether the object color is black; if the object color is black, determine that the target object is an obstacle; if the object color is not black, determine that the target object is not an obstacle.

[0138] Exemplarily, the first feature information includes an initial height, and the third determination module 83 is further configured to: if the initial height is not less than a second preset threshold, then determine that the target object is an obstacle.

[0139] Exemplarily, when the first determination module 81 determines the first feature information of the target object in the target scene based on the laser data collected by the laser sensor, it specifically is configured to: determine the two-dimensional coordinates and the number of echo signals of the target object based on the laser data; determine the three-dimensional coordinates of the target object based on the two-dimensional coordinates, the pose of the laser sensor, and the pose of the mobile robot, and determine the initial height of the target object based on the three-dimensional coordinates; and determine the first feature information of the target object based on the number of echo signals, the three-dimensional coordinates, and the initial height.

[0140] Exemplarily, when the second determination module 82 determines the second feature information of the candidate object in the target scene based on the image data collected by the image sensor, it specifically is configured to: input the image data into a trained target network model to obtain the position coordinate frame of the candidate object, the object type, and the object color of the candidate object in the image data; select the candidate feature points within the position coordinate frame, and select the target feature points corresponding to the candidate feature points from the image data of the previous frame of the image data; determine the three-dimensional coordinates corresponding to the candidate feature points based on the pixel coordinates of the candidate feature points and the pixel coordinates of the target feature points; determine the suspicious area corresponding to the candidate object based on the three-dimensional coordinates; and determine the second feature information based on the suspicious area, object type, and object color.

[0141] Exemplarily, the position coordinate box includes a plurality of candidate feature points, and the three-dimensional coordinates corresponding to each candidate feature point include a horizontal axis coordinate, a vertical axis coordinate, and a vertical axis coordinate. When the second determination module 82 determines the suspicious area corresponding to the candidate object based on the three-dimensional coordinates, it is specifically configured to: select the minimum horizontal axis coordinate and the maximum horizontal axis coordinate from the horizontal axis coordinates corresponding to all candidate feature points, select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points, and select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points; Based on the minimum horizontal axis coordinate, the maximum horizontal axis coordinate, the minimum vertical axis coordinate, the maximum vertical axis coordinate, the minimum vertical axis coordinate, and the maximum vertical axis coordinate, determine the suspicious area corresponding to the candidate object.

[0142] Based on the same concept as the above method, an embodiment of the present application proposes a mobile robot. Refer to Figure 9 As shown, it is a schematic structural diagram of the mobile robot. The mobile robot may include:

[0143] A laser sensor 91 for collecting laser data of the target scene;

[0144] An image sensor 92 for collecting image data of the target scene;

[0145] A processor 93 for determining first feature information of a target object in the target scene based on the laser data; determining second feature information of a candidate object in the target scene based on the image data; and determining whether the target object is an obstacle based on the first feature information and the second feature information.

[0146] Exemplarily, the first feature information includes an initial height and three-dimensional coordinates, and the second feature information includes a suspicious area and an object type. When the processor 93 determines whether the target object is an obstacle based on the first feature information and the second feature information, it is specifically configured to: if the initial height is greater than a first preset threshold and less than a second preset threshold, and the three-dimensional coordinates are within the suspicious area, then determine a target height corresponding to the object type; if the initial height matches the target height, then determine that the target object is an obstacle; if the initial height does not match the target height, then determine that the target object is not an obstacle.

[0147] Exemplarily, the first feature information includes the number of echo signals and three-dimensional coordinates, and the second feature information includes a suspicious area and an object color. When the processor 93 determines whether the target object is an obstacle based on the first feature information and the second feature information, it is specifically configured to: if it is determined based on the number of echo signals that the area where the target object is located is a hollow area and the three-dimensional coordinates are within the suspicious area, then determine whether the object color is black; if the object color is black, determine that the target object is an obstacle; if the object color is not black, determine that the target object is not an obstacle.

[0148] Exemplarily, the first feature information includes an initial height, and the processor 93 is further configured to: if the initial height is not less than a second preset threshold, then determine that the target object is an obstacle.

[0149] Exemplarily, when the processor 93 determines the first feature information of the target object in the target scene based on the laser data, it is specifically configured to: determine the two-dimensional coordinates and the number of echo signals of the target object based on the laser data; determine the three-dimensional coordinates of the target object based on the two-dimensional coordinates, the pose of the laser sensor, and the pose of the mobile robot, and determine the initial height of the target object based on the three-dimensional coordinates; and determine the first feature information of the target object based on the number of echo signals, the three-dimensional coordinates, and the initial height.

[0150] Exemplarily, when the processor 93 determines the second feature information of the candidate object in the target scene based on the image data, it is specifically configured to: input the image data into a trained target network model to obtain the position coordinate frame of the candidate object in the image data, the object type and the object color of the candidate object; select the candidate feature points within the position coordinate frame, and select the target feature points corresponding to the candidate feature points from the image data of the previous frame of the image data; determine the three-dimensional coordinates corresponding to the candidate feature points based on the pixel coordinates of the candidate feature points and the pixel coordinates of the target feature points; determine the suspicious area corresponding to the candidate object based on the three-dimensional coordinates; and determine the second feature information based on the suspicious area, the object type, and the object color.

[0151] Exemplarily, the position coordinate box includes a plurality of candidate feature points, and the three-dimensional coordinates corresponding to each candidate feature point each include a horizontal axis coordinate, a vertical axis coordinate, and a vertical axis coordinate. When the processor 93 determines the suspicious area corresponding to the candidate object based on the three-dimensional coordinates, it is specifically configured to: select the minimum horizontal axis coordinate and the maximum horizontal axis coordinate from the horizontal axis coordinates corresponding to all candidate feature points, select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points, and select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points; Based on the minimum horizontal axis coordinate, the maximum horizontal axis coordinate, the minimum vertical axis coordinate, the maximum vertical axis coordinate, the minimum vertical axis coordinate, and the maximum vertical axis coordinate, determine the suspicious area corresponding to the candidate object.

[0152] Based on the same application concept as the above method, an embodiment of the present application provides a mobile robot, which may further include: a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; wherein, the processor is configured to execute the machine-executable instructions to implement the obstacle detection method disclosed in the above example of the present application.

[0153] Based on the same application concept as the above method, an embodiment of the present application further provides a machine-readable storage medium, on which a number of computer instructions are stored, and when the computer instructions are executed by a processor, the obstacle detection method disclosed in the above example of the present application can be implemented.

[0154] Among them, the above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, and may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drive (such as a hard disk drive), solid state drive, any type of storage disk (such as an optical disk, dvd, etc.), or a similar storage medium, or a combination thereof.

[0155] The system, device, module or unit illustrated in the above embodiments may be specifically implemented by a computer chip or entity, or by a product with a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0156] For convenience of description, when describing the above device, various units are described separately according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in one or more software and / or hardware.

[0157] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0159] Moreover, these computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0161] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. An obstacle detection method, characterized in that, Applied to a mobile robot including a laser sensor and an image sensor, where the laser sensor is used to collect laser data of a target scene, and the image sensor is used to collect image data of the target scene, the method includes: Determine first feature information of a target object in the target scene based on the laser data; the first feature information includes an initial height and three-dimensional coordinates; Determine second feature information of a candidate object in the target scene based on the image data; the second feature information includes a suspicious area and an object type; Determine whether the target object is an obstacle based on the first feature information and the second feature information; Wherein, determining whether the target object is an obstacle based on the first feature information and the second feature information includes: if the initial height is greater than a first preset threshold and less than a second preset threshold, and the three-dimensional coordinates are within the suspicious area, then determine a target height corresponding to the object type; if the initial height matches the target height, determine that the target object is an obstacle; if the initial height does not match the target height, determine that the target object is not an obstacle.

2. The method according to claim 1, wherein The first feature information further includes the number of echo signals, and the second feature information further includes the object color. Determining whether the target object is an obstacle based on the first feature information and the second feature information includes: If it is determined based on the number of echo signals that the area where the target object is located is a void area, and the three-dimensional coordinates are within the suspicious area, then determine whether the object color is black; If the object color is black, determine that the target object is an obstacle; If the object color is not black, determine that the target object is not an obstacle.

3. The method according to claim 1, wherein After determining the first feature information of the target object in the target scene based on the laser data, the method further includes: If the initial height is not less than the second preset threshold, determine that the target object is an obstacle.

4. The method according to any one of claims 1 to 3, characterized in that Determining the first feature information of the target object in the target scene based on the laser data includes: Determine the two-dimensional coordinates and the number of echo signals of the target object based on the laser data; Determine the three-dimensional coordinates of the target object based on the two-dimensional coordinates, the pose of the laser sensor, and the pose of the mobile robot, and determine the initial height of the target object based on the three-dimensional coordinates; Determine the first feature information based on the number of echo signals, three-dimensional coordinates, and initial height.

5. The method according to any one of claims 1 to 3, characterized in that Determining the second feature information of the candidate object in the target scene based on the image data includes: Input the image data into a trained target network model to obtain the position coordinate frame of the candidate object in the image data, the object type of the candidate object, and the object color; Select candidate feature points within the position coordinate frame, and select target feature points corresponding to the candidate feature points from the image data of the previous frame of the image data; determine the three-dimensional coordinates corresponding to the candidate feature points based on the pixel coordinates of the candidate feature points and the pixel coordinates of the target feature points; Determine the suspicious area corresponding to the candidate object based on the three-dimensional coordinates; Determine the second feature information based on the suspicious area, object type, and object color.

6. The method according to claim 5, characterized in that, The position coordinate frame includes a plurality of candidate feature points, and the three-dimensional coordinates corresponding to each candidate feature point include a horizontal axis coordinate, a vertical axis coordinate, and a vertical axis coordinate. The determining of the suspicious area corresponding to the candidate object based on the three-dimensional coordinates includes: Select the minimum horizontal axis coordinate and the maximum horizontal axis coordinate from the horizontal axis coordinates corresponding to all candidate feature points, select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points, and select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points; Based on the minimum horizontal axis coordinate, the maximum horizontal axis coordinate, the minimum vertical axis coordinate, the maximum vertical axis coordinate, the minimum vertical axis coordinate, and the maximum vertical axis coordinate, determine the suspicious area corresponding to the candidate object.

7. A mobile robot, characterized in that, Includes: A laser sensor for collecting laser data of the target scene; An image sensor for collecting image data of the target scene; A processor for determining first feature information of a target object in the target scene based on the laser data, the first feature information including an initial height and three-dimensional coordinates; determining second feature information of a candidate object in the target scene based on the image data, the second feature information including a suspicious area and an object type; Determine whether the target object is an obstacle based on the first feature information and the second feature information. When determining whether the target object is an obstacle based on the first feature information and the second feature information, it is specifically used for: if the initial height is greater than a first preset threshold and less than a second preset threshold, and the three-dimensional coordinates are within the suspicious area, then determine the target height corresponding to the object type; if the initial height matches the target height, then determine that the target object is an obstacle; if the initial height does not match the target height, then determine that the target object is not an obstacle.

8. The mobile robot according to claim 7, characterized in that, Wherein, The first feature information further includes the number of echo signals, and the second feature information further includes the object color. When the processor determines whether the target object is an obstacle based on the first feature information and the second feature information, it is specifically used for: if it is determined that the area where the target object is located is a void area based on the number of echo signals, and the three-dimensional coordinates are within the suspicious area, then determine whether the object color is black; If the object color is black, determine that the target object is an obstacle; If the object color is not black, determine that the target object is not an obstacle; Wherein, the processor is further used for: if the initial height is not less than the second preset threshold, then determine that the target object is an obstacle; Wherein, when determining the first feature information of the target object in the target scene based on the laser data, the processor is specifically configured to: determine the two-dimensional coordinates and the number of echo signals of the target object based on the laser data; determine the three-dimensional coordinates of the target object based on the two-dimensional coordinates, the pose of the laser sensor, and the pose of the mobile robot, and determine the initial height of the target object based on the three-dimensional coordinates; and determine the first feature information of the target object based on the number of echo signals, the three-dimensional coordinates, and the initial height; Wherein, when determining the second feature information of the candidate object in the target scene based on the image data, the processor is specifically configured to: input the image data into a trained target network model to obtain the position coordinate frame of the candidate object in the image data, the object type and the object color of the candidate object; select the candidate feature points within the position coordinate frame, and select the target feature points corresponding to the candidate feature points from the image data of the previous frame of the image data; determine the three-dimensional coordinates corresponding to the candidate feature points based on the pixel coordinates of the candidate feature points and the pixel coordinates of the target feature points; determine the suspicious area corresponding to the candidate object based on the three-dimensional coordinates; and determine the second feature information based on the suspicious area, the object type, and the object color; Wherein, the position coordinate frame includes a plurality of candidate feature points, and the three-dimensional coordinates corresponding to each candidate feature point include a horizontal axis coordinate, a vertical axis coordinate, and a vertical axis coordinate. When determining the suspicious area corresponding to the candidate object based on the three-dimensional coordinates, the processor is specifically configured to: select the minimum horizontal axis coordinate and the maximum horizontal axis coordinate from the horizontal axis coordinates corresponding to all candidate feature points, select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points, and select the minimum vertical axis coordinate and the maximum vertical axis coordinate from the vertical axis coordinates corresponding to all candidate feature points; determine the suspicious area corresponding to the candidate object based on the minimum horizontal axis coordinate, the maximum horizontal axis coordinate, the minimum vertical axis coordinate, the maximum vertical axis coordinate, the minimum vertical axis coordinate, and the maximum vertical axis coordinate.

9. A machine-readable storage medium, characterized in that Several computer instructions are stored on the machine-readable storage medium, and when the computer instructions are executed by a processor, the method steps described in any one of claims 1-6 are implemented.

Images