Stair detection method, device and mobile robot

By acquiring point cloud data and segmenting and classification in the ground coordinate system, the mobile robot can accurately detect the position of the stair structure, solving the problem of not being able to identify stairs in the prior art, and achieving safe movement between different floors.

CN112529963BActive Publication Date: 2024-12-06SHENZHEN UNITY-DRIVE INNOVATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011461928.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-11
Publication Date
2024-12-06
Estimated Expiration
2040-12-11

AI Technical Summary

Technical Problem

Existing mobile robots cannot accurately identify and detect stair structures, resulting in the inability to move safely when shuttled between different floors.

Method used

By obtaining point cloud data, determining the ground coordinate system, and performing plane segmentation and classification in the coordinate system, filtering out the water platform step surface, and establishing and outputting the three-dimensional model of the staircase and its spatial pose information.

Benefits of technology

The mobile robot can accurately detect the position of the stair structure and establish a three-dimensional model to ensure safe movement in the stair environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112529963B_ABST
    Figure CN112529963B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention relates to the field of environmental perception technology, and discloses a stair detection method and device. The method first obtains point cloud data around the current scene where a mobile robot is located, and then determines the ground in the current scene based on the point cloud data, and establishes a ground coordinate system, and then performs plane segmentation on the point cloud data to determine the plane in the current scene, and then classifies the plane in the ground coordinate system, and screens out the horizontal step surface in the plane, and finally establishes and outputs a visualized three-dimensional model of the stair structure and its spatial posture information based on the horizontal step surface. The stair detection method provided by the embodiment of the present invention enables the mobile robot to accurately detect the posture of the stair structure in the current scene and establish a three-dimensional model of the stair structure.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of environmental perception technology, and in particular to a stair detection method, device and mobile robot. Background Art

[0002] Stairs are a very common and important building structure in indoor scenes. They are erected between the two floors of a building and are used for people to move up and down between floors. They are shaped like steps. In order to assist or replace humans in their work, various indoor mobile robots have appeared on the market. They are usually small in size and can move freely indoors. They concentrate the research results of multiple disciplines such as sensor technology, information processing, electronic engineering, computer engineering, automation control engineering, and artificial intelligence. They represent the highest achievement of mechatronics and are one of the most active areas of scientific and technological development. At present, in order to enable indoor mobile robots to move autonomously indoors, detection devices are usually installed on the mobile robots to detect indoor environmental information. The mobile robots can autonomously plan routes and perform moving tasks based on the detected environmental information.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems in the above-mentioned related technologies: for indoor mobile robots, if they want to shuttle between different floors, identifying stairs becomes an indispensable function, but the mobile robots currently on the market do not have the function of detecting stairs, and only regard the stairs as non-walkable areas, and cannot ensure that the mobile robots move on the stairs. Summary of the invention

[0004] In view of the above-mentioned defects in the prior art, an object of the embodiments of the present invention is to provide a stair detection method, device and mobile robot which can easily and accurately obtain a three-dimensional model of the stairs.

[0005] The purpose of the embodiment of the present invention is achieved through the following technical solutions:

[0006] In order to solve the above technical problems, in a first aspect, an embodiment of the present invention provides a staircase detection method, which is applied to a mobile robot, and the method includes:

[0007] Acquire point cloud data around the current scene where the mobile robot is located;

[0008] Determine the ground in the current scene according to the point cloud data, and establish a ground coordinate system;

[0009] Performing plane segmentation on the point cloud data to determine a plane in the current scene;

[0010] In the ground coordinate system, classifying the planes, and screening out horizontal step surfaces in the planes;

[0011] According to the horizontal step surface, a visualized three-dimensional model of the stair structure and its spatial posture information are established and output.

[0012] In some embodiments, the step of performing plane segmentation on the point cloud data to determine the plane in the current scene further comprises:

[0013] Clustering the point cloud data by using a region growing algorithm;

[0014] Determining whether the point cloud data after clustering is a plane by a random sampling consistency algorithm;

[0015] If yes, jump to the step of classifying the plane;

[0016] If not, the non-planar point cloud data is classified as an obstacle, and the non-clustered point cloud data is further classified as the obstacle through the Euclidean clustering algorithm.

[0017] In some embodiments, the step of determining the ground in the current scene based on the point cloud data and establishing a ground coordinate system further includes:

[0018] Segmenting the point cloud data into planes by a random sampling consensus algorithm;

[0019] Obtaining the number of point clouds and normal directions of each of the planes;

[0020] Determine a plane whose number of point clouds is greater than a preset threshold and whose normal direction is parallel to the gravity direction as the ground in the current scene;

[0021] Constructing an ideal ground normal vector in the camera coordinate system according to the orientation and tilt angle of the depth camera;

[0022] The ground coordinate system is determined according to the ideal ground normal vector, and a first transformation matrix from the camera coordinate system to the ground coordinate system is calculated.

[0023] In some embodiments, the step of classifying the planes in the ground coordinate system and screening out the horizontal step surfaces in the planes further comprises:

[0024] According to the first transformation matrix, the point cloud data of each of the planes is converted into a representation in a ground coordinate system, and the plane equation of each of the planes in the ground coordinate system is obtained;

[0025] According to the plane equations of each of the planes represented by the ground coordinate system, the horizontal step surfaces in the planes are screened out.

[0026] In some embodiments, the step of establishing and outputting a visualized three-dimensional model of the stair structure and its spatial posture information according to the horizontal step surface further comprises:

[0027] Obtaining the centroid coordinates of each of the horizontal step surfaces in a ground coordinate system;

[0028] According to the centroid coordinates of the horizontal step surfaces, determining the height of each horizontal step surface relative to the ground and sorting the horizontal step surfaces;

[0029] Constructing a two-dimensional circumscribed rectangle of each of the horizontal step surfaces;

[0030] Calculate the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles;

[0031] Obtaining the convex hull area of ​​each of the horizontal step surfaces and the area of ​​its two-dimensional circumscribed rectangle, and calculating the ratio of the convex hull area to the area of ​​its two-dimensional circumscribed rectangle;

[0032] Determining the optimal plane in the horizontal step surface according to the ratio;

[0033] According to the sorted horizontal step surfaces, the length, width of each two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles, a three-dimensional model of the stairs in the stair coordinate system is constructed and visually output, and the spatial posture information of the stairs is determined, wherein the centroid coordinates of the optimal plane in the stair coordinate system are used as the overall center of the three-dimensional model of the entire stair, and the three eigenvectors of the optimal plane are used as the main directions of the three-dimensional model of the entire stair.

[0034] In some embodiments, the step of calculating the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles further includes:

[0035] Determine the covariance matrix of each of the horizontal step surfaces in the camera coordinate system, wherein three eigenvectors of the covariance matrix form a second transformation matrix in the form of columns;

[0036] According to the second transformation matrix, the point cloud of each of the horizontal step surfaces in the camera coordinate system is transformed into a representation in the stair coordinate system;

[0037] Calculate the difference between the maximum and minimum values ​​of the point coordinates of each point cloud in the two coordinate axis directions of the staircase coordinate system as the length and width of the two-dimensional circumscribed rectangle;

[0038] The height difference between two adjacent two-dimensional circumscribed rectangles is obtained according to the difference in Y-axis coordinates of the centroids of the horizontal step surface.

[0039] In order to solve the above technical problems, in a second aspect, an embodiment of the present invention provides a stair detection device, which is applied to a mobile robot, and the device includes:

[0040] A data acquisition module, used to obtain point cloud data around the current scene where the mobile robot is located;

[0041] A ground detection module, used to determine the ground in the current scene according to the point cloud data, and establish a ground coordinate system;

[0042] A scene segmentation module, used for performing plane segmentation on the point cloud data to determine a plane in the current scene;

[0043] A stair detection module, used to classify the plane in the ground coordinate system and filter out horizontal step surfaces in the plane;

[0044] The stair modeling module is used to establish and output a visualized three-dimensional model of the stair structure and its spatial posture information based on the horizontal step surface.

[0045] In some embodiments, the scene segmentation module is also used to cluster the point cloud data through a region growing algorithm; determine whether the clustered point cloud data is a plane through a random sampling consistency algorithm; if so, jump to the step of classifying the plane; if not, classify the non-planar point cloud data as obstacles, and further classify the non-clustered point cloud data as obstacles through a Euclidean clustering algorithm.

[0046] In some embodiments, the ground detection module is also used to divide the point cloud data into planes through a random sampling consistency algorithm; obtain the number of point clouds and the normal direction of each plane; determine the plane whose point cloud number is greater than a preset threshold and whose normal direction is parallel to the gravity direction as the ground in the current scene; construct an ideal ground normal vector in the camera coordinate system according to the orientation and tilt angle of the depth camera; determine the ground coordinate system according to the ideal ground normal vector, and calculate the first transformation matrix from the camera coordinate system to the ground coordinate system.

[0047] In some embodiments, the stair detection module is also used to convert the point cloud data of each of the planes into a representation using a ground coordinate system according to the first transformation matrix, and obtain the plane equations of each of the planes in the ground coordinate system; and filter out the horizontal step surfaces in the plane according to the plane equations of each of the planes represented by the ground coordinate system.

[0048] In some embodiments, the stair modeling module is also used to obtain the centroid coordinates of each of the horizontal step surfaces in the ground coordinate system; determine the height of each of the horizontal step surfaces relative to the ground and sort the horizontal step surfaces based on the centroid coordinates of each of the horizontal step surfaces; construct a two-dimensional circumscribed rectangle of each of the horizontal step surfaces; calculate the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles; obtain the convex hull area of ​​each of the horizontal step surfaces and the area of ​​its two-dimensional circumscribed rectangle, and calculate the ratio of the convex hull area to the area of ​​its two-dimensional circumscribed rectangle; determine the best plane in the horizontal step surface based on the ratio; construct and visualize the output of the three-dimensional model of the stairs in the stair coordinate system based on the sorted horizontal step surfaces, as well as the length, width of each of the two-dimensional circumscribed rectangles and the height difference between two adjacent two-dimensional circumscribed rectangles, and determine the spatial posture information of the stairs, wherein the centroid coordinates of the best plane in the stair coordinate system are used as the overall center of the three-dimensional model of the entire stair, and the three eigenvectors of the best plane are used as the main directions of the three-dimensional model of the entire stair.

[0049] In some embodiments, the stair modeling module is also used to determine the covariance matrix of each of the horizontal step surfaces in the camera coordinate system, and the three eigenvectors of the covariance matrix form a second transformation matrix in the form of columns; according to the second transformation matrix, the point cloud of each of the horizontal step surfaces in the camera coordinate system is converted to a representation in the stair coordinate system; the difference between the maximum and minimum values ​​of the point coordinates of each point cloud in the two coordinate axis directions of the stair coordinate system is calculated as the length and width of the two-dimensional circumscribed rectangle; the height difference between two adjacent two-dimensional circumscribed rectangles is obtained according to the difference in the Y-axis coordinates of the center of mass of the horizontal step surfaces.

[0050] In order to solve the above technical problems, in a third aspect, an embodiment of the present invention provides a mobile robot, comprising:

[0051] a depth camera mounted on the mobile robot;

[0052] at least one processor coupled to the depth camera; and,

[0053] a memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0055] To solve the above technical problems, in a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method described in the first aspect above.

[0056] To solve the above technical problems, in the fifth aspect, an embodiment of the present invention further provides a computer program product, wherein the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method described in the first aspect above.

[0057] Compared with the prior art, the beneficial effects of the present invention are: different from the prior art, the embodiment of the present invention provides a stair detection method and device, which first obtains point cloud data around the current scene where the mobile robot is located, and then determines the ground in the current scene based on the point cloud data, and establishes a ground coordinate system, and then performs plane segmentation on the point cloud data to determine the plane in the current scene, and then classifies the plane in the ground coordinate system, and filters out the horizontal step surface in the plane, and finally establishes and outputs a visualized three-dimensional model of the stair structure and its spatial posture information based on the horizontal step surface. The stair detection method provided by the embodiment of the present invention enables the mobile robot to accurately detect the posture of the stair structure in the current scene and establish a three-dimensional model of the stair structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise specified, the figures in the drawings do not constitute proportional limitations.

[0059] Figure 1 is a schematic diagram of one application scenario of the stair detection method provided by an embodiment of the present invention;

[0060] Figure 2 is a flow chart of a stair detection method provided by an embodiment of the present invention;

[0061] Figure 3 yes Figure 2 A sub-flow chart of step 130 in the method shown;

[0062] Figure 4 yes Figure 2 A sub-flow chart of step 120 in the method shown;

[0063] Figure 5 yes Figure 2A sub-flow chart of step 140 in the method shown;

[0064] Figure 6 yes Figure 2 A sub-flowchart of step 150 in the method shown;

[0065] Figure 7 yes Figure 6 A sub-flow chart of step 154 ​​in the method shown;

[0066] Figure 8 is a structural schematic diagram of a stair detection device provided by an embodiment of the present invention;

[0067] Fig. 9 It is a schematic diagram of the hardware structure of a mobile robot provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the flow chart. In addition, the words "first", "second", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0071] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0072] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0073] In order to solve the problem that the current mobile robots used indoors cannot accurately determine the position of stairs and the structure of stairs, the embodiment of the present invention provides a stair detection method that can be applied to mobile robots. Figure 1 , which shows one of the application environments of the stair detection method provided by an embodiment of the present invention, wherein the application environment includes: a mobile robot 10, on which a depth camera 11 is installed. In an embodiment of the present invention, the mobile robot 10 can make use of the depth information perception capability of the depth camera 11. The mobile robot 10 interacts with the depth camera 11 through a robot operating system (ROS) set up inside the mobile robot 10 to obtain point cloud data collected by the depth camera. Then, the mobile robot 10 performs a series of processing on the obtained point cloud data to determine whether there is a stair step surface in the current scene. After determining that there is a stair step surface, the number, length, width, height and position information of the steps are determined to finally determine the spatial posture information of the entire stair structure and establish a three-dimensional model of the stair structure.

[0074] Furthermore, in the embodiment of the present invention, the program run by the robot operating system set up by the mobile robot 10 can be capable of real-time processing of point cloud data or processing of offline point cloud data files. If real-time processing is to be adopted, the depth camera 11 needs to adopt a depth camera with a sampling frequency that can be maintained at 30Hz to avoid affecting the processing performance of the program. Preferably, the depth camera 11 is installed at a height of at least one meter from the ground to ensure the stability of the depth camera 11.

[0075] Specifically, the embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0076] Embodiment 1

[0077] The present invention provides a method for detecting stairs. Figure 2 , which shows a flow chart of a stair detection method provided by an embodiment of the present invention, the stair detection method can be applied in the above application scenarios and Figure 1 In the mobile robot, the method includes but is not limited to the following steps:

[0078] Step 110: Acquire point cloud data around the current scene where the mobile robot is located;

[0079] In an embodiment of the present invention, a depth camera as described in the above application scenario may be used to collect the point cloud data. Preferably, the depth camera adopts an RGB-D camera, which has the ability to perceive depth information and color information, and can provide fast and stable geometric reconstruction capabilities on a single frame of data. In addition, the method of obtaining the point cloud data around the current scene where the mobile robot is located may be to convert the depth image collected by the RGB-D camera into point cloud data. Furthermore, it may also be point cloud data directly scanned by a laser, a lidar, etc. Specifically, it may be set according to actual needs without being restricted to the limitations of the embodiment of the present invention.

[0080] Step 120: determining the ground in the current scene according to the point cloud data, and establishing a ground coordinate system;

[0081] After acquiring the point cloud data, after clustering analysis of the point cloud data, the ground in the current scene where the mobile robot is located can be determined first, so as to further calculate the ground coordinate system, so as to use the ground as a reference to determine whether there are stairs in the scene, and when there are stairs, to determine the location of the stairs. Preferably, the depth perception capability of the depth camera can be used, and the mobile robot can also automatically find the ground and establish a ground coordinate system.

[0082] Step 130: performing plane segmentation on the point cloud data to determine a plane in the current scene;

[0083] After acquiring the point cloud data, further, plane segmentation is performed on the point cloud of a single frame to determine the plane in the current scene. By further performing feature analysis on the point cloud data of the plane, it can be determined whether the plane is the plane of the stair structure.

[0084] Step 140: classifying the plane in the ground coordinate system, and screening out the horizontal step surface in the plane;

[0085] Specifically, the point cloud data is mainly classified into planes such as walls, ground, and stair steps, and the point cloud data that cannot form a plane is classified as an obstacle. Among them, the plane determined to be a stair step can be used to determine the spatial position of the stair structure, the number of stair steps, the size of the stair steps, etc. through further analysis.

[0086] Step 150: Based on the horizontal step surface, establish and output a visualized three-dimensional model of the stair structure and its spatial posture information.

[0087] After determining the spatial position of the staircase structure, the number of stair steps, the size of the stair steps and other information, a three-dimensional model of the staircase structure can be established based on this information, and further, a visualized three-dimensional model and its spatial position information can be output. The visualized output refers to displaying the three-dimensional model on a display screen, and visually displaying the spatial position information of the staircase structure through text, numerical values, direction axes, pointers and the like.

[0088] In some embodiments, see Figure 3 , which shows Figure 2 A sub-process of step 130 in the method shown, wherein step 130 includes but is not limited to the following steps:

[0089] Step 131: clustering the point cloud data by using a region growing algorithm;

[0090] In an embodiment of the present invention, each region in the entire point cloud data set is segmented by a region growing algorithm. The curvature and normal angle of the plane in a very small area around each point cloud are used as two thresholds of the region growing algorithm. The curvature is used to determine whether the domain point of the point cloud (i.e., several point clouds around the point cloud that are closer to the point cloud) should enter the seed point queue, and the normal angle is used to determine whether the domain point should enter the area where the current seed point is located. After traversing each point cloud, the point cloud data in the seed point queue finally obtained is the point cloud data clustered in the same area.

[0091] Furthermore, the originally collected point cloud data may be preprocessed. Specifically, 3D voxel grid filtering may be performed to reduce the number of points and provide a certain smoothing effect.

[0092] Furthermore, since the RGB-D camera used in this application can also collect color information of the point cloud, the color information of each point cloud can also be used as an auxiliary. Usually, the point clouds belonging to the stairs in the room have similar color information. Therefore, a certain weight can be assigned to the color information of each point cloud to be used as a standard for clustering the point cloud data. Among them, the color information and the above-mentioned normal angle can be used together as a judgment condition for judging whether the domain point should enter the area where the current seed point is located.

[0093] Step 132: Determine whether the clustered point cloud data is a plane by using a random sampling consensus algorithm; if so, jump to step 140; if not, jump to step 133;

[0094] After clustering the preprocessed point cloud data, it is further necessary to determine whether each of the clustered point cloud data is a plane. Here, similar to step 161, the embodiment of the present invention determines whether each of the clustered point cloud data is a plane through a random sampling consensus algorithm (RANSAC fitting algorithm), wherein only when the number of point clouds in the plane within the cluster is greater than a set threshold, the point cloud data in the cluster will be determined as point cloud data in the plane. And after determining that it is a plane, jump to the step of classifying the plane.

[0095] Step 133: Classify the non-planar point cloud data as obstacles, and further classify the non-clustered point cloud data as obstacles by using the Euclidean clustering algorithm.

[0096] After determining that the cluster is not a plane through the above step 132, these non-planar point cloud data are classified into non-planar point clouds, that is, classified as obstacles. At the same time, further, the point clouds that are not clustered by the region growing algorithm are also classified as obstacles through the Euclidean clustering algorithm.

[0097] In some embodiments, see Figure 4 , which shows Figure 2 A sub-process of step 120 in the method shown, wherein step 120 includes but is not limited to the following steps:

[0098] Step 121: Segment the point cloud data into planes using a random sampling consensus algorithm;

[0099] In an embodiment of the present invention, the pre-processed point cloud data is segmented into planes using a random sampling consensus algorithm (RANSAC fitting algorithm) to determine point cloud data that can constitute a plane.

[0100] Step 122: Obtain the number of point clouds and normal directions of each of the planes;

[0101] In the embodiment of the present invention, the normal of each point cloud in the preprocessed point cloud data can be calculated through a nearest neighbor search algorithm (Neighbor Search algorithm) and a kd tree (k-dtree) data structure.

[0102] Step 123: Determine the plane whose number of point clouds is greater than a preset threshold and whose normal direction is parallel to the gravity direction as the ground in the current scene;

[0103] When the plane is segmented by the random sampling consensus algorithm (RANSAC fitting algorithm), the plane whose point cloud number is greater than a preset threshold and whose normal direction is parallel to the gravity direction is confirmed as the ground, and the plane equation representing the ground can be calculated.

[0104] Step 124: constructing an ideal ground normal vector in the camera coordinate system according to the orientation and tilt angle of the depth camera;

[0105] In an embodiment of the present invention, the mobile robot is also equipped with a posture sensor, or the depth camera is also equipped with a posture sensor, and the IMU data of the depth camera is collected through the posture sensor to obtain the shooting direction of the depth camera and its specific tilt angle data when it is in the current shooting direction.

[0106] Alternatively, the direction and tilt angle of the depth camera may also be preset manually, and the direction and tilt angle data of the depth camera may be obtained through the posture data of the depth camera stored in the system storage medium.

[0107] Furthermore, according to the orientation and tilt angle of the depth camera, the offset of the depth camera from the ground normal vector in the y-axis direction can be determined, so that the ideal ground normal vector can be calculated according to the tilt angle and the coordinate point on the y-axis of the camera coordinate system.

[0108] Step 125: Determine the ground coordinate system according to the ideal ground normal vector, and calculate a first transformation matrix from the camera coordinate system to the ground coordinate system.

[0109] In an embodiment of the present invention, a plane parallel to the ground can be determined based on the ground normal vector, and the plane equation of the ground can be obtained by calculation, and then the distance of the ground relative to the depth camera can be further determined based on the preset height of the depth camera to determine the height position of the ground origin, thereby determining the ground coordinate system. After determining the ground coordinate system, further, the first change matrix of the two can be calculated in combination with the camera coordinate system to achieve the conversion of the coordinate system.

[0110] Usually, when converting the coordinate system, it is usually necessary to rotate the camera coordinate system first and then translate it (translate it until the plane formed by the x-axis and the y-axis coincides with the determined ground) so that the camera coordinate system is converted to the ground coordinate system. Based on this, the calculation formula of the first transformation matrix can be obtained as follows:

[0111]

[0112] Wherein, T represents the first transformation matrix, R represents the rotation of the depth camera around itself until the x-axis has the same normal vector direction as the x-axis of the ideal ground (that is, the x-axis and y-axis planes of the camera coordinate system are parallel to the ground, and the z-axis is parallel to the gravity direction j = (0, 1, 0) T ), and t represents the translation vector that translates the origin of the rotated camera coordinate system to the plane where the ground is located.

[0113] Specifically, the calculation formula of the rotation matrix is ​​as follows:

[0114]

[0115] Among them, γ represents the rotation angle of the depth camera around its own z-axis, and α represents the rotation angle of the depth camera around its own x-axis.

[0116] In an embodiment of the present invention, since the point cloud data collected by the depth camera is represented in the camera coordinate system, in order to further perform plane segmentation on the point cloud data and further classify the planes according to the orientations of each plane, it is necessary to transform the point cloud data collected by the depth camera in the current scene into a ground coordinate system through the first transformation matrix.

[0117] In some embodiments, see Figure 5 , which shows Figure 2 A sub-process of step 140 in the method shown, wherein step 140 includes but is not limited to the following steps:

[0118] Step 141: according to the first transformation matrix, convert the point cloud data of each plane into a representation in a ground coordinate system, and obtain the plane equation of each plane in the ground coordinate system;

[0119] Specifically, the general equation of the plane in the ground coordinate system is as follows:

[0120] Ax+By+Cz+D=0

[0121] In the embodiment of the present invention, first, the point cloud data on each plane is converted from the camera coordinate system to the ground coordinate system through the above-mentioned first change matrix, and then, the plane equation coefficient (A, B, C, D) can be determined by substituting the coordinates of each point cloud in the plane represented by the ground coordinate system into the general equation. And, when the plane equation coefficient A = 0, the plane equation becomes By + Cz + D = 0, and the plane normal vector is n = (0, B, C), which represents a plane parallel to the x-axis; when the plane equation coefficient B = 0, the equation becomes Ax + Cz + D = 0, and the plane normal vector is n = (A, 0, C), which represents a plane parallel to the y-axis; when the plane equation coefficient C = 0, the equation becomes Ax + By + D = 0, and the plane normal vector is n = (A, B, 0), which represents a plane parallel to the x-axis.

[0122] Step 142: Filter out the horizontal step surfaces in the planes according to the plane equations of the planes expressed by the ground coordinate system.

[0123] In an embodiment of the present invention, after obtaining the above-mentioned plane equation, the horizontal step surface can be screened out according to the normal vector of the plane equation. Specifically, since the horizontal step surface is the same as the ground, the plane normal vectors are parallel to the y-axis. Therefore, the plane whose plane normal vector is n = (A, 0, C) and / or the plane whose plane equation satisfies Ax + Cz + D = 0 can be confirmed as a horizontal step surface.

[0124] In some embodiments, see Figure 6 , which shows Figure 2 A sub-process of step 150 in the method shown, wherein step 150 includes but is not limited to the following steps:

[0125] Step 151: Obtaining the centroid coordinates of each of the horizontal step surfaces in the ground coordinate system;

[0126] Wherein, the centroid coordinates can be expressed as μ = (μ x ,μ y ,μ z ).

[0127] Step 152: determining the height of each horizontal step surface relative to the ground according to the centroid coordinates of each horizontal step surface and sorting each horizontal step surface;

[0128] For example, when detecting the first level horizontal step surface, the height of each horizontal step surface is determined according to the centroid coordinates represented by the ground coordinate system, that is, the y coordinate in the centroid coordinates, and then the horizontal plane with a height within the height range of the first level horizontal step surface is taken as the first level horizontal step surface. At the same time, the first level horizontal step surface and the ground must be connected. Similarly, the other horizontal step surfaces of the stair structure are determined, and the horizontal step surfaces are sorted according to the height of each level of the horizontal step surface.

[0129] Furthermore, since a complete horizontal step surface may be divided into several horizontal planes, in this case, the point cloud data of the several horizontal planes can be merged into a cluster according to the heights of the several horizontal planes to obtain point cloud data for representing the complete first-level horizontal step surface.

[0130] Furthermore, it is also possible to determine whether the horizontal surface of the first step is the horizontal step surface of the ascending stairs or the horizontal step surface of the descending stairs based on the centroid coordinates. Specifically, the y coordinate in the centroid coordinates of the horizontal step surface, if greater than zero, is the horizontal step surface of the ascending stairs, and if less than zero, is the horizontal step surface of the descending stairs.

[0131] Step 153: constructing a two-dimensional circumscribed rectangle of each of the horizontal step surfaces;

[0132] In an embodiment of the present invention, after determining the main direction of each of the horizontal step surfaces through principal component analysis (PCA) analysis, a two-dimensional circumscribed rectangle of each of the horizontal step surfaces is constructed according to the main direction of the horizontal step surfaces, wherein the two-dimensional circumscribed rectangle is the minimum rectangle that encloses all point clouds in the horizontal plane.

[0133] Step 154: Calculate the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles;

[0134] After confirming the two-dimensional circumscribed matrix, the length and width of the two-dimensional circumscribed matrix are further calculated according to the coordinates of each point cloud on the two-dimensional circumscribed matrix, so as to calculate the area of ​​the two-dimensional circumscribed matrix. At the same time, the height difference between two adjacent two-dimensional circumscribed matrices can also be calculated according to the point cloud coordinates of the plane where the two adjacent two-dimensional circumscribed matrices are located, so as to determine the height of each step.

[0135] Step 155: obtaining the convex hull area of ​​each of the horizontal step surfaces and the area of ​​its two-dimensional circumscribed rectangle, and calculating the ratio of the convex hull area to the area of ​​its two-dimensional circumscribed rectangle;

[0136] In an embodiment of the present invention, a proportionality coefficient is defined, which is the ratio of the convex hull area of ​​the horizontal step surface to the area of ​​its two-dimensional circumscribed rectangle, and the ratio can be used to measure the quality of each layer of point cloud of the horizontal step surface collected by the depth camera. In addition, the convex hull refers to the circumscribed polygon of all point clouds in the plane.

[0137] Step 156: determining the best plane in the horizontal step surface according to the ratio;

[0138] In the embodiment of the present invention, the horizontal step surface with the largest ratio is determined as the optimal plane, the convex hull area of ​​the optimal plane is closest to the area of ​​its circumscribed rectangle, the directions of its three eigenvectors will be taken as the initial values ​​of the main directions of the entire stair model, and its width will be taken as the width of the entire stair model. The main direction of the optimal plane is taken as the main direction of the entire stair structure.

[0139] Step 157: construct and visualize the three-dimensional model of the stairs in the staircase coordinate system and determine the spatial position information of the stairs according to the ordered horizontal step surfaces, the length and width of each two-dimensional circumscribed rectangle, and the height difference between two adjacent two-dimensional circumscribed rectangles.

[0140] The centroid coordinates of the optimal plane in the staircase coordinate system are used as the overall center of the three-dimensional model of the entire staircase, and the three eigenvectors of the optimal plane are used as the main directions of the three-dimensional model of the entire staircase.

[0141] In an embodiment of the present invention, each of the horizontal step surfaces can be regarded as a rectangle. After calculating the length, width and height difference between two adjacent two-dimensional circumscribed rectangles of each of the horizontal step surfaces, the four vertices of the horizontal step surface (rectangle) can be determined based on the length, width and height difference between two adjacent two-dimensional circumscribed rectangles of the horizontal step surface and the centroid coordinates of each of the horizontal step surfaces.

[0142] Furthermore, in an embodiment of the present invention, after calculating the overall center of the three-dimensional model of the stair structure, the origin of the entire stair structure can be calculated based on the center coordinates of the center and the length, width and height of the entire stair structure. Using the origin coordinates of the stair structure and the main direction of the entire staircase, the following second transformation matrix can also be determined.

[0143] In some embodiments, please see Figure 7 , which shows Figure 6 A sub-process of step 154 ​​in the method shown, wherein step 154 ​​includes but is not limited to the following steps:

[0144] Step 1541: determining the covariance matrix of each of the horizontal step surfaces in the camera coordinate system, wherein the three eigenvectors of the covariance matrix form a second transformation matrix in the form of columns;

[0145] The covariance matrix is ​​a 3×3 matrix, and the three eigenvectors of the covariance matrix are the principal components of each of the horizontal step surfaces, which are used to characterize the main directions of each of the horizontal step surfaces. The second transformation matrix can be expressed as:

[0146] Step 1542: According to the second transformation matrix, transform the point cloud of each horizontal step surface in the camera coordinate system into a representation in the stair coordinate system;

[0147] According to the second transformation matrix, a coordinate system with the eigenvector as the coordinate axis can be obtained, that is, the staircase coordinate system can be expressed as: φ =(P-μ)·φ, wherein φ represents the second transformation matrix, and μ represents the centroid coordinates of the horizontal step surface.

[0148] Step 1543: Calculate the difference between the maximum value and the minimum value of the point coordinates of each point cloud in the two coordinate axis directions of the staircase coordinate system as the length and width of the two-dimensional circumscribed rectangle;

[0149] That is, calculate the horizontal step surface in the stair coordinate system, and in the eigenvector, the difference in the maximum value of the point coordinates calculated in the direction of the first eigenvector φ1 is the width of the horizontal step surface, and the difference in the maximum value of the point coordinates calculated in the direction of the second eigenvector φ2 is the length of the horizontal step surface. Since the horizontal step surface of each layer is almost flat, the height of the horizontal step surface can be ignored, that is, the eigenvector φ3 with the smallest eigenvalue can be ignored.

[0150] Step 1544: Obtain the height difference between two adjacent two-dimensional circumscribed rectangles according to the difference in Y-axis coordinates of the centroids of the horizontal step surface.

[0151] In an embodiment of the present invention, since the stair steps are parallel to the ground, the coordinates of the center of mass of two adjacent horizontal step surfaces can be obtained, and the difference in the Y-axis coordinates of the center of mass of the two adjacent horizontal step surfaces can be calculated to obtain the height difference between the two adjacent two-dimensional circumscribed rectangles, thereby obtaining the height of each level of the stair steps.

[0152] Embodiment 2

[0153] The embodiment of the present invention provides a stair detection device, please refer to Figure 8 , which shows the structure of a stair detection device provided by an embodiment of the present invention, the stair detection device 200 includes: a data acquisition module 210, a ground detection module 220, a scene segmentation module 230, a stair detection module 240 and a stair modeling module 250. Among them,

[0154] The data acquisition module 210 is used to obtain point cloud data around the current scene where the mobile robot is located;

[0155] The ground detection module 220 is used to determine the ground in the current scene according to the point cloud data and establish a ground coordinate system;

[0156] The scene segmentation module 230 is used to perform plane segmentation on the point cloud data to determine the plane in the current scene;

[0157] The stair detection module 240 is used to classify the plane in the ground coordinate system and filter out the horizontal step surface in the plane;

[0158] The stair modeling module 250 is used to establish and output a visualized three-dimensional model of the stair structure and its spatial position information according to the horizontal step surface.

[0159] In some embodiments, the scene segmentation module 230 is also used to cluster the point cloud data through a region growing algorithm; determine whether the clustered point cloud data is a plane through a random sampling consistency algorithm; if so, jump to the step of classifying the plane; if not, classify the non-planar point cloud data as obstacles, and further classify the non-clustered point cloud data as obstacles through a Euclidean clustering algorithm.

[0160] In some embodiments, the ground detection module 220 is also used to divide the point cloud data into planes through a random sampling consistency algorithm; obtain the number of point clouds and the normal direction of each plane; determine the plane whose point cloud number is greater than a preset threshold and whose normal direction is parallel to the gravity direction as the ground in the current scene; construct an ideal ground normal vector in the camera coordinate system according to the orientation and tilt angle of the depth camera; determine the ground coordinate system according to the ideal ground normal vector, and calculate the first transformation matrix from the camera coordinate system to the ground coordinate system.

[0161] In some embodiments, the stair detection module 240 is also used to convert the point cloud data of each of the planes into a representation using a ground coordinate system according to the first transformation matrix, and obtain the plane equations of each of the planes in the ground coordinate system; and filter out the horizontal step surfaces in the plane according to the plane equations of each of the planes represented by the ground coordinate system.

[0162] In some embodiments, the stair modeling module 250 is also used to obtain the centroid coordinates of each of the horizontal step surfaces in the ground coordinate system; determine the height of each of the horizontal step surfaces relative to the ground and sort the horizontal step surfaces based on the centroid coordinates of each of the horizontal step surfaces; construct a two-dimensional circumscribed rectangle of each of the horizontal step surfaces; calculate the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles; obtain the convex hull area of ​​each of the horizontal step surfaces and the area of ​​its two-dimensional circumscribed rectangle, and calculate the ratio of the convex hull area to the area of ​​its two-dimensional circumscribed rectangle; determine the best plane in the horizontal step surface based on the ratio; construct and visualize the output of the three-dimensional model of the stairs in the stair coordinate system based on the sorted horizontal step surfaces, as well as the length, width of each of the two-dimensional circumscribed rectangles and the height difference between two adjacent two-dimensional circumscribed rectangles, and determine the spatial posture information of the stairs, wherein the centroid coordinates of the best plane in the stair coordinate system are used as the overall center of the three-dimensional model of the entire stair, and the three eigenvectors of the best plane are used as the main directions of the three-dimensional model of the entire stair.

[0163] In some embodiments, the stair modeling module 250 is also used to determine the covariance matrix of each of the horizontal step surfaces in the camera coordinate system, and the three eigenvectors of the covariance matrix form a second transformation matrix in the form of columns; according to the second transformation matrix, the point cloud of each of the horizontal step surfaces in the camera coordinate system is converted to a representation in the stair coordinate system; the difference between the maximum and minimum values ​​of the point coordinates of each point cloud in the two coordinate axis directions of the stair coordinate system is calculated as the length and width of the two-dimensional circumscribed rectangle; the height difference between two adjacent two-dimensional circumscribed rectangles is obtained according to the difference in the Y-axis coordinates of the center of mass of the horizontal step surface.

[0164] Embodiment 3

[0165] The present invention also provides a mobile robot. Fig. 9 , which shows that it is possible to execute Figures 2 to 7 The hardware structure of the mobile robot of the stair detection method. The mobile robot 10 can be Figure 1 The mobile robot 10 shown in FIG. 1 includes a depth camera 11 mounted on the mobile robot 10 .

[0166] The mobile robot 10 further includes: at least one processor 12 connected to the depth camera 11; and a memory 13 in communication with the at least one processor 12. Fig. 9 The memory 13 stores instructions that can be executed by the at least one processor 12, and the instructions are executed by the at least one processor 12 so that the at least one processor 12 can perform the above Figures 2 to 7 The processor 12 and the memory 13 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0167] The memory 13 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the stair detection method in the embodiment of the present application, for example, Figure 8 The processor 12 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 13, that is, implements the stair detection method of the above method embodiment.

[0168] The memory 13 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the stair detection device, etc. In addition, the memory 13 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 13 may optionally include a memory remotely arranged relative to the processor 12, and these remote memories may be connected to the stair detection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The one or more modules are stored in the memory 13, and when executed by the one or more processors 12, perform the stair detection method in any of the above method embodiments, for example, perform the above described Figures 2 to 7 Steps to achieve Figure 8 The functions of each module and unit in it.

[0170] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0171] The present application also provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are executed by one or more processors, for example, to execute the above-described Figures 2 to 7 Steps to achieve Figure 8 The functions of each module in.

[0172] The present application also provides a computer program product, including a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the stair detection method in any of the above method embodiments, for example, executing the above described Figures 2 to 7 Steps to achieve Figure 8 The functions of each module in.

[0173] An embodiment of the present invention provides a stair detection method and device. The method first acquires point cloud data around a current scene where a mobile robot is located. Secondly, based on the point cloud data, the ground in the current scene is determined and a ground coordinate system is established. The point cloud data is then plane segmented to determine the plane in the current scene. Next, the plane is classified in the ground coordinate system and the horizontal step surfaces in the plane are screened out. Finally, based on the horizontal step surfaces, a visualized three-dimensional model of the stair structure and its spatial posture information are established and output. The stair detection method provided by the embodiment of the present invention enables the mobile robot to accurately detect the posture of the stair structure in the current scene and establish a three-dimensional model of the stair structure.

[0174] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A staircase detection method, characterized in that: Applied to a mobile robot, the mobile robot includes a depth camera, the depth camera is used to interact with the mobile robot for data, the method includes: Acquire point cloud data around the current scene where the mobile robot is located; Determine the ground in the current scene according to the point cloud data, and establish a ground coordinate system; Performing plane segmentation on the point cloud data to determine a plane in the current scene; In the ground coordinate system, classifying the planes, and screening out horizontal step surfaces in the planes; According to the horizontal step surface, a three-dimensional model of a visualized staircase structure and its spatial position information are established and output; The step of establishing and outputting a three-dimensional model of the stair structure and its spatial posture information for visualization according to the horizontal step surface specifically includes: Obtaining the centroid coordinates of each of the horizontal step surfaces in a ground coordinate system; According to the centroid coordinates of the horizontal step surfaces, determining the height of each horizontal step surface relative to the ground and sorting the horizontal step surfaces; Constructing a two-dimensional circumscribed rectangle of each of the horizontal step surfaces; Calculate the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles; Obtaining the convex hull area of ​​each of the horizontal step surfaces and the area of ​​its two-dimensional circumscribed rectangle, and calculating the ratio of the convex hull area to the area of ​​its two-dimensional circumscribed rectangle; Determining the optimal plane in the horizontal step surface according to the ratio; According to the sorted horizontal step surfaces, the length, width of each two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles, a three-dimensional model of the stairs in the stair coordinate system is constructed and visually output, and the spatial posture information of the stairs is determined, wherein the centroid coordinates of the optimal plane in the stair coordinate system are used as the overall center of the three-dimensional model of the entire stair, and the three eigenvectors of the optimal plane are used as the main directions of the three-dimensional model of the entire stair.

2. The staircase detection method according to claim 1, characterized in that: The step of performing plane segmentation on the point cloud data to determine the plane in the current scene further comprises: Clustering the point cloud data by using a region growing algorithm; Determining whether the point cloud data after clustering is a plane by a random sampling consistency algorithm; If yes, jump to the step of classifying the plane; If not, the non-planar point cloud data is classified as an obstacle, and the non-clustered point cloud data is further classified as the obstacle through the Euclidean clustering algorithm.

3. The staircase detection method according to any one of claims 1 or 2, characterized in that: The step of determining the ground in the current scene according to the point cloud data and establishing a ground coordinate system further comprises: Segmenting the point cloud data into planes by a random sampling consensus algorithm; Obtaining the number of point clouds and normal directions of each of the planes; Determine a plane whose number of point clouds is greater than a preset threshold and whose normal direction is parallel to the gravity direction as the ground in the current scene; Constructing an ideal ground normal vector in the camera coordinate system according to the orientation and tilt angle of the depth camera; The ground coordinate system is determined according to the ideal ground normal vector, and a first transformation matrix from the camera coordinate system to the ground coordinate system is calculated.

4. The staircase detection method according to claim 3, characterized in that: The step of classifying the planes in the ground coordinate system and screening out the horizontal step surfaces in the planes further comprises: According to the first transformation matrix, the point cloud data of each of the planes is converted into a representation in a ground coordinate system, and the plane equation of each of the planes in the ground coordinate system is obtained; According to the plane equations of each of the planes represented by the ground coordinate system, the horizontal step surfaces in the planes are screened out.

5. The staircase detection method according to claim 1, characterized in that: The step of calculating the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles further includes: Determine the covariance matrix of each of the horizontal step surfaces in the camera coordinate system, wherein three eigenvectors of the covariance matrix form a second transformation matrix in the form of columns; According to the second transformation matrix, the point cloud of each of the horizontal step surfaces in the camera coordinate system is transformed into a representation in the stair coordinate system; Calculate the difference between the maximum and minimum values ​​of the point coordinates of each point cloud in the two coordinate axis directions of the staircase coordinate system as the length and width of the two-dimensional circumscribed rectangle; The height difference between two adjacent two-dimensional circumscribed rectangles is obtained according to the difference in Y-axis coordinates of the centroids of the horizontal step surface.

6. A staircase detection device, characterized in that: Applied to a mobile robot, the mobile robot includes a depth camera, the depth camera is used to interact with the mobile robot for data, and the device includes: A data acquisition module, used to obtain point cloud data around the current scene where the mobile robot is located; A ground detection module, used to determine the ground in the current scene according to the point cloud data, and establish a ground coordinate system; A scene segmentation module, used for performing plane segmentation on the point cloud data to determine a plane in the current scene; A stair detection module, used to classify the plane in the ground coordinate system and filter out horizontal step surfaces in the plane; A stair modeling module, used to establish and output a visualized three-dimensional model of the stair structure and its spatial posture information according to the horizontal step surface; The stair modeling module is also used to obtain the centroid coordinates of each horizontal step surface in the ground coordinate system; determine the height of each horizontal step surface relative to the ground and sort each horizontal step surface according to the centroid coordinates of each horizontal step surface; construct a two-dimensional circumscribed rectangle of each horizontal step surface; calculate the length and width of the two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles; obtain the convex hull area of ​​each horizontal step surface and the area of ​​its two-dimensional circumscribed rectangle, and calculate the ratio of each convex hull area to the area of ​​its two-dimensional circumscribed rectangle; determine the best plane in the horizontal step surface according to the ratio; construct and visualize the output of the three-dimensional model of the stairs in the stair coordinate system according to the sorted horizontal step surfaces, and the length, width of each two-dimensional circumscribed rectangle and the height difference between two adjacent two-dimensional circumscribed rectangles, and determine the spatial posture information of the stairs, wherein the centroid coordinates of the best plane in the stair coordinate system are used as the overall center of the three-dimensional model of the entire stair, and the three eigenvectors of the best plane are used as the main directions of the three-dimensional model of the entire stair.

7. The staircase detection device according to claim 6, characterized in that: The scene segmentation module is also used to cluster the point cloud data through a region growing algorithm; determine whether the clustered point cloud data is a plane through a random sampling consistency algorithm; if so, jump to the step of classifying the plane; if not, classify the non-planar point cloud data as obstacles, and further classify the non-clustered point cloud data as obstacles through a Euclidean clustering algorithm.

8. The staircase detection device according to any one of claims 6 or 7, characterized in that: The ground detection module is also used to divide the point cloud data into planes through a random sampling consistency algorithm; obtain the number of point clouds and the normal direction of each plane; determine the plane whose point cloud number is greater than a preset threshold and whose normal direction is parallel to the gravity direction as the ground in the current scene; construct an ideal ground normal vector in the camera coordinate system according to the orientation and tilt angle of the depth camera; determine the ground coordinate system according to the ideal ground normal vector, and calculate the first transformation matrix from the camera coordinate system to the ground coordinate system.

9. The staircase detection device according to claim 8, characterized in that: The stair detection module is also used to convert the point cloud data of each plane into a representation in a ground coordinate system according to the first transformation matrix, and obtain the plane equation of each plane in the ground coordinate system; and screen out the horizontal step surface in the plane according to the plane equation of each plane represented by the ground coordinate system.

10. The staircase detection device according to claim 6, characterized in that: The stair modeling module is also used to determine the covariance matrix of each of the horizontal step surfaces in the camera coordinate system, and the three eigenvectors of the covariance matrix form a second transformation matrix in the form of columns; according to the second transformation matrix, the point cloud of each of the horizontal step surfaces in the camera coordinate system is converted to a representation in the stair coordinate system; Calculate the difference between the maximum and minimum values ​​of the point coordinates of each point cloud in the two coordinate axis directions of the stair coordinate system as the length and width of the two-dimensional circumscribed rectangle; obtain the height difference between two adjacent two-dimensional circumscribed rectangles based on the difference in the Y-axis coordinates of the center of mass of the horizontal step surface.

11. A mobile robot, characterized in that: include: a depth camera mounted on the mobile robot; at least one processor coupled to the depth camera; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.