Indoor top boundary position detection method and device, robot, and storage medium

Abstract

Embodiments of the present application provide an indoor top boundary position detection method and device, a robot and a storage medium. By performing line feature detection on each depth image, the position relationship of the line features is determined to quickly obtain the position information of the door frame and the roof of the indoor scene, thereby obtaining the indoor top boundary position. Compared with detecting only the obstacles on the travel route to obtain the boundary of the indoor scene, the indoor top boundary is less affected by the ground terrain and less affected by the obstruction of obstacles, thereby improving the boundary accuracy of the obtained indoor scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image analysis, in particular to an indoor top boundary position detection method and device, a robot and a storage medium. BACKGROUND

[0002] The navigation path planning of an indoor mobile robot during movement is generally a map established by collected environment information. In the prior art, the establishment of the map is generally achieved by exploration and identification of the ground environment by the mobile robot. However, due to the complex and changeable ground environment, such as irregular movement of personnel and pets, and changes in the placement of various furniture, the establishment of the map is slow and time-consuming and the accuracy is insufficient.

[0003] How to improve the efficiency and image accuracy of the map building of the mobile robot has become a technical problem that the industry is concerned about and explores. Using the feature information of the indoor roof to build a map and position, avoiding the interference caused by the uncertainty of the ground environment, to solve the above problems, has become one of the research directions. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an indoor top boundary position detection method and device, a robot and a storage medium, to improve the boundary accuracy of an indoor scene. The specific technical solutions are as follows:

[0005] In a first aspect, the embodiments of the present application provide an indoor top boundary position detection method, which comprises:

[0006] Obtaining a plurality of depth images of an indoor scene collected by a binocular camera and camera poses under each depth image, wherein the camera pose under any depth image is the pose of the binocular camera in a world coordinate system when collecting the depth image;

[0007] Performing line feature detection on each depth image to obtain a line feature region in each depth image;

[0008] Determining position information of a door frame and a roof of the indoor scene according to the line feature region in each depth image and the camera pose under each depth image.

[0009] In a possible implementation, the determination of the position information of the door frame and the roof of the indoor scene according to the line feature region in each depth image and the camera pose under each depth image comprises:

[0010] For each depth image, detecting the position relationship of the line feature region in the depth image to obtain a door frame feature region and / or a roof feature region in the depth image;

[0011] According to the camera pose under the depth image, the door frame feature region and / or the roof feature region in the depth image are converted to the world coordinate system;

[0012] The door frame feature region and the roof feature region in the world coordinate system are fused and corrected to obtain the position information of the door frame and the roof of the indoor scene.

[0013] In a possible implementation, the position relationship detection of the line feature regions in the depth image is performed to obtain the door frame feature region and / or the roof feature region in the depth image, including:

[0014] The position relationship detection of the line feature regions in the depth image is performed to obtain the position relationship detection result.

[0015] Based on the position relationship detection result, if there are three line feature regions satisfying that two line feature regions perpendicularly intersect at the same end point, the combination of the three line feature regions is determined as a roof feature region.

[0016] Based on the position relationship detection result, if there are two line feature regions satisfying that the two line feature regions perpendicularly intersect at an end point and there is no other line feature region intersecting at the end point, the combination of the two line feature regions is determined as a partial door frame feature region.

[0017] Based on the position relationship detection result, if there are a first line feature region, a second line feature region and a third line feature region satisfying that the first line feature region and the second line feature region perpendicularly intersect at one end point of the second line feature region, the second line feature region and the third line feature region perpendicularly intersect at the other end point of the second line feature region, and the first line feature region and the third line feature region are parallel, the combination of the first line feature region, the second line feature region and the third line feature region is determined as a door frame feature region.

[0018] In a possible implementation, the position relationship detection of the line feature regions in the depth image is performed to obtain the position relationship detection result, including:

[0019] For each depth image, a direction vector of each line feature region in the depth image is determined.

[0020] selecting two line feature regions in the depth image, determining a first vector of the two selected line feature regions, the first vector being a vector formed by a first end point and a second end point, the first end point being an end point of one of the two selected line feature regions, and the second end point being an end point of the other of the two selected line feature regions;

[0021] calculating a product of the first vector of the two selected line feature regions and the direction vector to obtain a first calculation result, and determining that the two selected line feature regions are coplanar if the first calculation result is zero;

[0022] determining that the two selected line feature regions intersect perpendicularly if the first calculation result is zero and a product of the direction vectors of the two selected line feature regions is -1.

[0023] In a possible implementation, after the detection of the positional relationship of the line feature regions in each depth image to obtain the door frame feature region and / or the roof feature region in the depth image, the method further comprises:

[0024] determining a center point of the door frame feature region and / or the roof feature region in each depth image, respectively;

[0025] calculating a first minimum Euclidean distance between the center point of the door frame feature region in each depth image and a position of the binocular camera when the depth image is captured, and determining that the door frame feature region is a false detection when the first minimum Euclidean distance is less than a preset threshold; and / or

[0026] calculating a second minimum Euclidean distance between the center point of the roof feature region in each depth image and the position of the binocular camera when the depth image is captured, and determining that the roof feature region is a false detection when the second minimum Euclidean distance is less than the preset threshold.

[0027] In a possible implementation, after the conversion of the door frame feature region and / or the roof feature region in the depth image to the world coordinate system according to the camera pose in the depth image, the method further comprises:

[0028] determining a motion trajectory of the binocular camera in a two-dimensional grid map according to the camera pose in each depth image;

[0029] projecting the door frame feature region and the roof feature region in each depth image into the two-dimensional grid map to obtain a projection line segment of the door frame feature region and the roof feature region;

[0030] calculate a third minimum Euclidean distance between a motion trajectory of the binocular camera in the two-dimensional grid map and a center point of a projected line segment of the door frame feature region; when the third minimum Euclidean distance is less than a preset threshold, determine that the door frame feature region is a false detection;

[0031] calculate a fourth minimum Euclidean distance between a motion trajectory of the binocular camera in the two-dimensional grid map and a center point of a projected line segment of the roof feature region; when the fourth minimum Euclidean distance is less than a preset threshold, determine that the roof feature region is a false detection.

[0032] In a possible implementation, after the door frame feature region and / or the roof feature region in the depth image are converted into the world coordinate system according to the camera pose under the depth image, the method further includes:

[0033] determine a motion trajectory of the binocular camera according to the camera pose under each depth image;

[0034] in a case where a first plane normal vector of a plane where the door frame feature region is located is parallel to a second plane normal vector of a plane where the motion trajectory of the binocular camera is located, determine that the door frame feature region is a false detection.

[0035] In a possible implementation, the determining the position information of the door frame and the roof of the indoor scene according to the line feature region in each depth image and the camera pose under each depth image includes:

[0036] for each depth image, convert each line feature region in the depth image into the world coordinate system according to the camera pose under the depth image;

[0037] perform position relationship detection on each line feature region in the world coordinate system, to obtain each door frame feature region and each roof feature region in the world coordinate system;

[0038] fuse and correct each door frame feature region and each roof feature region in the world coordinate system, to obtain the position information of the door frame and the roof of the indoor scene.

[0039] In a second aspect, an embodiment of the present application provides an indoor top boundary position detection device, and the device includes:

[0040] an image acquisition module, configured to acquire a plurality of depth images of an indoor scene captured by a binocular camera and a camera pose under each depth image, wherein for any depth image, the camera pose under the depth image is a pose of the binocular camera in a world coordinate system when the depth image is captured;

[0041] a line feature detection module, configured to perform line feature detection on each of the depth images to obtain a line feature region in each of the depth images;

[0042] a first position information determination module, configured to determine position information of a door frame and a roof of the indoor scene according to the line feature region in each of the depth images and a camera pose under each of the depth images.

[0043] In a possible implementation, the first position information determination module comprises:

[0044] a line feature detection sub-module, configured to, for each depth image, perform position relationship detection on a line feature region in the depth image to obtain a door frame feature region and / or a roof feature region in the depth image;

[0045] a first coordinate system conversion module, configured to convert the door frame feature region and / or the roof feature region in the depth image to the world coordinate system according to a camera pose under the depth image;

[0046] a fusion correction sub-module, configured to fuse and correct each of the door frame feature regions and the roof feature regions in the world coordinate system to obtain the position information of the door frame and the roof of the indoor scene.

[0047] In a possible implementation, the line feature detection sub-module comprises:

[0048] a position relationship detection unit, configured to, for each depth image, perform vertical intersection and coplanar position relationship detection on each line feature region in the depth image to obtain a position relationship detection result;

[0049] a roof feature determination unit, configured to, based on the position relationship detection result, if there are three line feature regions that satisfy: two of the line feature regions vertically intersect at a same end point, determine a combination of the three line feature regions as a roof feature region;

[0050] a door frame feature determination unit, configured to, based on the position relationship detection result, if there are two line feature regions that satisfy: the two line feature regions vertically intersect at an end point, and there is no other line feature region intersecting at the end point, determine a combination of the two line feature regions as a partial door frame feature region;

[0051] The door frame feature determination unit is further configured to, based on the position relationship detection result, if a first line feature region, a second line feature region and a third line feature region satisfy: the first line feature region and the second line feature region perpendicularly intersect at one end point of the second line feature region, the second line feature region and the third line feature region perpendicularly intersect at another end point of the second line feature region, and the first line feature region and the third line feature region are parallel, determine that the combination of the first line feature region, the second line feature region and the third line feature region is a door frame feature region.

[0052] In a possible implementation, the position relationship detection unit is specifically configured to:

[0053] For each depth image, determine a direction vector of each line feature region in the depth image;

[0054] select two line feature regions in the depth image, determine a first vector of the currently selected two line feature regions, the first vector being a vector formed by a first end point and a second end point, the first end point being an end point of one of the currently selected two line feature regions, and the second end point being an end point of the other of the currently selected two line feature regions;

[0055] calculate a product of the first vector of the currently selected two line feature regions and the direction vector to obtain a first calculation result; if the first calculation result is zero, the currently selected two line feature regions are coplanar;

[0056] if the first calculation result is zero and a product of the direction vectors of the currently selected two line feature regions is -1, the currently selected two line feature regions perpendicularly intersect.

[0057] In a possible implementation, the apparatus further includes:

[0058] a center point determination module configured to determine a center point of the door frame feature region and / or the roof feature region in each depth image;

[0059] a first false detection determination module configured to, for each depth image, calculate a first minimum Euclidean distance between a center point of the door frame feature region in the depth image and a position of the binocular camera when the depth image is captured; and determine that the door frame feature region is a false detection when the first minimum Euclidean distance is less than a preset threshold value.

[0060] and / or,

[0061] calculate a second minimum Euclidean distance between a center point of the roof feature region in the depth image and the position of the binocular camera when the depth image is captured; and determine that the roof feature region is a false detection when the second minimum Euclidean distance is less than a preset threshold value.

[0062] In a possible implementation, the apparatus further includes:

[0063] a first motion trajectory determination module configured to determine a motion trajectory of the binocular camera in a two-dimensional grid map according to the camera poses under each of the depth images;

[0064] a projection module configured to project each of the door frame feature regions and the roof feature region into the two-dimensional grid map to obtain a projection line segment of the door frame feature region and the roof feature region;

[0065] a second false detection determination module configured to calculate a third minimum Euclidean distance between the motion trajectory of the binocular camera in the two-dimensional grid map and a center point of the projection line segment of the door frame feature region; and determine the door frame feature region as a false detection when the third minimum Euclidean distance is less than a preset threshold.

[0066] The second false detection determination module is further configured to calculate a fourth minimum Euclidean distance between the motion trajectory of the binocular camera in the two-dimensional grid map and a center point of the projection line segment of the roof feature region; and determine the roof feature region as a false detection when the fourth minimum Euclidean distance is less than a preset threshold.

[0067] In a possible implementation, the apparatus further includes:

[0068] a second motion trajectory determination module configured to determine a motion trajectory of the binocular camera according to the camera poses under each of the depth images;

[0069] a third false detection determination module configured to determine the door frame feature region as a false detection when a first plane normal vector of a plane on which the door frame feature region is located is parallel to a second plane normal vector of a plane on which the motion trajectory of the binocular camera is located.

[0070] In a possible implementation, the first position information determination module includes:

[0071] a second coordinate system conversion module configured to, for each depth image, convert each line feature region in the depth image to the world coordinate system according to the camera pose under the depth image;

[0072] a position relationship detection module configured to detect position relationships of each line feature region in the world coordinate system to obtain each door frame feature region and each roof feature region in the world coordinate system.

[0073] A second position information determination module is configured to fuse and correct each door frame feature region and each roof feature region in the world coordinate system to obtain position information of the door frame and the roof of the indoor scene.

[0074] In a third aspect, an embodiment of the present application provides a mobile robot, comprising: a binocular camera, a memory, and a processor, wherein the binocular camera is configured to collect left and right eye images of an indoor scene and generate a depth image of the indoor scene based on the left and right eye images;

[0075] The memory is configured to store a computer program.

[0076] The processor is configured to execute the program stored in the memory to implement the indoor top boundary position detection method described in any of the embodiments of the present application.

[0077] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the indoor top boundary position detection method described in any of the embodiments of the present application.

[0078] The indoor top boundary position detection method, device, robot, and storage medium provided by the embodiments of the present application obtain a plurality of depth images of an indoor scene collected by a binocular camera and camera poses under each depth image, wherein the camera pose under any depth image is the pose of the binocular camera in a world coordinate system when the binocular camera collects the depth image; perform line feature detection on each depth image to obtain line feature regions in each depth image; and determine position information of a door frame and a roof of the indoor scene according to the line feature regions in each depth image and the camera poses under each depth image. As can be seen, in the present application, the position information of the door frame and the roof of the indoor scene can be quickly obtained by performing line feature detection on each depth image and judging the positional relationship of the line features, so that the indoor top boundary position is obtained. Compared with detecting only obstacles on a travel route to obtain the boundary of the indoor scene, the indoor top boundary is less affected by the ground terrain and less affected by the obstruction of obstacles, so that the boundary accuracy of the obtained indoor scene can be improved.

[0079] Of course, implementing any product or method of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0081] Figure 1 An example diagram of the indoor top boundary position detection method provided for the embodiments of the present application is shown in FIG. 1.

[0082] Figure 2 An example diagram of the mobile robot carrying a binocular camera provided for the embodiments of the present application is shown in FIG. 2.

[0083] Figure 3a An example diagram of the door frame feature provided for the embodiments of the present application is shown in FIG. 3.

[0084] Figure 3b An example diagram of the roof feature provided for the embodiments of the present application is shown in FIG. 4.

[0085] Figure 4 An example diagram of the indoor top boundary position detection method provided for the embodiments of the present application is shown in FIG. 1. Figure 1 An example diagram of the first refinement of step S3 in the method is shown in FIG. 5. An example diagram of the second refinement of step S3 in the method is shown in FIG. 6.

[0086] An example diagram of the refinement of step S31 in the method is shown in FIG. 7. Figure 5 An example diagram of the refinement of step S31 in the method is shown in FIG. 8. Figure 4 An example diagram of the refinement of step S311 in the method is shown in FIG. 9. An example diagram of the refinement of step S311 in the method is shown in FIG. 10.

[0087] An example diagram of the second refinement of step S3 in the method is shown in FIG. 6. Figure 6 An example diagram of the second refinement of step S3 in the method is shown in FIG. 6. Figure 5 An example diagram of the second refinement of step S3 in the method is shown in FIG. 6. An example diagram of the second refinement of step S3 in the method is shown in FIG. 6.

[0088] An example diagram of the second refinement of step S3 in the method is shown in FIG. 6. Figure 7 An example diagram of the second refinement of step S3 in the method is shown in FIG. 6. Figure 1 An example diagram of the second refinement of step S3 in the method is shown in FIG. 6. An example diagram of the second refinement of step S3 in the method is shown in FIG. 6.

[0089] An example diagram of the second refinement of step S3 in the method is shown in FIG. 6. Figure 8 An example diagram of the indoor top boundary position detection method provided for the embodiments of the present application is shown in FIG. 1. An example diagram of the indoor top boundary position detection method provided for the embodiments of the present application is shown in FIG. 1.

[0090] An example diagram of the indoor top boundary position detection method provided for the embodiments of the present application is shown in FIG. 1. Figure 9 An example diagram of the indoor top boundary position detection method provided for the embodiments of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0092] First, the professional terms in the embodiments of the present application are explained:

[0093] SLAM: English full name is Simultaneous Localization and Mapping, SLAM for short, real-time positioning and mapping;

[0094] Line feature: a representation unit taking a line segment as a whole as a constraint;

[0095] Key frame: an image frame that can represent key information in a motion process or a scene change;

[0096] Intersect: two straight lines have any intersection point, then they are intersected;

[0097] Parallel: the extension lines of two straight lines do not have any intersection point, then they are parallel;

[0098] Coplanar: if two straight lines in a three-dimensional space have an intersection point or are parallel, then the two straight lines are coplanar.

[0099] In order to solve the problem of low detection accuracy of indoor scene boundary position in the related art, the first aspect of the embodiments of the present application first provides an indoor top boundary position detection method, which can be applied to an electronic device. In specific applications, the electronic device can be a mobile robot with computing function, or a server for online control of the mobile robot, etc., all of which are within the protection scope of the present application.

[0100] Referring to Figure 1 , the embodiments of the present application provide an indoor top boundary position detection method, which comprises:

[0101] Step S1, acquiring a plurality of depth images of an indoor scene and camera poses under each depth image collected by a binocular camera, wherein the camera pose under any depth image is the pose of the binocular camera in a world coordinate system when collecting the depth image;

[0102] Among them, the binocular camera can be carried on the mobile robot. For example, as shown in Figure 2 , the ground mobile robot equipped with a logical operation unit carries a binocular camera, the angle between the binocular camera and the horizontal plane is α, which can be set to be greater than or equal to 45°, and the vertical field of view angle of the binocular camera is β (i.e. the vertical field of view range of the camera), which is related to the actual model of the binocular camera, and the size of the angle α between the binocular camera sensor and the horizontal plane and the vertical field of view angle β is not limited in the embodiments of the present application.

[0103] Among them, the depth image is obtained in real time by the binocular camera. Specifically, the left eye and the right eye of the binocular camera collect left eye images and right eye images respectively, match the pixel points of the left and right eye images, and calculate the depth of each pixel according to the matching result, thereby obtaining the depth image. The specific algorithm for obtaining the depth image based on the left eye image and the right eye image can be referred to the prior art, which is not limited in the present application. In one example, the depth image can be the depth image of the key frame created after the left and right eye images are synchronized, and the key frame is created continuously, thereby obtaining a plurality of depth images.

[0104] wherein the world coordinate system can be a three-dimensional coordinate system of the indoor scene established by self-definition, in an example, a mobile robot coordinate system Or can be established with the center O of the mobile robot, and the position of the mobile robot at the moment when the binocular camera is initialized is taken as the initial position, and the origin O of the robot coordinate system of the mobile robot at this moment is taken as the origin of the world coordinate system. The camera pose change can include rotation and translation; the camera pose under the depth image refers to the position of the binocular camera in the world coordinate system when the depth image is collected, and the camera pose under each depth image can be obtained according to the prior art, for example, can be calculated according to the depth image by a SLAM algorithm, or obtained by a positioning module carried on the mobile robot, which is not specifically limited in the present application.

[0105] Step S2, line feature detection is performed on each depth image to obtain a line feature region in each depth image;

[0106] The line feature region can be a set of point clouds in the region where the line feature is located, and the line feature detection on each depth image can be to respectively detect the line feature in each depth image by using computer vision technology, point cloud clustering algorithm or line feature extractor, etc. In an example, the line feature in the two-dimensional left and right images collected by the binocular camera can be recognized by using computer vision technology, and then the line feature region in the depth image is obtained by coordinate conversion. In an example, the point clouds in the depth image can be clustered by point cloud clustering, and then the point cloud clusters satisfying the line feature are determined based on the clustering results, so as to obtain the line feature region. In an example, the line feature in the depth image can also be detected by a line feature extractor. For example, as shown in Figure 3a and 3b the line feature region can be a line segment region shown in any one of Figure 3a and 3b .

[0107] Step S3, according to the line feature region in each depth image and the camera pose under each depth image, the position information of the door frame and the roof of the indoor scene is determined.

[0108] The camera pose under each depth image can realize the coordinate conversion between the image coordinate system (or camera coordinate system) of the depth image and the world coordinate system; according to the positional relationship of the line feature region in each depth image, the recognition of the door frame and the roof feature region can be realized. Combined with the line feature region in each depth image and the camera pose under each depth image, the position information of the door frame and the roof of the indoor scene can be finally determined.

[0109] In one example, the line feature regions in each depth image can be converted to the world coordinate system based on the camera pose of each depth image, and then the door frame feature region and the roof feature region can be determined according to the geometric positional relationship of the line feature regions in the world coordinate system, so as to determine the position of the door frame and the position of the roof of the indoor scene.

[0110] In one example, the door frame feature region and the roof feature region in each depth image can be determined according to the geometric positional relationship of the line feature regions in each depth image, and then the door frame feature region and the roof feature region in each depth image can be converted to the world coordinate system based on the camera pose of each depth image, so as to determine the door frame feature region and the roof feature region in the world coordinate system, and thus determine the position of the door frame and the position of the roof of the indoor scene.

[0111] It can be seen that, by the method of the embodiments of the present application, the position information of the door frame and the roof of the indoor scene can be quickly obtained by detecting the line features in each depth image and judging the positional relationship of the line features, so as to obtain the position of the top boundary of the indoor scene. Compared with detecting only the obstacles on the route to obtain the boundary of the indoor scene, the top boundary of the indoor scene is less affected by the ground terrain and less affected by the obstruction of the obstacles, so as to improve the accuracy of the obtained boundary of the indoor scene. In addition, only the line features in each depth image need to be detected, and the positional relationship of the line features can be quickly obtained, so as to simplify the detection steps of the door frame and the roof in the indoor scene and reduce the detection cost.

[0112] Figure 4 For Figure 1 The first refinement of step S3 is shown in FIG. 3, in which the step S3 can include the following steps in one possible implementation. Figure 4

[0113] Step S31, for each depth image, detecting the positional relationship of the line feature region in the depth image to obtain the door frame feature region and / or the roof feature region in the depth image.

[0114] The positional relationship of the line feature region can be perpendicular, intersecting, parallel, coplanar, etc. between the line features, and the positional relationship of the line features can be detected by computer vision technology when detecting the line feature region, or can be calculated based on the three-dimensional coordinates of the line segments corresponding to each line feature region.

[0115] For example, l j1 = (x j1 , y j1 , z j1 ) represents the coordinates of one end point of the line feature region L j in the three-dimensional space.​m1 = (x m1 , y m1 , z m1 ) represents the coordinates of one end point of the line feature region L m in the three-dimensional space, the direction vectors of the line feature region L j and the line feature region L m are s j = (q j , n j , p j ) and s m = (q m , n m , p m ) respectively, the positional relationship between the line feature region L j and the line feature region L m can be determined through the positional relationship between s j and s m . Thus, the line feature region combination satisfying the positional relationship of the door frame feature region can be obtained, and the line feature region combination satisfying the positional relationship of the roof feature region can be obtained. r(w) rc c

[0116] In step S32, the door frame feature region and / or the roof feature region in the depth image are converted into the world coordinate system according to the camera pose in the depth image.

[0117] Generally, the binocular camera is mounted on a mobile robot, and the mobile robot moves in an indoor scene. Each depth image is captured by the mobile robot (binocular camera) at different positions, so the pose of the device (binocular camera or mobile robot) when capturing each depth image is different. It can be considered that the relative pose between the binocular camera and the mobile robot does not change; the conversion relationship between the robot coordinate system and the camera coordinate system can be:

[0118] O r(w) = T c *O rc

[0119] wherein O r(w) represents the robot coordinate system, O c represents the camera coordinate system, T rc represents the conversion parameter between the binocular camera coordinate system and the robot coordinate system, in one example, the conversion parameter can be represented by a selection matrix and a translation vector. The conversion parameter can be obtained through pre-calibration, and the calibration process of the conversion parameter is prior art, which is not limited in the present application.

[0120] Generally, the image coordinate system of the depth image is the camera coordinate system. In some other embodiments, the image coordinate system of the depth image is different from the camera coordinate system. In this case, the conversion relationship between the image coordinate system and the camera coordinate system needs to be obtained. For example, for the depth image corresponding to the i-th key frame, i.e., the i-th depth image, the conversion relationship between the image coordinate system and the camera coordinate system of the i-th depth image can be represented as i = 0, 1, 2, … n, n is the total number of depth images; the j-th point in the i-th depth image can be P ij = (x ij ,y ij ,z ij ), i = 0, 1, 2, … n, j = 1, 2, 3, …, m, m is the number of points in the i-th depth image.

[0121] The point cloud set of the door frame and the roof feature in the i-th depth image is represented as and Convert them to the robot coordinate system:

[0122]

[0123]

[0124] T rm is the point cloud set of the door frame in the i-th depth image in the robot coordinate system, T rd is the point cloud set of the roof in the i-th depth image in the robot coordinate system.

[0125] According to T rm and T rd and the conversion relationship between the robot coordinate system and the world coordinate system, the point cloud set of the door frame and / or the roof in the i-th depth image in the world coordinate system can be obtained.

[0126] In step S33, the door frame feature regions and the roof feature regions in the world coordinate system are fused and corrected to obtain the position information of the door frame and the roof of the indoor scene.

[0127] In this case, for the same roof or door frame, the line features with large deviations can be removed, and then the position of the final door frame or roof can be obtained by averaging.

[0128] According to the position relationship of the line features, the door frame features and the roof features in the depth images can be determined, and the position information of the door frame and the roof of the indoor scene can be determined, thereby accelerating the detection efficiency of the door frame and the roof.

[0129] Figure 5 For Figure 4 A refinement of step S31 is shown in Fig. 3, which illustrates a possible implementation of step S31. In the implementation, step S31 can include the following steps. Figure 5

[0130] Step S311, for each depth image, detecting the position relationship between the line feature regions in the depth image to obtain a position relationship detection result.

[0131] The position relationship detection can include one or more of perpendicular, intersecting, parallel, and coplanar detection between the line feature regions, and the position relationship detection result can be one or more of perpendicular, intersecting, parallel, and coplanar between the line feature regions. The position relationship detection can be obtained by computer vision technology or based on the three-dimensional coordinates of the line segments corresponding to the line feature regions.

[0132] Specifically, the position relationship between the line features can be determined by detecting the quantitative relationship between the direction vectors of the line features. For example, if l j1 = (x j1 , y j1 , z j1 ) represents the coordinates of one end of line feature region L j in three-dimensional space, l m1 = (x m1 , y m1 , z m1 ) represents the coordinates of one end of line feature region L m in three-dimensional space, and the direction vectors of line feature region L j and line feature region L m are s j = (q j , n j , p j ) and s m = (q m , n m , p m ) respectively, then the position relationship between line feature region L j and line feature region L m can be determined by the position relationship between s j and s m .

[0133] ​​Step S312, based on the position relationship detection result, if there are three line feature regions satisfying that two line feature regions perpendicularly intersect at the same end point, the combination of the three line feature regions is determined as a roof feature region.

[0134] In the embodiment of the present application, if the line segments formed by the three line features perpendicularly intersect each other, the intersection points are the end points of a line segment of the three line features, and the three line segments are not coplanar, then the right angle of the coordinate system composed of the three line segments is determined as a roof feature. As shown in FIG. 6, the three line segments constituting the roof feature perpendicularly intersect at the same point. Figure 3b

[0135] Step S313, based on the position relationship detection result, if there are two line feature regions satisfying that the two line feature regions perpendicularly intersect at one end point and there is no other line feature region intersecting at the end point, the combination of the two line feature regions is determined as a partial door frame feature region.

[0136] Generally, a door frame feature is usually composed of two right angle features, but the binocular camera can only collect part of the door frame in the depth image collected at some angles, and cannot collect the complete door frame, in this case, only when the line segments formed by the two line features perpendicularly intersect at a point, the right angle feature composed of the two line features can be determined as a partial door frame. For example, the line segment 1 and the line segment 2 formed by the two line features intersect, and the end point of the line segment 2 is the intersection point of the line segment 1 and the line segment 2, then the right angle feature composed of the line segment 1 and the line segment 2 is regarded as a part of the door frame.

[0137] Step S314, the position information of the door frame and the roof of the indoor scene is obtained by fusing and correcting the door frame feature regions and the roof feature regions in the world coordinate system.

[0138] In the embodiment of the present application, based on the position relationship between the line features constituting the two partial door frame features, it can be determined whether the line features constituting the two partial door frame features can be determined as a door frame feature region. For example, there are line segment 1, line segment 2 and line segment 3 formed by three line features in a certain depth image, if the line segment 1 and the line segment 2 perpendicularly intersect at one end point of the line segment 2, the line segment 3 and the line segment 2 perpendicularly intersect at the other end point of the line segment 2, the line segment 1 and the line segment 3 are parallel and the line segment 1, the line segment 2 and the line segment 3 are coplanar, then it is considered that the line segment 1, the line segment 2 and the line segment 3 constitute a door frame feature region.

[0139] By judging the position relationship between the line features, the door frame feature and the roof feature are determined, the relationship between the point, line and plane is combined with the visual feature, and the complexity of the door frame and the roof detection is reduced by using the embodiment of the present application.

[0140] Figure 6 For Figure 5 ​A detailed schematic diagram of step S311 is shown below. Figure 6 As shown, in one possible implementation, step S311 may specifically include the following steps:

[0141] Step S3111: For each depth image, determine the direction vector of each line feature region in that depth image.

[0142] In one example, determining the direction vector of each line feature region can be done based on the left-to-right and top-to-bottom directions in each depth image. For instance, in a certain depth image, line segment 1 corresponding to line feature 1 has its left endpoint A and right endpoint B. Then, based on... The direction determines the direction vector of line feature 1. The direction vector of the line feature can be calculated based on the coordinates of any two points on the corresponding line segment. For example, l j1 =(x j1 ,y j1 ,z j1 ) is a line feature L j The coordinates of the left endpoint in three-dimensional space, l j2 =(x j2 ,y j2 ,z j2 ) is a line feature L j The coordinates of the right endpoint in three-dimensional space, l m1 =(x m1 ,y m1 ,z m1 ) is a line feature L m The coordinates of the left endpoint in three-dimensional space, l m2 =(x m2 ,y m2 ,z m2 ) is a line feature L m The coordinates of the right endpoint in three-dimensional space can be determined according to l. j1 and l j2 Calculate line feature L j The direction vector is represented as: s j =(q j ,n j ,p j According to l m1 and l m2 Calculate line feature L m The direction vector is represented as: s m =(q m ,n m ,p m Then the line feature L j Line feature L m The corresponding line segment can be represented by the following equation:

[0143]

[0144]

[0145] wherein the direction vectors of the line features L j and L m are s j =(q j ,n j ,p j ) and s m =(q m ,n m ,p m ) respectively.

[0146] In step S3112, two line feature regions are selected in the depth image, and a first vector of the two selected line feature regions is determined, the first vector being a vector formed by a first end point and a second end point, the first end point being an end point of one of the two selected line feature regions, and the second end point being an end point of the other of the two selected line feature regions.

[0147] In one example, the first vector of the two selected line feature regions is determined by first determining the coordinates of the end points of the two selected line feature regions, and the end points of the two selected line features are either both start points or both end points of the line segments, and the end points of the two selected line features are not the same point, then a vector formed by the two different end point coordinates is determined as the first vector of the two selected line features. For example, in a certain depth image, line feature 1 corresponds to line segment AB, and line feature 2 corresponds to line segment CD, A and C are the start points of the two line segments respectively, and B and D are the end points of the two line segments respectively, if A and C are the same point, then a vector formed by B and D is determined as the first vector of line feature 1 and line feature 2.

[0148] In step S3113, a product of the first vector and the direction vector of the two selected line feature regions is calculated to obtain a first calculation result, and if the first calculation result is zero, the two selected line feature regions are coplanar.

[0149] In the embodiments of the present application, the positional relationship of the line feature regions can be determined according to the quantitative relationship between the first vector between the line features and the direction vector corresponding to the line feature region. Specifically, if the product of the first vector and the direction vector of the two line feature regions is zero, it is determined that the two line feature regions are coplanar. For example, in a certain depth image, l j1 =(x j1 ,y j1 ,z j1 ) is a line feature L j ​The coordinates of an endpoint in three-dimensional space, l m1 =(x m1 ,y m1 ,z m1 ) is a line feature L m The coordinates of an endpoint in three-dimensional space, line feature L j Line feature L m The direction vectors are s j =(q j ,n j ,p j ) and s m =(q m ,n m ,p m ),like This indicates the line characteristic L j Line feature L m Coplanar; if not 0, it indicates line characteristic L. j Line feature L m Opposite sides.

[0150] Step S3114: If the first calculation result is zero, and the product of the direction vectors of the two currently selected line feature regions is -1, then the two currently selected line feature regions intersect perpendicularly.

[0151] In one example, two line feature regions can be determined to intersect perpendicularly if the product of the first vector and the direction vector of the two line feature regions is zero, and the product of the direction vectors of the two line feature regions is -1. For example, in a certain depth image, l j1 =(x j1 ,y j1 ,z j1 ) is a line feature L j The coordinates of an endpoint in three-dimensional space, l m1 =(x m1 ,y m1 ,z m1 ) is a line feature L m The coordinates of an endpoint in three-dimensional space, line feature L j Line feature L m The direction vectors are s j =(q j ,n j ,p j ) and s m =(q m ,n m ,p m ),like And s j ·s m =-1 indicates that the line characteristic L j Line feature Lm Perpendicular intersection.

[0152] In addition, the positional relationship between the line features can also be determined according to the parallel positional relationship between the direction vector and the first vector, for example, if S j parallel to Sm and s j not collinear, the line feature L m and the line feature L j are not parallel; m

[0153] if S s j not collinear, the line feature L m and the line feature L j are coincident; m

[0154] if S and s j not parallel, the line feature L m and the line feature L j intersect. m

[0155] According to the quantity relationship between the direction vector of the line feature and the vectors formed by the end points of different line features, the positional relationship between the line features can be determined, and the door frame feature and the roof feature can be determined, so that the detection method is simple and the detection efficiency is further improved.

[0156] In a possible implementation, after the step S31, the following steps can be further included:

[0157] Step 1, respectively determining the center points of the door frame feature region and / or the roof feature region in each depth image.

[0158] The center point of the door frame feature or the roof feature can be the center of the door frame feature or the roof feature in the geometric sense, or can be the cluster center of the point cloud of the door frame feature or the roof feature, and the like, which are all within the protection scope of the present application.

[0159] Step 2, for each depth image, calculating a first minimum Euclidean distance between the center point of the door frame feature region in the depth image and the position of the binocular camera when collecting the depth image; when the first minimum Euclidean distance is less than a preset threshold, determining that the door frame feature region is a false detection; and / or,

[0160] calculating a second minimum Euclidean distance between the center point of the roof feature region in the depth image and the position of the binocular camera when collecting the depth image; when the second minimum Euclidean distance is less than a preset threshold, determining that the roof feature region is a false detection. ​​​

[0161] In the embodiments of the present application, the minimum Euclidean distance between the center point of the door frame feature or the roof feature and the position of the binocular camera when collecting the depth image can be the first minimum Euclidean distance between the center point of the door frame feature or the roof feature and the mobile robot carrying the binocular camera. The preset threshold value can be an empirical value set according to actual conditions, and the principle of setting the preset threshold value is to meet: when the Euclidean distance between the door frame / roof and the mobile robot is less than the preset threshold value, it is considered to be a false detection. For example, the preset threshold value can be the radius of the robot. In one example, the coordinate point set of the door frame or the roof in the robot coordinate system in the depth image is Pij, the coordinate point of the robot in the depth image is Q, the radius of the robot is 10 cm, and the Euclidean distance between the coordinate point Q of the robot and each coordinate point in the coordinate point set Pij is calculated respectively. When the minimum Euclidean distance is less than 10 cm, it is considered that the detected door frame feature or roof feature is a false detection.

[0162] By comparing the minimum Euclidean distance between the center point of the door frame feature in each depth image and the position of the binocular camera with the threshold value, and comparing the minimum Euclidean distance between the center point of the roof feature and the position of the binocular camera, and determining whether the door frame feature and the roof feature are false detections according to the comparison result, the detection accuracy of the door frame and the roof is improved.

[0163] In a possible implementation, after the step S32, the following steps can also be included:

[0164] Step 11, determining the motion trajectory of the binocular camera in the two-dimensional grid map according to the camera pose in each depth image;

[0165] In the embodiments of the present application, the two-dimensional grid map can be obtained by the depth image collected by the binocular camera. In one example, a three-dimensional point cloud of an indoor scene can be established by the depth image, and the three-dimensional point cloud is projected on a two-dimensional map to form a two-dimensional discrete obstacle map, and then the two-dimensional discrete obstacle map is scanned to obtain a two-dimensional grid map. The specific way of establishing a three-dimensional point cloud of an indoor scene or a two-dimensional grid map by a depth image can refer to a related SLAM algorithm, which is not limited in the present application.

[0166] The motion trajectory of the binocular camera can be the motion trajectory of the binocular camera in the world coordinate system, which can be determined according to the pose of the binocular camera in each depth image.

[0167] Step 12, projecting each door frame feature region and roof feature region into the two-dimensional grid map to obtain the projection line segment of the door frame feature region and the roof feature region;

[0168] In the embodiments of the present application, the line segment representing the door frame feature or the roof feature can be projected into the two-dimensional grid map.

[0169] In step 13, a third minimum Euclidean distance between the motion trajectory of the binocular camera in the two-dimensional grid map and the center point of the projected line segment of the door frame feature region is calculated; when the third minimum Euclidean distance is less than a preset threshold, the door frame feature region is determined as a false detection.

[0170] In step 14, a fourth minimum Euclidean distance between the motion trajectory of the binocular camera in the two-dimensional grid map and the center point of the projected line segment of the roof feature region is calculated; when the fourth minimum Euclidean distance is less than a preset threshold, the roof feature region is determined as a false detection.

[0171] In the embodiments of the present application, the minimum Euclidean distance between the center point of the door frame feature or the roof feature and the motion trajectory of the binocular camera in the two-dimensional grid map can be the minimum Euclidean distance between the position of the door frame feature or the roof feature in the world coordinate system in the depth image and the motion trajectory of the binocular camera in the world coordinate system. The preset threshold is set in the same way as in step 2 described above, and will not be described here.

[0172] By using the embodiments of the present application, after obtaining the door frame feature and the roof feature, the minimum Euclidean distance between the door frame feature or the roof feature and the motion trajectory of the binocular camera in the two-dimensional grid map is compared, and whether the door frame feature and the roof feature are false detections is further determined according to the comparison result, thereby improving the detection accuracy of the door frame and the roof in the indoor scene.

[0173] In a possible implementation, after step S32, the following steps can also be included:

[0174] In step S21, the motion trajectory of the binocular camera is determined according to the camera pose in each depth image.

[0175] In step S22, in the case that the first plane normal vector of the plane where the door frame feature region is located is parallel to the second plane normal vector of the plane where the motion trajectory of the binocular camera is located, the door frame feature region is determined as a false detection.

[0176] In one example, after obtaining the motion trajectory of the binocular camera, the normal vector of the plane where the motion trajectory is located can be obtained. The normal vector of the plane where the door frame feature is located can be calculated according to the line segment constituting the door frame feature

[0177] By using the embodiments of the present application, whether the door frame feature is a false detection is determined according to the positional relationship between the normal vector of the plane where the door frame is located and the normal vector of the plane where the motion trajectory of the binocular camera is located, thereby further improving the detection accuracy of the door frame.

[0178] Figure 7 For Figure 1The second refinement of step S3 is illustrated in FIG. 3B, which shows that in one possible implementation, step S3 can include the following steps: Figure 7

[0179] Step S301, for each depth image, converting each line feature region in the depth image to a world coordinate system according to the camera pose under the depth image.

[0180] In the above step S301, after detecting the line features in the depth image, the line features in each depth image can be converted from a camera coordinate system to a world coordinate system, and the conversion relationship between the camera coordinate system and the robot coordinate system is the same as that in step S32, which will not be described again here.

[0181] Step S302, detecting the positional relationship of each line feature region in the world coordinate system to obtain each door frame feature region and each roof feature region in the world coordinate system.

[0182] In the above step S302, the detection of the positional relationship of each line feature region in the world coordinate system can be the detection of the positional relationship of all line features in the world coordinate system, or the detection of the positional relationship of the line features in the current depth image in the world coordinate system.

[0183] Step S303, fusing and correcting each door frame feature region and each roof feature region in the world coordinate system to obtain the position information of the door frame and the roof of the indoor scene.

[0184] In the above embodiment, after detecting the line features in each depth image, the door frame features and the roof features in each depth image can be determined through the positional relationship of the line features, and then the position information of the door frame and the roof of the indoor scene can be determined, thereby accelerating the detection efficiency of the door frame and the roof.

[0185] In order to illustrate the method of the above embodiment, the following will be described in combination with specific embodiments, referring to FIG. 3B. Figure 8 Figure 8 ​​As a flowchart of the indoor top boundary position detection method, firstly, it is judged whether the left and right eye images of the binocular camera are synchronous (if not, the timing needs to be adjusted to be synchronous), after synchronization, the left and right eye images of the binocular camera are obtained to match to obtain the initial pose of the camera, then the key frame data (the key frame data includes a depth image, and can also include at least one of a left eye image, a right eye image, and a left and right eye synthesized image) is continuously created, on one hand, the line feature detection is performed on the key frame image, on the other hand, the camera pose and the map point position are estimated based on the local optimization of each key frame data, then the position of the line feature is estimated according to the end points of the line feature and the projection, and the door frame and the roof are detected according to the relative position relationship between the line features, finally, the door frame and the roof features are filtered and selected according to the geometric relationship between the camera pose and the line features, to determine the appropriate door frame and roof.

[0186] Corresponding to the method embodiment, the embodiment of the present application also provides an indoor top boundary position detection device, as shown in the figure, the device can include the following modules: Figure 9

[0187] The image acquisition module 901 is configured to acquire a plurality of depth images of an indoor scene and camera poses under each depth image collected by a binocular camera, wherein the camera pose under any depth image is the pose of the binocular camera in a world coordinate system when collecting the depth image.

[0188] The line feature detection module 902 is configured to perform line feature detection on each depth image to obtain a line feature region in each depth image.

[0189] The first position information determination module 903 is configured to determine the position information of the door frame and the roof of the indoor scene according to the line feature region in each depth image and the camera pose under each depth image.

[0190] It can be seen that by the method of the embodiment of the present application, the line feature detection is only needed to be performed on each depth image, and the position relationship of the line feature is judged to quickly obtain the door frame and the roof of the indoor scene, thereby simplifying the detection steps of the door frame and the roof in the indoor scene and reducing the detection cost.

[0191] Optionally, in a possible implementation manner, the first position information determination module 903 includes:

[0192] The line feature detection submodule is configured to, for each depth image, perform position relationship detection on the line feature region in the depth image to obtain a door frame feature region and / or a roof feature region in the depth image.

[0193] ​The first coordinate system conversion module is configured to convert the door frame feature region and / or the roof feature region in the depth image to the world coordinate system according to a camera pose under the depth image.

[0194] The fusion and correction submodule is configured to fuse and correct the door frame feature regions and the roof feature regions in the world coordinate system to obtain position information of the door frame and the roof of the indoor scene.

[0195] Optionally, in a possible implementation, the line feature detection submodule includes:

[0196] The position relationship detection unit is configured to, for each depth image, perform vertical intersection and coplanar position relationship detection on the line feature regions in the depth image to obtain a position relationship detection result.

[0197] The roof feature determination unit is configured to, based on the position relationship detection result, if there are three line feature regions that satisfy vertical intersection at the same end point, determine a combination of the three line feature regions as a roof feature region.

[0198] The door frame feature determination unit is configured to, based on the position relationship detection result, if there are two line feature regions that satisfy vertical intersection at an end point and no other line feature region intersects with the end point, determine a combination of the two line feature regions as a partial door frame feature region.

[0199] The door frame feature determination unit is further configured to, based on the position relationship detection result, if there are a first line feature region, a second line feature region, and a third line feature region that satisfy vertical intersection of the first line feature region and the second line feature region at one end point of the second line feature region, vertical intersection of the second line feature region and the third line feature region at the other end point of the second line feature region, and parallelism of the first line feature region and the third line feature region, determine a combination of the first line feature region, the second line feature region, and the third line feature region as a door frame feature region.

[0200] Optionally, in a possible implementation, the position relationship detection unit is specifically configured to:

[0201] For each depth image, determine a direction vector of each line feature region under the depth image.

[0202] Select two line feature regions in the depth image, and determine a first vector of the currently selected two line feature regions, where the first vector is a vector formed by a first end point and a second end point, the first end point is an end point of one of the currently selected two line feature regions, and the second end point is an end point of the other of the currently selected two line feature regions.

[0203] calculating a product of the first vector and the direction vector of the two currently selected line feature regions, to obtain a first calculation result; if the first calculation result is zero, the two currently selected line feature regions are coplanar;

[0204] if the first calculation result is zero and the product of the direction vectors of the two currently selected line feature regions is -1, the two currently selected line feature regions intersect perpendicularly.

[0205] Optionally, in a possible implementation, the apparatus further includes:

[0206] a center point determination module configured to determine a center point of the door frame feature region and / or the roof feature region in each of the depth images;

[0207] a first false detection determination module configured to, for each of the depth images, calculate a first minimum Euclidean distance between the center point of the door frame feature region in the depth image and a position of the binocular camera when the depth image is captured; and determine the door frame feature region as a false detection when the first minimum Euclidean distance is less than a preset threshold.

[0208] and / or,

[0209] calculate a second minimum Euclidean distance between the center point of the roof feature region in the depth image and the position of the binocular camera when the depth image is captured; and determine the roof feature region as a false detection when the second minimum Euclidean distance is less than a preset threshold.

[0210] Optionally, in a possible implementation, the apparatus further includes:

[0211] a first motion trajectory determination module configured to determine a motion trajectory of the binocular camera in a two-dimensional grid map according to the camera poses in the depth images;

[0212] a projection module configured to project the door frame feature region and the roof feature region in the depth images into the two-dimensional grid map, to obtain a projection line segment of the door frame feature region and the roof feature region;

[0213] a second false detection determination module configured to calculate a third minimum Euclidean distance between the motion trajectory of the binocular camera in the two-dimensional grid map and a center point of the projection line segment of the door frame feature region; and determine the door frame feature region as a false detection when the third minimum Euclidean distance is less than a preset threshold.

[0214] The second false detection determination module is further configured to calculate a fourth minimum Euclidean distance between a motion trajectory of the binocular camera in the two-dimensional grid map and a center point of a projection line segment of the roof feature region; and determine the roof feature region as a false detection when the fourth minimum Euclidean distance is less than a preset threshold.

[0215] Optionally, in a possible implementation, the apparatus further includes:

[0216] A second motion trajectory determination module configured to determine a motion trajectory of the binocular camera according to camera poses in each of the depth images.

[0217] A third false detection determination module configured to determine the door frame feature region as a false detection when a first plane normal vector of a plane in which the door frame feature region is located is parallel to a second plane normal vector of a plane in which the motion trajectory of the binocular camera is located.

[0218] Optionally, in a possible implementation, the first position information determination module 903 includes:

[0219] A second coordinate system conversion module configured to, for each depth image, convert each line feature region in the depth image to the world coordinate system according to a camera pose in the depth image.

[0220] A position relationship detection module configured to detect position relationships of each line feature region in the world coordinate system to obtain each door frame feature region and each roof feature region in the world coordinate system.

[0221] A second position information determination module configured to fuse and correct each door frame feature region and each roof feature region in the world coordinate system to obtain position information of a door frame and a roof of the indoor scene.

[0222] In yet another embodiment provided in the present application, a robot is also provided, which is equipped with a binocular camera and used to implement the indoor top boundary position detection method in any of the above embodiments.

[0223] Embodiments of the present application also provide a mobile robot, which includes:

[0224] A binocular camera, a memory, and a processor, wherein the binocular camera is configured to collect left and right eye images of an indoor scene and generate depth images of the indoor scene based on the left and right eye images.

[0225] The memory is configured to store a computer program.

[0226] The processor is configured to execute the program stored in the memory to implement the indoor top boundary position detection method in any of the embodiments of the present application.

[0227] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0228] The aforementioned processor can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0229] In yet another embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the aforementioned indoor top boundary position detection methods.

[0230] In yet another embodiment provided in the present application, a computer program product containing instructions, which, when run on a computer, causes the computer to execute any of the aforementioned indoor top boundary position detection methods.

[0231] In the embodiments described above, all or some of the steps can be implemented by using software, hardware, firmware or any combination thereof. When implemented with software, all or some of the steps can be implemented in the form of one or more computer programs which are stored in a computer readable storage medium and executed by one or more computers. The computer readable storage medium can include one or more of a floppy drive, a flexible disk drive, a hard disk drive, a solid state drive, a magnetic tape, a CD-ROM, a DVD, a Blu-ray disc, a memory stick, a memory card, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a flash memory, or any other suitable device. The computer readable storage medium can be fixed or removable. The computer readable storage medium can be loaded into a computer or other suitable device and executed by the computer or other suitable device. The computer readable storage medium can be distributed over a wired or wireless network, for example, the Internet, Intranet, Extranet, LAN, WLAN, or the like.

[0232] It should be noted that, in the present document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "includes", "including", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0233] Each of the embodiments described in the present document is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the embodiments of the device, the mobile robot, and the storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments.

[0234] The preferred embodiments of the present application have been described above with the aid of drawing provided only by way of example and therefore changes in form and detail can be made therein without departing from the spirit and scope of the application. Therefore, the above-described embodiments are intended to be illustrative only and not restrictive of the scope of the present application. Any modification, equivalent replacement or improvement made without departing from the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A method of detecting a position of a top edge of a room, characterized by, The method comprises the following steps: acquiring a plurality of depth images of an indoor scene captured by a binocular camera and camera poses under each depth image, wherein the camera pose under any depth image is the pose of the binocular camera in a world coordinate system when capturing the depth image; performing line feature detection on each depth image to obtain a line feature region in each depth image; determining position information of a door frame and a roof of the indoor scene according to the line feature regions in each depth image and the camera poses under each depth image; the step of determining the position information of the door frame and the roof of the indoor scene according to the line feature regions in each depth image and the camera poses under each depth image comprises: for each depth image, performing position relationship detection on the line feature regions in the depth image to obtain a door frame feature region and / or a roof feature region in the depth image; converting the door frame feature region and / or the roof feature region in the depth image to the world coordinate system according to the camera pose under the depth image; fusing and correcting each door frame feature region and roof feature region in the world coordinate system to obtain the position information of the door frame and the roof of the indoor scene; the step of performing position relationship detection on each line feature region in the depth image to obtain a door frame feature region and / or a roof feature region in the depth image comprises: for each depth image, performing perpendicular intersection and coplanar position relationship detection on each line feature region in the depth image to obtain a position relationship detection result; based on the position relationship detection result, if there are three line feature regions satisfying that two line feature regions perpendicularly intersect at the same end point, the combination of the three line feature regions is determined as a roof feature region; based on the position relationship detection result, if there are two line feature regions satisfying that the two line feature regions perpendicularly intersect at an end point and there is no other line feature region intersecting at the end point, the combination of the two line feature regions is determined as a partial door frame feature region; based on the position relationship detection result, if there are a first line feature region, a second line feature region and a third line feature region satisfying that the first line feature region and the second line feature region perpendicularly intersect at one end point of the second line feature region, the second line feature region and the third line feature region perpendicularly intersect at the other end point of the second line feature region, and the first line feature region and the third line feature region are parallel, the combination of the first line feature region, the second line feature region and the third line feature region is determined as a door frame feature region.

2. The method of claim 1, wherein, the step of performing perpendicular intersection and coplanar position relationship detection on each line feature region in the depth image to obtain a position relationship detection result comprises: for each depth image, determining a direction vector of each line feature region under the depth image; selecting two line feature regions in the depth image, determining a first vector of the two selected line feature regions, the first vector being a vector formed by a first end point and a second end point, the first end point being an end point of one of the two selected line feature regions, and the second end point being an end point of the other of the two selected line feature regions; calculating a product of the first vector of the two selected line feature regions and the direction vector to obtain a first calculation result; if the first calculation result is zero, the two selected line feature regions are coplanar; if the first calculation result is zero and a product of the direction vectors of the two selected line feature regions is -1, the two selected line feature regions intersect perpendicularly.

3. The method of claim 1, wherein, After the detection of the positional relationship of the line feature regions in each depth image to obtain the door frame feature region and / or the roof feature region in the depth image, the method further comprises: determining a center point of the door frame feature region and / or the roof feature region in each depth image, respectively; calculating a first minimum Euclidean distance between the center point of the door frame feature region in the depth image and a position of the binocular camera when the depth image is captured; when the first minimum Euclidean distance is less than a preset threshold, the door frame feature region is determined as a false detection; and / or calculating a second minimum Euclidean distance between the center point of the roof feature region in the depth image and the position of the binocular camera when the depth image is captured; when the second minimum Euclidean distance is less than the preset threshold, the roof feature region is determined as a false detection.

4. The method of claim 1, wherein, After the conversion of the door frame feature region and / or the roof feature region in the depth image to the world coordinate system according to the camera pose of the depth image, the method further comprises: determining a motion trajectory of the binocular camera in a two-dimensional grid map according to the camera pose of each depth image; projecting each of the door frame feature regions and the roof feature regions into the two-dimensional grid map to obtain a projection line segment of the door frame feature region and the roof feature region; calculating a third minimum Euclidean distance between the motion trajectory of the binocular camera in the two-dimensional grid map and a center point of the projection line segment of the door frame feature region; when the third minimum Euclidean distance is less than a preset threshold, the door frame feature region is determined as a false detection; calculating a fourth minimum Euclidean distance between the motion trajectory of the binocular camera in the two-dimensional grid map and a center point of the projection line segment of the roof feature region; when the fourth minimum Euclidean distance is less than a preset threshold, the roof feature region is determined as a false detection.

5. The method of claim 1, wherein, After the conversion of the door frame feature region and / or the roof feature region in the depth image to the world coordinate system according to the camera pose of the depth image, the method further comprises: determining a motion trajectory of the binocular camera according to the camera pose of each depth image; In a case that a first plane normal vector of a plane where the door frame feature region is located is parallel to a second plane normal vector of a plane where a motion trajectory of the binocular camera is located, the door frame feature region is determined as a false detection.

6. The method of claim 1, wherein, The determining the position information of the door frame and the roof of the indoor scene according to the line feature regions in each of the depth images and the camera poses under each of the depth images comprises: For each depth image, the line feature regions in the depth image are converted into the world coordinate system according to the camera pose under the depth image; The position relationship detection is performed on the line feature regions in the world coordinate system, to obtain each door frame feature region and each roof feature region in the world coordinate system; The position information of the door frame and the roof of the indoor scene is obtained by fusing and correcting the door frame feature regions and the roof feature regions in the world coordinate system.

7. An indoor top boundary position detection device characterized by comprising: The apparatus comprises: an image acquisition module configured to acquire a plurality of depth images of an indoor scene and camera poses under each depth image, which are captured by a binocular camera, wherein the camera pose under any depth image is a pose of the binocular camera in a world coordinate system when the depth image is captured; a line feature detection module configured to perform line feature detection on each of the depth images to obtain line feature regions in each of the depth images; a first position information determination module configured to determine position information of a door frame and a roof of the indoor scene according to the line feature regions in each of the depth images and the camera poses under each of the depth images; the first position information determination module comprises: a line feature detection submodule configured to, for each depth image, perform position relationship detection on line feature regions in the depth image to obtain door frame feature regions and / or roof feature regions in the depth image; a first coordinate system conversion module configured to convert the door frame feature regions and / or the roof feature regions in the depth image into the world coordinate system according to the camera pose under the depth image; a fusion and correction submodule configured to fuse and correct each of the door frame feature regions and the roof feature regions in the world coordinate system to obtain the position information of the door frame and the roof of the indoor scene; the line feature detection submodule comprises: a position relationship detection unit configured to, for each depth image, perform vertical intersection and coplanar position relationship detection on line feature regions in the depth image to obtain a position relationship detection result; a roof feature determination unit configured to, based on the position relationship detection result, determine a combination of three line feature regions as a roof feature region if the three line feature regions satisfy the condition that each pair of line feature regions intersects perpendicularly at a same end point; a door frame feature determination unit configured to, based on the position relationship detection result, determine a combination of two line feature regions as a partial door frame feature region if the two line feature regions satisfy the condition that the two line feature regions intersect perpendicularly at an end point and no other line feature region intersects at the end point. The door frame feature determination unit is further configured to determine, based on the position relationship detection result, that a combination of the first line feature region, the second line feature region, and the third line feature region is a door frame feature region if the first line feature region, the second line feature region, and the third line feature region satisfy the following conditions: the first line feature region and the second line feature region are perpendicular and intersect at one end point of the second line feature region, the second line feature region and the third line feature region are perpendicular and intersect at another end point of the second line feature region, and the first line feature region and the third line feature region are parallel.

8. A mobile robot, characterized by The mobile robot comprises a binocular camera, a memory, and a processor, wherein the binocular camera is configured to capture left and right eye images of an indoor scene and generate a depth image of the indoor scene based on the left and right eye images; The memory is configured to store a computer program; The processor is configured to execute the computer program stored in the memory to implement the indoor top boundary position detection method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the indoor top boundary position detection method according to any one of claims 1-6.

Images