Information processing device, information processing method, and program

Abstract

This technology relates to an information processing device, an information processing method, and a program that make it possible to robustly detect an arbitrary plane. First points on prescribed direction planes, the normal directions of which are a prescribed direction, are detected from points on objects, and second points on a specific plane, which is a specific prescribed direction plane, are detected from the first points. Then, the specific plane is detected by using the second points. This technology can be applied, for example, when an arbitrary plane such as a floor surface is detected in a VR system, etc.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable, for example, robust detection of an arbitrary plane. [Background technology]

[0002] In a VR (virtual reality) system, a user wears an HMD (head-mounted display) on their head, and an image of the virtual world is displayed that matches the movement of the user's head, giving the user the feeling that they are actually in the virtual world.

[0003] When an image of a virtual world is displayed on an HMD, the user cannot see the real world around them, so the VR system sets up a play area within which the user can move so that the user does not collide with obstacles such as walls in the real world.

[0004] In addition, the VR system needs to recognize the floor of the real world in order to display an image of the floor on which the user can move, etc. Therefore, the user needs to manually specify the floor (height) using a controller, etc., which is cumbersome.

[0005] Therefore, Patent Document 1 describes a technique for detecting the floor surface.

[0006] In the technology of Patent Document 1, points on an object captured in an image taken by a camera are plotted in a three-dimensional space, and when the three-dimensional space is sliced by a plurality of planes perpendicular to gravity, the number of points plotted in each slice space between adjacent planes is counted.The point plotted in the slice space with the lowest height and with a number of points equal to or greater than a threshold is detected as the floor surface. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-031673 Summary of the Invention [Problem to be solved by the invention]

[0008] In the technology of Patent Document 1, a point cloud that does not have information on normal directions is used as information on points on an object, so points plotted in slice space are not necessarily points on a horizontal plane.

[0009] Therefore, the points plotted in the slice space may include many points on planes other than the horizontal plane, and if the floor area on the image is small, the floor points plotted in the slice space may be buried in other points, including points on planes other than the horizontal plane, making it impossible to accurately detect the floor.

[0010] This technology was developed in light of these circumstances, and enables robust detection of floors and other arbitrary flat surfaces in three-dimensional space. [Means for solving the problem]

[0011] The information processing device or program of the present technology is an information processing device that includes a first point detection unit that detects a first point on a predetermined direction plane whose normal direction is in a predetermined direction from a point on an object, a second point detection unit that detects a second point on a specific surface that is a specific predetermined direction plane from the first point, and a specific surface detection unit that detects the specific surface using the second point, or a program for causing a computer to function as such an information processing device.

[0012] The information processing method of the present technology is an information processing method that includes detecting a first point on a predetermined direction plane whose normal direction is a predetermined direction from a point on an object, detecting a second point on a specific surface that is a specific predetermined direction plane from the first point, and detecting the specific surface using the second point.

[0013] In the information processing device, information processing method, and program of the present technology, a first point on a predetermined direction plane whose normal direction is a predetermined direction is detected from a point on an object, and a second point on a specific surface that is a specific predetermined direction plane is detected from the first point. Then, the specific surface is detected using the second point.

[0014] The information processing device may be an independent device or an internal block constituting one device.

[0015] The program can be provided by being recorded on a recording medium or by being transmitted via a transmission medium. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a configuration example of an embodiment of an information processing system to which the present technology is applied. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a plane estimation unit 33. [Figure 3] FIG. 1 is a diagram illustrating a model of a horizontal plane including a floor surface. [Figure 4] FIG. 1 is a diagram illustrating RANSAC. [Figure 5] FIG. 10 is a diagram illustrating the processing of the point detection unit 42 when detecting points on a floor surface using RANSAC. [Figure 6] FIG. 10 is a diagram illustrating the processing of a point detection unit 42 that employs a first detection method that detects points on a floor surface using a voting method. [Figure 7] 10 is a diagram illustrating the processing of the point detection unit 42 that employs the second detection method of detecting points on the floor surface using the voting method. FIG. [Figure 8] FIG. 10 is a block diagram showing an example of the configuration of a point detection unit 42 that employs a second detection method that detects points on a floor surface using a voting method. [Figure 9] 10 is a flowchart illustrating the processing of the information processing device 12 when detecting a floor surface. [Figure 10] This is a diagram showing a case where the peak class of the inlier point histogram is a class of height lower than the true height of the floor surface. [Figure 11] FIG. 10 is a diagram illustrating another peak class detection method. [Figure 12] FIG. 1 is a diagram illustrating DBSCAN. [Figure 13] FIG. 10 is a diagram illustrating class clustering of inlier point histograms using DBSCAN. [Figure 14] 1 is a block diagram illustrating an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE INVENTION

[0017] <Information processing system applying this technology>

[0018] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of an information processing system to which the present technology is applied.

[0019] 1, an information processing system 10 provides a user with images (and sounds) that allow the user to enjoy VR or AR (augmented reality). Use cases of the information processing system 10 include, for example, computer games, theme park attractions, training simulators, and the like.

[0020] The information processing system 10 includes an HMD 11 and an information processing device 12. The HMD 11 and the information processing device 12 can exchange various types of information (data) by performing wireless or wired communication.

[0021] The HMD 11 is a non-transparent HMD that is worn on the user's head and covers the entire field of view of the user, and displays images and the like supplied (transmitted) from the information processing device 12 under the control of the information processing device 12.

[0022] The HMD 11 includes a camera 21 , an IMU (inertial measurement unit) 22 , a distance measurement sensor 23 , and a display unit 24 .

[0023] The camera 21 captures an image and supplies (transmits) the image to the information processing device 12. A plurality of cameras may be provided as the camera 21. For example, the plurality of cameras 21 may include cameras that capture images in front of, behind, and to the sides of the user.

[0024] The IMU 22 senses acceleration and angular velocity and supplies the sensed values to the self-position estimation unit 31 .

[0025] The ranging sensor 23 senses distance and supplies the distance to the information processing device 12. For example, the ranging sensor 23 senses the distance to each point on a subject (object) shown in an image captured by the camera 21, and supplies a distance image in which the distance is represented as a pixel value to the information processing device 12. As the ranging sensor 23, for example, a ToF (time of flight) sensor such as a LiDAR (light detection and ranging) sensor can be provided.

[0026] Furthermore, a stereo camera can be provided as the distance measurement sensor 23. When a stereo camera is provided as the distance measurement sensor 23, the stereo camera as the distance measurement sensor 23 can also be used as the camera 21.

[0027] The display unit 24 displays an image supplied from the information processing device 12. As the display unit 24, for example, a display unit for the left eye and a display unit for the right eye of the user can be provided.

[0028] The information processing device 12 performs various processes using the images, acceleration, angular velocity, and distance images supplied from the HMD 11. The information processing device 12 also generates images for VR or AR, supplies them to the HMD 11, and displays them on the display unit 24.

[0029] The information processing device 12 includes a self-position estimation unit 31, a point information generation unit 32, and a plane estimation unit 33.

[0030] The self-position estimation unit 31 estimates its own position by SLAM (simultaneous localization and mapping) or the like using the image, acceleration, and angular velocity supplied from the HMD 11 to the information processing device 12. The self-position estimation unit 31 performs self-position estimation with the camera 21 (HMD 11) as the target of self-position estimation, thereby calculating (finding) the camera position and attitude of the camera 21 and the direction of gravity, and supplies these to the point information generation unit 32.

[0031] The information processing device 12 generates an image (computer graphics or the like) for VR or AR according to the camera position, posture, etc., and displays it on the display unit 24 of the HMD 11.

[0032] By viewing a VR image displayed on the display unit 24 of the HMD 11 according to, for example, the camera position and posture, the user can enjoy the sensation of being in a virtual world.

[0033] The point information generation unit 32 uses the distance image supplied from the distance measurement sensor 23 to the information processing device 12, the camera position and attitude from the self-position estimation unit 31, and the direction of gravity to generate point information relating to each point on an object, for example, where the distance to each point has been sensed.

[0034] The point information of a point on an object includes the coordinates (position) and normal direction of the point on the object in a predetermined three-dimensional (absolute) coordinate system, for example, a three-dimensional coordinate system aligned with the direction of gravity (for example, a three-dimensional coordinate system with the direction of gravity as one axis).

[0035] The normal direction of a point on an object means the normal direction of the object surface on which the point exists.

[0036] In the point information generating unit 32, the normal direction of a point on an object is calculated, for example, as follows.

[0037] The point information generator 32 converts a point cloud of points on an object into a TSDF (truncated signed distance function), which is a three-dimensional volume representation. TSDF is described, for example, in B. Curless and M. Levoy, "A volumetric method for building complex models from range images," in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH '96, pages 303-312, New York, NY, USA, 1996. ACM.

[0038] The point information generation unit 32 applies the marching cubes method to the TSDF to generate a mesh (information) that represents an object. The marching cubes method is described, for example, in "Marching Cubes: A High Resolution 3D Surface Construction Algorithm" by William E. Lorensen and Harvey E. Cline in "Computer Graphics," Vol. 21, No. 4, July 1987.

[0039] Each face of a mesh representing an object is, for example, a (plane of) a triangle consisting of three vertices, and the point information generation unit 32 calculates the coordinates of the vertices of each face as the coordinates of a point on the object. Furthermore, the point information generation unit 32 calculates the normal direction of the face as the normal direction of the vertex of that face (point on the object).

[0040] The method for calculating the normal direction of a point on an object is not limited to the method using a mesh as described above, and any method can be used. For example, the normal of each point can be calculated using SVD or PCA from a point cloud of points on an object whose coordinates in a three-dimensional coordinate system are known. SVD and PCA are described in, for example, Krzysztof Jordan and Philippos Mordohai: A Quantitative Evaluation of Surface Normal Estimation in Point Clouds, IROS 2014.

[0041] The point information generating unit 32 supplies the point information to the plane estimating unit 33 together with the direction of gravity from the self-position estimating unit 31 .

[0042] The plane estimation unit 33 uses the point information and gravity direction from the point information generation unit 32 to estimate a specific plane that is a specific specified direction plane among the specified direction planes whose normal direction is in a specified direction among the planes that are the surfaces of real-world objects photographed by the camera 21.

[0043] For example, the plane estimation unit 33 estimates a horizontal plane whose normal direction is parallel to the direction of gravity as a predetermined direction plane, and estimates the floor surface of the real world as a specific plane among the horizontal planes.

[0044] In the information processing device 12, for example, an image of the floor surface of the virtual world (an image for VR or AR) is generated in accordance with the floor surface of the real world as the specific surface estimated by the plane estimation unit 33, and is displayed on the display unit 24.

[0045] The plane estimation unit 33 can estimate horizontal planes other than the floor surface of the real world, such as the ceiling surface of the real world.

[0046] Furthermore, the plane estimation unit 33 can estimate not only one horizontal plane, the floor plane, but also multiple horizontal planes. For example, two horizontal planes, the floor plane and the ceiling plane, can be estimated.

[0047] Furthermore, the plane estimation unit 33 can estimate a specific plane by using a plane other than a horizontal plane, for example, a plane whose normal direction forms a predetermined angle other than 0 degrees with the direction of gravity, that is, for example, a plane whose normal direction is perpendicular to the direction of gravity, as a predetermined direction plane.

[0048] In this case, for example, among walls perpendicular to the floor in the real world, the left and right walls can be estimated as specific surfaces. The information processing device 12 can recognize the range between the left and right walls as the range in which the user can move left and right.

[0049] In the information processing system 10, a part or all of the information processing device 12 can be built into the HMD 11.

[0050] Furthermore, in order to enhance the sense of immersion in the virtual world, the HMD 11 can be configured to be opaque so that the user cannot see the real world around them while playing a game or the like.

[0051] The information processing system 10 having a non-transparent HMD 11 can be provided with a video see-through function that displays images of the real world captured by the camera 21 on the display unit 24 as if the images were being viewed directly.

[0052] By displaying only the image of the virtual world generated by the information processing device 12 on the display unit 24, the information processing system 10 can function as a VR system that provides VR.

[0053] By superimposing an image of the virtual world generated by the information processing device 12 on an image of the real world captured by the camera 21 while matching it with the floor, ceiling, and wall surfaces estimated by the plane estimation unit 33 and displaying it on the display unit 24, the information processing system 10 can function as an AR system that provides AR.

[0054] Furthermore, the information processing system 10 may be configured by providing a wearable device other than the HMD 11 that has an image capturing function and an image display function, such as so-called smart glasses.

[0055] <Configuration example of the plane estimation unit 33>

[0056] FIG. 2 is a block diagram showing an example of the configuration of the plane estimation unit 33 in FIG.

[0057] The plane estimation unit 33 includes a point detection unit 41 (on a plane in a predetermined direction), a point detection unit 42 (on a specific surface), and a plane parameter calculation unit 43.

[0058] The point detection unit 41 receives the point information and the direction of gravity from the point information generation unit 32.

[0059] The point detection unit 41 functions as a first point detection unit that detects vertices (first points) on a predetermined direction plane whose normal direction is in a predetermined direction with respect to the direction of gravity from the vertices of each face of the mesh as points on the object included in the point information. The predetermined direction plane is a plane parallel to the specific plane to be estimated by the plane estimation unit 33.

[0060] For example, the point detection unit 41 detects vertices on a horizontal plane, which is a plane whose normal direction coincides (or can be considered to coincide) with a direction parallel to the direction of gravity.

[0061] The point detection unit 41 supplies the vertices on the predetermined direction plane to the point detection unit 42 .

[0062] The point detection unit 42 functions as a second point detection unit that detects, from the vertices on the plane in the predetermined direction detected by the point detection unit 41, a vertex (second point) on a specific surface that is a specific plane in the predetermined direction.

[0063] For example, if the vertex on the specified direction plane supplied from the point detection unit 41 to the point detection unit 42 is a vertex on a horizontal plane as described above, the point detection unit 42 detects from the vertex on the horizontal plane a specific horizontal plane as a specific plane, for example, a point (vertex) on the floor surface that is the estimation target of the plane estimation unit 33.

[0064] The point detection unit 42 supplies the vertices on the specific surface to the plane parameter calculation unit 43 .

[0065] The plane parameter calculation unit 43 functions as a specific plane detection unit that detects a specific plane by using the vertices on the specific plane supplied from the point detection unit 42 to calculate plane parameters that define the specific plane, i.e., model parameters that define the model of the plane as the specific plane.

[0066] Since the specific surface is a plane, it can be defined by, for example, the normal direction of the plane and the distance of the plane from the origin of the three-dimensional coordinate system that defines the point information. Therefore, the plane parameters that define the specific surface can be the normal direction of the plane as the specific surface and the distance of the plane from the origin of the three-dimensional coordinate system.

[0067] In the information processing device 12, the point detection unit 41 detects vertices on a predetermined direction plane whose normal direction is in a predetermined direction, and since the specific plane is one of the predetermined direction planes, the normal direction of the specific plane is known.

[0068] For example, if the specific plane is a horizontal plane such as a floor, the normal direction of the horizontal plane is, for example, the opposite direction to the direction of gravity, which is known in the information processing system 10 that calculates the direction of gravity. Therefore, the specific plane can be defined only by the distance from the origin of the three-dimensional coordinate system of the specific plane, and the distance from the origin of the three-dimensional coordinate system of the specific plane can be used as the plane parameter that defines the specific plane.

[0069] For example, when a three-dimensional coordinate system is adopted in which the direction of gravity is one axis as explained in Figure 1, the distance from the origin of the three-dimensional coordinate system to the horizontal plane can be called the height. Therefore, the distance from the origin of the three-dimensional coordinate system to the horizontal plane is also called the height.

[0070] For ease of explanation, the horizontal plane will be used as the predetermined direction plane, and the floor surface of the horizontal plane will be used as the specific plane.

[0071] Furthermore, for example, it is assumed that a user wears the HMD 11 on his / her head and uses the information processing system 10 in a room where a table and a sofa are placed on the floor.

[0072] In this case, the point information generating unit 32 generates point information on the surfaces of the table, sofa, and floor, and supplies the information to the point detecting unit 41 together with the direction of gravity.

[0073] The point detection unit 41 detects vertices (estimated to be) on a horizontal plane whose normal direction is along the direction of gravity from the vertices of each face of the mesh as points on the object included in the point information, and supplies them to the point detection unit 42.

[0074] For example, the normal direction of the vertex is represented by vector n, and the direction of gravity is represented by vector g. The point detection unit 41 detects, for example, a vertex that satisfies formula (1) as a vertex on a horizontal plane.

[0075] n·(-g) / (|n||-g|) > THplane ···(1)

[0076] THplane is a threshold value, and according to equation (1), if the cosine of the angle between vector n and vector -g is greater than the threshold value THplane (greater than or equal to the threshold value THplane), the normal direction is considered to be along the direction of gravity, and the vertex having that normal direction is detected as a vertex on a horizontal plane.

[0077] In the point detection unit 41, for example, points on the table top, points on the sofa seat, and points on the floor surface are detected as points on a horizontal plane, and are supplied to the point detection unit .

[0078] The point detection unit 42 detects vertices that are (presumably) on the floor surface as the specific surface from the vertices on the horizontal plane supplied from the point detection unit 41, and supplies the detected vertices to the plane parameter calculation unit 43.

[0079] For example, the point detection unit 42 clusters the vertices on the horizontal plane from the point detection unit 41, and detects the vertices clustered into a cluster that is estimated to be points on the floor surface as vertices on the floor surface.

[0080] The plane parameter calculation unit 43 detects the floor surface by calculating plane parameters, which are model parameters that define the floor surface model, using the vertices on the floor surface supplied from the point detection unit 42, in this case, for example, the height of the floor surface.

[0081] FIG. 3 is a diagram illustrating a model of a horizontal plane including a floor surface.

[0082] If the height (distance) from the origin (coordinate system origin) of the three-dimensional coordinate system that defines the point information is represented by d, and the position of a point on a horizontal plane including the floor is represented by coordinate x, then the model of the horizontal plane including the floor is expressed by equation (2).

[0083] (-g)·x - d = 0 ···(2)

[0084] Since the direction of gravity g is known, the plane parameter calculation unit 43 calculates the height d as the plane parameter of the floor surface (model).

[0085] As described above, in the information processing system 10, the point detection unit 41 uses point information including the coordinates and normal direction of a point on an object to detect a point on a predetermined direction plane whose normal direction is in a predetermined direction, such as a horizontal plane whose normal direction is parallel to the direction of gravity.

[0086] Then, the point detection unit 42 detects points on a specific surface, which is a specific plane in the predetermined direction, such as a floor surface, from points on the plane in the predetermined direction.

[0087] Therefore, the points that the point detection unit 42 targets for detecting points on a specific surface such as a floor surface (points detected by the point detection unit 41) are (almost) only points on a plane in a specified direction, and (almost) do not include points that are not on a plane in a specified direction.

[0088] That is, in the point detection unit 42, for example, points on the floor surface are detected from among points on a (substantially) horizontal plane.

[0089] Therefore, it is possible to prevent points on the floor surface from being obscured by other points, including points on planes other than the horizontal plane, and thus preventing the floor surface from being detected accurately.

[0090] As a result, floors and other arbitrary flat surfaces can be detected robustly.

[0091] As described above, by detecting the floor surface, for example, in a VR system, the user does not have to manually specify the floor surface in order to display an image of the floor surface on which the user can move, thereby reducing the burden on the user.

[0092] The detection of points on the floor surface from among points on a horizontal plane by the point detection unit 42 will be described in detail below.

[0093] As a method for detecting points on the floor surface from points on a horizontal plane, for example, there is a method using RANSAC (Random Sample Consensus).

[0094] RANSAC is described, for example, in Martin A. Fischler & Robert C. Bolles (1981), "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography", Comm. ACM. 24 (6): 381-395.

[0095] Also, a technique for calculating the plane parameters of each plane (model) from a plurality of planes not limited to the horizontal plane by repeatedly performing model fitting by RANSAC is described in Ruwen Schnabel, Roland Wahl, and Reinhard Klein. Efficient ransac for point-cloud shape detection. In Computer graphics forum, volume 26, pages 214?226. Wiley Online Library, 2007. By using this technique, points on the floor can be detected from the points on the horizontal plane.

[0096] <Method for Detecting Points on the Floor Using RANSAC>

[0097] FIG. 4 is a diagram for explaining RANSAC.

[0098] In RANSAC, when fitting data including outliers (outlier points) to a model such as a straight line or a plane, the model parameters of the model representing the data are calculated by fitting the data to the model while suppressing the influence of the outliers.

[0099] For example, when calculating the line parameters as the model parameters of the straight line representing a point group of N points scattered on a two-dimensional plane and including outliers, the process shown in FIG. 4 is performed in RANSAC. The line parameters of the straight line are, for example, the slope and intercept of the straight line.

[0100] First, in step S11, two points are randomly sampled from the point group of N points, and the line parameters of the straight line passing through the two points are calculated as the model parameters of the temporary model of the straight line from the coordinates of the two points.

[0101] In step S12, points whose distance from the straight line as the temporary model in the point group of N points is less than or equal to the threshold δ (less than) are calculated as inlier points, that is, points that fit the temporary model.

[0102] In step S13, it is determined whether the number of repetitions of the loop processing of steps S11 to S13 has reached the upper limit value UL, and if it is determined that the number of repetitions has not reached the upper limit value UL, the processing returns to step S11, and the loop processing of steps S11 to S13 is repeated.

[0103] Then, when the number of times the loop processing of steps S11 to S13 is repeated reaches the upper limit value UL, that is, when an inlier point is calculated for each line as a provisional model passing through two points in the randomly selected UL pair, the processing proceeds from step S13 to step S14.

[0104] In step S14, from the lines as a provisional model passing through two points of the UL pair, the line with the largest number of inlier points is selected as the dominant line (straight line) representing the dominant point group in the point group of N points, that is, the line that appropriately represents the point group of N points (the line that is consistent with the point group) and is calculated.

[0105] When calculating a line model (with line parameters as model parameters) that represents a point group of N points using RANSAC, all combinations of two points out of the N points are calculated. N For C2, it is ideal to calculate the inlier point.

[0106] However, all combinations of two points out of N points N When calculating the inlier points for C2, the number of combinations of two points increases as the number of points N increases. N The larger the C2, the larger the amount of calculation. As a result, processing (calculating the line model representing the point cloud) takes time.

[0107] All combinations of two points out of N points N When calculating the inlier points for C2, the amount of calculation required to calculate a line as a provisional model for a combination of two points is on the order of O(N).

[0108] All combinations of two points out of N points N C2 is approximately O(N 2 ), so there are all combinations of two points out of N points. N For C2, the final computational complexity for calculating the inlier points is approximately O(N 3 ) order.

[0109] Therefore, as explained in FIG. 4, the number of times the loop processing of steps S11 to S13 is repeated is limited to an upper limit value UL, that is, the number of pairs of two points selected from N points is limited to UL pairs, thereby reducing the amount of calculation.

[0110] However, if the point cloud of N points contains many outliers, limiting the number of pairs of two points selected from the N points to a small number UL through censoring increases the likelihood that the pairs of two points will contain outliers. As a result, it becomes more likely that an appropriate model, i.e., a linear model that properly represents the point cloud of N points, cannot be obtained.

[0111] Random numbers are used for random sampling, which randomly selects two points from a point cloud of N points. Random sampling behaves probabilistically, and it is necessary to verify whether the line model obtained by such random sampling properly represents the point cloud of N points.

[0112] Model validation is performed by randomly sampling two points using different random numbers, calculating multiple line models, and comparing the average of the multiple line models with the model to be validated. Therefore, model validation requires a significant cost that exceeds the cost of calculating the model to be validated.

[0113] FIG. 5 is a diagram for explaining the processing of the point detection unit 42 when detecting points on a floor surface using RANSAC.

[0114] 5, as in the case of Fig. 2, a user wears an HMD 11 on his / her head in a room where a table and a sofa are placed on the floor, and uses the information processing system 10. Furthermore, points on the table top, points on the sofa seat, and points on the floor are supplied from the point detection unit 41 to the point detection unit 42 as points on a horizontal plane.

[0115] In step S21, the point detection unit 42 uses RANSAC to calculate the height d of equation (2) as a plane parameter (model parameter) of the dominant plane, which is the horizontal plane that represents the dominant point group at the point being processed among the points on the horizontal plane (vertices of the mesh face) from the point detection unit 41, i.e., the horizontal plane that has the most inlier points among the points being processed.

[0116] Here, when this is performed first in point detection unit 42, all of the points on the horizontal plane supplied from point detection unit 41 to point detection unit 42 are selected as points to be processed. Hereinafter, the points on the horizontal plane supplied from point detection unit 41 to point detection unit 42 are also referred to as the initial points to be processed.

[0117] In RANSAC, for example, one point is sampled (randomly sampled) from the point to be processed. Furthermore, a horizontal plane passing through the point obtained by random sampling is calculated as a tentative model, and the point to be processed that is at a distance within a threshold from the horizontal plane as the tentative model is calculated as an inlier point of the tentative model.

[0118] The above process is repeated a number of times equal to the upper limit UL, and UL temporary models and the inlier points of each temporary model are calculated. Then, of the UL temporary models, the temporary model with the largest number of inlier points is calculated as the dominant surface.

[0119] In step S22, the point detection unit 42 removes the inlier points of the dominant surface calculated immediately before (in the immediately preceding step S21) from the point (vertex) currently being processed, and sets the remaining points as new points to be processed.

[0120] In step S23, the point detection unit 42 determines whether the number of points to be newly processed (points remaining after removing the inlier points of the dominant surface in step S22) is equal to or less than the threshold value (less than the threshold value).

[0121] If the number of new processing target points is not equal to or less than the threshold value, the process returns to step S21, and the loop process of steps S21 to S23 is repeated for the new processing target points.

[0122] For example, it is assumed that the points to be initially processed include, in order of frequency, points on the table top, points on the sofa seat, and points on the floor.

[0123] In this case, in the first loop of the loop processing of steps S21 to S23, the horizontal plane as the table top, which has the largest number of points, is calculated as the dominant surface. Then, points on the table top that are inlier points of the dominant surface are excluded from the points to be processed.

[0124] By excluding points on the table top from the points to be processed, in the second loop, the horizontal plane as the sofa seat, which has the second largest number of points, is calculated as the dominating surface. Then, points on the sofa seat, which are inlier points of the dominating surface, are excluded from the points to be processed.

[0125] By excluding points on the table top and sofa seat from the points to be processed, the horizontal plane as the floor surface with the fewest points is calculated as the dominating surface in the third loop. Then, points on the floor surface that are inlier points of the dominating surface are excluded from the points to be processed.

[0126] Suppose the initial points to be processed are only points on the table top, points on the sofa seat, and points on the floor. In this case, by excluding the points on the table top, points on the sofa seat, and points on the floor from the points to be processed, the number of points to be processed becomes 0, which is below the threshold.

[0127] When the number of points to be processed becomes equal to or less than the threshold value, the process proceeds from step S23 to step S24.

[0128] Here, according to the loop processing of steps S21 to S23, the first processing target point can be considered to be clustered into points on the table top, points on the sofa seat, and points on the floor surface. Therefore, the loop processing of steps S21 to S23 is a process of clustering vertices on a horizontal plane.

[0129] In step S24, the point detection unit 42 selects the horizontal plane having the lowest height d as a plane parameter from among the horizontal planes as the dominant planes calculated in the loop processing of steps S21 to S23 as the floor surface, and detects the inlier point of that horizontal plane as a point on the floor surface.

[0130] As explained in Figure 4, RANSAC may increase the possibility of not obtaining an appropriate model (here, a model of the horizontal plane) due to processing termination. Furthermore, RANSAC requires a large cost for verification due to random sampling.

[0131] Therefore, instead of RANSAC, a voting method can be used to detect points on the floor surface from points on the horizontal plane.

[0132] In the voting method, multiple provisional models (model parameters) are prepared in advance. Each data item is voted for the provisional model that best fits it, and the model that is most appropriate for the entire data set is selected from the multiple provisional models according to the number of votes for each provisional model.

[0133] The voting method prevents the possibility of not obtaining a good model due to censoring, and avoids the need for model validation that would be required due to random sampling.

[0134] In the voting method, when the number of model parameters of the temporary model is large, for example, when the dimensionality of the model parameters is large or the range of values that the model parameters can take is vast, the number of temporarily prepared models becomes extremely large.

[0135] When the number of temporarily prepared models is extremely large, the memory capacity required for storing the number of votes and the like increases, and the cost of voting, for example, the amount of computation required to detect the model that best fits the data, also increases.

[0136] However, for the information processing apparatus 12, the temporary models to be prepared in advance are planar models, and the model parameters (plane parameters) of the planar models are only the distance from the origin of the plane, as described in FIG. 2. For example, the model parameters of a horizontal plane are only the one-dimensional height (distance from the origin).

[0137] Also, as a use case of the information processing system 10, a use case where the range of values that the distance from the origin of the plane can take as the plane model parameter becomes vast is unlikely to occur. <000047l><00z0472>In particular, when detecting the floor surface as a horizontal plane, the range of values that the height can take as the model parameter of the floor surface is, for example, up to several meters from the position of the HMD 11 (of the camera 21) when the user is wearing the HMD 11.

[0139] Therefore, even when using the voting method, the required memory capacity and the cost of voting can be suppressed.

[0140] <Method for Detecting Points on the Floor Surface Using the Voting Method>

[0141] FIG. 6 is a diagram for explaining the processing of the point detection unit 42 when detecting points on the floor surface using the voting method instead of RANSAC.

[0142] That is, FIG. 6 is a diagram for explaining the processing of the point detection unit 42 that employs the first detection method of detecting points on the floor surface using the voting method.

[0143] In FIG. 6, the processing of step S21 in FIG. 5, in which the height as a plane parameter of the horizontal plane expressing the dominant point group at the point to be processed is calculated by RANSAC, is performed by the voting method.

[0144] In other words, in RANSAC, a horizontal plane passing through one point obtained by random sampling from the point to be processed is calculated as a tentative model (hypothesis), and points that are within a threshold distance from the horizontal plane as the tentative model are calculated as inlier points of the tentative model.This process is repeated up to an upper limit (number of times) UL.

[0145] For simplicity, it is assumed that the number of points to be processed is N, and that instead of random sampling, each of the N points to be processed is sampled. In this case, for each point to be processed, a horizontal plane passing through that point is calculated as a provisional model, and inlier points of the provisional model are calculated. That is, the calculation of the provisional model and the calculation of the inlier points of the provisional model are performed N times.

[0146] Therefore, in RANSAC, for N processing target points, N provisional models (their heights as model parameters) and the number of inlier points for each of the N provisional models are calculated.

[0147] As described above, calculating N hypothetical models and the number of inlier points for each of the N hypothetical models for the N points to be processed can be understood as generating a histogram for the N points to be processed, with the heights of the horizontal planes as model parameters of the N hypothetical models as classes (bins) and the number of inlier points of the hypothetical models as frequencies.

[0148] Hereinafter, a histogram in which model parameters are used as classes and the number of inlier points of a model defined by the model parameters is used as a frequency will also be referred to as an inlier point histogram.

[0149] In RANSAC, the height (hypothesis) of the horizontal plane is calculated as a model parameter of a tentative model from the points to be processed, and an inlier point histogram is generated using the height of the horizontal plane as a class.

[0150] Then, in the inlier point histogram, the maximum value of the number of inlier points as a frequency is detected, and the height of the horizontal plane as the frequency class of that maximum value is calculated as the height (best hypothesis) of the dominant surface that represents the dominant point group at the point being processed.

[0151] For example, suppose the points being processed contain the most points on a tabletop. In this case, in the inlier point histogram, the inlier point (frequency) of the height (class) of a horizontal plane generated from one of the points on the tabletop and passing through that point is the maximum value. The height of that horizontal plane is then calculated as the height (best hypothesis) of the dominant surface that represents the dominant point group at the point being processed.

[0152] On the other hand, in the voting method, horizontal planes (with heights as model parameters) are prepared in advance as multiple provisional models, and each of the N points to be processed is voted for the provisional model that best fits the multiple provisional models prepared in advance.

[0153] That is, for each point to be processed, the horizontal plane with the closest distance is detected from among multiple horizontal planes that serve as virtual models prepared in advance, and the point to be processed is counted as the inlier point of the horizontal plane with the closest distance.

[0154] Therefore, if the number of pre-prepared provisional models is M, then by voting for N points of the points to be processed, an inlier point histogram is generated for the N points to be processed, with the height of the horizontal plane as a model parameter of the M pre-prepared provisional models as the class and the number of inlier points of the provisional models as the frequency.

[0155] After generating the inlier point histogram, the maximum value of the number of inlier points as a frequency can be detected in the inlier point histogram, as in the case of RANSAC. Furthermore, the height of the horizontal plane as the class of the frequency of the maximum value can be calculated as the height (best hypothesis) of the dominant plane that represents the dominant point group at the point to be processed.

[0156] In RANSAC, a horizontal plane passing through one of the points to be processed is calculated as a tentative model, and the process of calculating the inlier points of the tentative model is referred to as the inlier point calculation process for one point.

[0157] In RANSAC, the inlier point calculation process for one point is performed for each of the N points to be processed, thereby generating an inlier point histogram.

[0158] The computational complexity of the inlier point calculation process for one point is on the order of O(N). Also, the computational complexity of performing the process for each of the N points to be processed is on the order of O(N).

[0159] Therefore, the computational complexity required to generate the inlier point histogram in RANSAC is O(N 2 ) order.

[0160] On the other hand, in the voting method, an inlier point histogram is generated by voting for each of the N points to be processed to the tentative model that best fits one of M tentative models prepared in advance.

[0161] The computational complexity of voting for each of the N points to be processed to the most fitting tentative model out of M tentative models prepared in advance is on the order of O(N).

[0162] Therefore, in the voting method, the amount of calculation required to generate the inlier point histogram is on the order of O(N).

[0163] From the above, the amount of calculation can be reduced by using the voting method instead of RANSAC in the processing of step S21 in Figure 5, which calculates the height as a plane parameter of the horizontal plane representing the dominant point group at the point to be processed.

[0164] When the processing of step S21 in FIG. 5 is performed using the voting method instead of RANSAC, it is not necessary to regenerate the inlier point histogram in step S21 which is repeatedly performed in the loop processing of steps S21 to S23.

[0165] That is, in step S21, which is performed using the voting method, an inlier point histogram is first generated, and then, by simply setting the frequency of the height of the horizontal plane as the dominant surface calculated using the inlier point histogram to 0, an inlier point histogram for the point to be processed after the inlier points of the dominant surface have been removed can be obtained.

[0166] Incidentally, the loop processing of steps S21 to S23 in Figure 5 repeatedly calculates the horizontal plane (or its height) that represents the dominant point group at the point being processed, excluding the inlier points of the horizontal plane calculated immediately before from the point being processed.

[0167] Therefore, the loop processing of steps S21 to S23 is repeated roughly the same number of times as the number of horizontal planes (hereinafter also referred to as actual horizontal planes) on which the processing target points supplied from point detection unit 41 to point detection unit 42 exist.

[0168] The number of actual horizontal surfaces is unknown, and if there are many horizontal surfaces in the environment in which the information processing system 10 is used, the number of actual horizontal surfaces increases, and the number of times the loop processing of steps S21 to S23 in Fig. 5 is repeated also increases. If the number of times the loop processing is repeated increases, the amount of calculation increases, and it may become difficult to detect the floor surface in real time.

[0169] Therefore, the point detection unit 42 can employ a second detection method that uses a voting method to detect points on the floor surface. The second detection method does not require loop processing the number of times that is affected by the number of actual horizontal planes, etc., and enables floor surface detection in real time with a constant, lightweight calculation load.

[0170] FIG. 7 is a diagram illustrating the processing of the point detection unit 42 that employs the second detection method of detecting points on the floor surface using the voting method.

[0171] In the point detection unit 42 employing the second detection method, an inlier point histogram is generated in the same manner as in the first detection method. Then, in the point detection unit 42, the peak class in which the frequency reaches its first peak when viewed from the lower side of the horizontal plane height, which is the class, is detected as the class of floor height.

[0172] FIG. 8 is a block diagram showing an example of the configuration of a point detection unit 42 that employs a second detection method that detects points on the floor surface using a voting method.

[0173] 8, the point detection unit 42 includes a histogram generation unit 51, a peak detection unit 52, and an inlier point extraction unit 53.

[0174] The histogram generating unit 51 generates an inlier point histogram for the points (vertices of the face) to be processed from the point detecting unit 41 by the voting method, and supplies the histogram to the peak detecting unit 52 .

[0175] The inlier point histogram is constructed using classes based on the height of the horizontal plane.

[0176] If the width of the class is represented by Δd and the mth class from the lowest height is represented by class m (m=1, 2, ..., M), the point to be processed is voted to class m that satisfies equation (3) according to the height d of the point, thereby generating an inlier point histogram.

[0177] dmin + (m-1)Δd ≦ d < dmin + mΔd ···(3)

[0178] dmin represents the minimum height of the horizontal plane.

[0179] For example, suppose that the floor surface (points on the floor surface) is detected while the HMD 11 as a wearable device is worn on the head of the user, and therefore the HMD 11 is always located higher than the floor surface. In this case, the minimum value dmin of the height of the horizontal plane can be dmin = dhmd - h, which is lower than the height dhmd of the camera position of the camera 21 obtained by self-position estimation by a certain height (value) h set according to the height of the head as a part of the user wearing the HMD 11.

[0180] As described above, if the HMD 11 is worn on the user's head, the certain height h can be, for example, about 2 m.

[0181] The class width Δd can be set according to, for example, the accuracy of detecting the floor surface (height) and the accuracy of the IMU 22 and the distance measurement sensor 23.

[0182] For example, the height dhmd of the camera position of the camera 21 is used as the maximum value of the height of the horizontal plane as a class of the inlier point histogram. In this case, if 2 m is used as the fixed height h and 1 cm is used as the class width Δd, the number of classes M of the inlier point histogram will be 200.

[0183] As described above, setting each class m expressed by equation (3) corresponds to preparing in advance a plurality of horizontal planes (heights as model parameters) as provisional models. Also, class m (median height) expressed by equation (3) corresponds to (the height of the horizontal plane as) the provisional model.

[0184] The inlier point histogram is generated by voting the point to be processed into a class m that satisfies equation (3) according to the height d of the point.

[0185] In the inlier point histogram generated by the voting method, the median height of class m, d[m] = dmin + (m - 1 / 2)Δd, corresponds to the height of the horizontal plane as a model parameter of the hypothetical model calculated from the points to be processed obtained by sampling when the inlier point histogram is generated by RANSAC.

[0186] In addition, in the inlier point histogram generated by the voting method, the processing points voted for class m (processing points with height d that satisfy equation (3)) correspond to processing points that become inlier points when the inlier point histogram is generated by RANSAC.

[0187] The peak detection unit 52 detects, in the inlier point histogram from the histogram generation unit 51, the peak class to which the peak of the frequency (the number of votes for the point to be processed) belongs.

[0188] Now, the detection target of the point detection unit 42 is a point on the floor surface, and is estimated to be the point on the horizontal plane with the lowest height among the points to be processed. Therefore, the peak detection unit 52 can detect frequency peaks from the lowest height class toward the highest height classes. In this case, the peak class to which the first detected frequency peak (first peak in frequency) belongs can be detected as the class to which the point on the floor surface belongs (the class to which the point on the floor surface was voted).

[0189] In addition, the peak detection unit 52 takes into account frequency errors caused by the accuracy of the IMU 22 and the distance measurement sensor 23, etc., and can consider frequencies below (less than) the threshold THnoise as noise and exclude them from frequency peak detection targets.

[0190] That is, peak classes can be detected by targeting only classes whose frequencies exceed the threshold value THnoise in the inlier point histogram (classes equal to or greater than the threshold value THnoise).

[0191] In addition, in the peak detection unit 52, in order to suppress noise, the inlier point histogram (frequency) is filtered using an LPF (low pass filter), and peak classes can be detected from the inlier point histogram after the filtering process.

[0192] As described above, when the peak detection unit 52 detects peak classes from the inlier point histogram after filtering and only classes whose frequencies exceed the threshold value THnoise, the frequency LPF(bin[m]) that first satisfies equation (4), from the lowest height class to the highest height class, is detected as the first peak of frequency.

[0193] LPF(bin[m]) > LPF(bin[m+1]), LPF(bin[m]) > THnoise ···(4)

[0194] bin[m] represents the frequency of class m, and LPF(bin[m]) represents the frequency of class m after filtering the inlier point histogram (frequency).

[0195] When a 3-tap (1,2,1) / 4 FIR (finite impulse response) filter is used as the LPF that performs filtering on the inlier point histogram, for example, LPF(bin[m]) is expressed by equation (5).

[0196] LPF(bin[m]) = (bin[m-1] + 2*bin[m] + bin[m+1]) / 4 ···(5)

[0197] According to equation (4), in the inlier point histogram after filtering, the frequency LPF(bin[m]) of the class just before the frequency LPF(bin[m+1]) where the frequency first decreases from the lowest height class to the highest height class, and which exceeds the threshold THnoise, is detected as the first peak of frequency.

[0198] The peak detection unit 52 detects the peak class to which the first peak of the frequency belongs, and supplies it to the inlier point extraction unit 53 .

[0199] The inlier point extraction unit 53 detects (extracts) the points to be processed that belong to the peak class from the peak detection unit 52 in the inlier point histogram as inlier points of a horizontal plane, i.e., a floor surface having the height of the peak class, and supplies them to the plane parameter calculation unit 43.

[0200] In the point detection unit 42, in voting of processing target points for generating an inlier point histogram, (the coordinates of) processing target points voted for each class are stored in association with the class to which the processing target points are voted.

[0201] The inlier point extraction unit 53 detects the processing target point associated with the peak class as an inlier point of a horizontal plane serving as a floor surface having the height of the peak class, and supplies the detected point to the plane parameter calculation unit 43 .

[0202] The plane parameter calculation unit 43 detects the floor surface by calculating the height d of the floor surface as a plane parameter that defines the floor surface model using the inlier points (vertices on the floor surface) of the floor surface from the inlier point extraction unit 53.

[0203] The height d of the floor surface can be calculated using the inlier points of the floor surface according to, for example, equation (6).

[0204] d = 1 / N'*Σ((-g)·x[i]) ···(6)

[0205] N' represents the number of inlier points on the floor, and x[i] represents the coordinates of the i-th inlier point among the N' inlier points on the floor. Σ represents the summation by changing i to an integer from 1 to N'.

[0206] The point detection unit 42 can stably detect points on the floor surface (inlier points on the floor surface) even when, for example, the points to be processed include multiple points on a horizontal plane or when the area of the floor surface captured in the image captured by the camera 21 is small.

[0207] In addition, if the points to be processed include points on multiple horizontal planes, the point detection unit 42 can detect all of the frequency peaks in the inlier point histogram, and for the peak class to which each peak belongs, the points to be processed that belong to that peak class can be detected as inlier points on the horizontal plane at the height of that peak class.

[0208] In this case, the plane parameter calculation unit 43 calculates the height of the horizontal plane using the inlier points that belong to the peak class to which each peak belongs, thereby making it possible to detect the height of each of the multiple horizontal planes.

[0209] By detecting the height of each of multiple horizontal planes, for example, in AR applications, it is possible to accurately place virtual world objects (images) on multiple horizontal planes of objects that exist in the real world, such as the top of a table, the seat of a sofa, or the floor, and to accurately move virtual world objects along the horizontal planes.

[0210] Note that techniques capable of detecting the floor surface from an image that shows multiple horizontal surfaces include the technique described in Patent Document 1 and the techniques described in Japanese Patent Laid-Open Nos. 2003-269937 and 2005-092820.

[0211] However, as mentioned above, with the technology described in Patent Document 1, points on the floor surface may be hidden by points on other planes, making it impossible to accurately detect the floor surface. Furthermore, with the technology described in Patent Document 1, frequency peaks are not detected in the inlier point histogram as with the present technology.

[0212] Furthermore, the techniques described in JP 2003-269937 A and JP 2005-092820 A require information on the robot's contact surface to determine which of multiple horizontal surfaces is the floor. To do this, it is necessary to capture an image showing the contact area between the robot's feet and the floor and recognize the robot's feet. This technique does not require the robot's feet to be recognized.

[0213] <Processing of information processing device 12>

[0214] FIG. 9 is a flowchart illustrating the processing of the information processing device 12 in FIG. 1 when detecting the floor surface.

[0215] For example, in response to a user's operation, the information processing device 12 sets the operation mode to a detection mode for detecting the floor surface, and starts detecting the floor surface.

[0216] In step S51, the information processing device 12 acquires an image, acceleration, angular velocity, and distance image from the HMD 11, and the process proceeds to step S52.

[0217] In step S52, the self-position estimation unit 31 estimates its own position using the image, acceleration, and angular velocity acquired by the information processing device 12 from the HMD 11. The self-position estimation unit 31 calculates the camera position and attitude of the camera 11 and the direction of gravity through the self-position estimation, and supplies these to the point information generation unit 32, and the process proceeds from step S52 to step S53.

[0218] In step S53, the camera position and attitude, the direction of gravity, and the distance image acquired by the information processing device 12 from the HMD 11 are used to generate point information including the coordinates and normal direction of each point on the object whose distance has been sensed by the distance measuring sensor 23. The point information generation unit 32 supplies the point information and the direction of gravity to the plane estimation unit 33, and the process proceeds from step S53 to step S54.

[0219] In step S54, in the plane estimation unit 33 (FIG. 2), the point detection unit 41 uses the point information and the direction of gravity from the point information generation unit 32 to detect a plane whose normal direction coincides with a direction parallel to the direction of gravity, i.e., a point (first point) on a horizontal plane, from (the coordinates of) the point on the object included in the point information. The point detection unit 41 supplies (the coordinates of) the point on the horizontal plane to the point detection unit 42, and the process proceeds from step S54 to step S55.

[0220] In step S55, the point detection unit 42 detects points on the floor surface from the points on the horizontal plane detected by the point detection unit 41, for example, by using a method using RANSAC (FIG. 5), a first detection method using a voting method (FIG. 6), or a second detection method using a voting method (FIG. 7). The point detection unit 42 supplies the points on the floor surface to the plane parameter calculation unit 43, and the process proceeds from step S55 to step S56.

[0221] In step S56, the plane parameter calculation unit 43 uses the points on the floor surface from the point detection unit 42 to calculate the height d of the floor surface according to equation (6), thereby detecting the floor surface, and the process proceeds to step S57.

[0222] In step S57, the information processing device 12 determines whether or not the user has performed an operation to end the detection mode, and if it determines that the user has not performed an operation to end the detection mode, the process returns to step S51, and the same process is repeated.

[0223] If it is determined in step S57 that the user has performed an operation to end the detection mode, the process ends.

[0224] <Other peak class detection methods>

[0225] FIG. 10 is a diagram showing a case where the peak class to which the first peak of the frequency of the inlier point histogram belongs is a class of height lower than the true height of the floor surface.

[0226] In the inlier point histogram, the first peak of frequency may appear in a lower height class than the true height class of the floor surface due to frequency noise (error) caused by the accuracy of the IMU 22 and the distance measurement sensor 23, etc.

[0227] Therefore, in the second detection method, as explained in Figures 7 and 8, when the peak class (first peak class) to which the first peak of frequency belongs when viewed from the lower horizontal plane height in the inlier point histogram is detected as the floor height class, a class (lower height class) in front of the class of the true floor height may become the first peak class, and may be erroneously detected as the floor height class.

[0228] Therefore, the point detector 42 can employ a peak class detection method other than the method of simply detecting the frequency peak in the inlier point histogram as a peak class detection method for detecting the peak class.

[0229] FIG. 11 is a diagram illustrating another peak class detection method.

[0230] In another peak bin detection method, after generating the inlier point histogram, the bins of the inlier point histogram are clustered according to frequency into clusters containing one or more consecutive bins that can be considered as bins of the same horizontal plane height.

[0231] Then, in each cluster, the maximum frequency is regarded as the peak frequency and detected as the peak class.

[0232] Another peak bin detection method can reduce the influence of frequency noise in the inlier point histogram.

[0233] In other words, with other peak class detection methods, when a frequency peak appears at a lower height class than the true height class of the floor surface due to the influence of frequency noise, it is possible to prevent the class of the frequency peak from being mistakenly detected as the floor height class.

[0234] For example, in FIG. 11, in the inlier point histogram, a frequency peak appears in a height class lower than the class of the true height of the floor surface.

[0235] In this case, if the peak class to which the first peak of frequency belongs, viewed from the lower side of the horizontal plane in the inlier point histogram, is detected as the class of floor height, a class of height lower than the true height of the floor will be erroneously detected as the class of floor height.

[0236] On the other hand, according to another peak tier detection method, the tiers of the inlier point histogram are clustered according to frequency into clusters containing one or more consecutive tiers that can be considered as tiers of the same horizontal height.

[0237] In FIG. 11, the classes of the inlier point histogram are clustered into three clusters, a first cluster (floor), a second cluster (sofa), and a third cluster (table), according to frequency.

[0238] The first cluster includes multiple consecutive classes that can be considered to be classes of floor height, the second cluster includes multiple consecutive classes that can be considered to be classes of sofa seat height, and the third cluster includes multiple consecutive classes that can be considered to be classes of table top height.

[0239] When detecting the floor surface, in another peak class detection method, the class with the highest frequency (the class with the maximum frequency) among the classes contained in the first cluster (floor), which is the first cluster when viewed from the lower horizontal plane height, is detected as the peak class in the inlier point histogram after class clustering.

[0240] Therefore, when a frequency peak appears in an inlier point histogram at a height class lower than the true height class of the floor surface, it is possible to prevent the lower height class from being mistakenly detected as the floor height class.

[0241] In addition, when another peak class detection method is adopted, the point detection unit 42 can detect the points to be processed that belong only to the peak class (the class with the maximum frequency) among the classes belonging to the first cluster when viewed from the lower height of the horizontal plane as points on the floor surface and supply them to the plane parameter calculation unit 43.

[0242] In addition, the point detection unit 42 can detect, as viewed from the lower side of the horizontal plane, points to be processed that belong to the peak class that belongs to the first cluster and points to be processed that belong to each of the other one or more classes, for example, points to be processed that belong to each class that belongs to the first cluster, as points on the floor surface and supply them to the plane parameter calculation unit 43.

[0243] As described above, according to another peak class detection method, the classes of the inlier point histogram are clustered into clusters containing one or more consecutive classes, and a clustered inlier point histogram is used. In each cluster, the maximum frequency is considered to be the peak frequency, and a peak class is detected, thereby suppressing the influence of frequency noise in the inlier point histogram.

[0244] Other peak class detection methods for clustering inlier point histogram classes include, for example, density-based spatial clustering of applications with noise (DBSCAN).

[0245] FIG. 12 is a diagram illustrating DBSCAN.

[0246] DBSCAN detects areas with a high density of observation points that are the subject of clustering, and detects these areas as clusters.

[0247] That is, DBSCAN sequentially focuses on observation points, and if there are more than the threshold minPoints of observation points within a radius ε of the observation point of interest, the observation point of interest is detected as a core point that constitutes a cluster. Then, for example, the area within the radius ε is defined as a core area, and the core areas of core points that overlap are detected as one cluster.

[0248] FIG. 13 is a diagram illustrating the clustering of classes of inlier point histograms using DBSCAN.

[0249] When using DBSCAN to cluster the classes of an inlier point histogram, the classes correspond to the observation points because they are the target of clustering. However, because the classes are arranged at equal intervals on the horizontal axis of the inlier point histogram in one dimension, there are no differences in density (sparseness or density) like there are with observation points.

[0250] Therefore, in this embodiment, the frequency of each class (the number of inlier points) corresponds to the density of the observation points (the number of observation points within a radius ε). In this case, classes whose frequency (the number of inlier points) is equal to or greater than a threshold minPts are detected as core classes, and core classes existing within ±ε are detected as classes belonging to the same cluster, thereby enabling clustering of classes in the inlier point histogram.

[0251] When detecting the floor surface, the point detection unit 42 scans the classes in the inlier point histogram after filtering, for example, from the lowest class to the highest class, and detects the first class m whose frequency LPF(bin[m]) satisfies equation (7) for all variables i = 1, 2, .., ε that count the number of classes, as the first class of the cluster.

[0252] LPF(bin[m]) > minPts and LPF(bin[mi]) < minPts (i = 1,2,..,ε) ···(7)

[0253] Thereafter, the point detector 42 continues scanning the classes and detects the first class m whose frequency LPF(bin[m]) satisfies equation (8) for all variables i=1, 2, . . . , ε as the last class of the cluster.

[0254] LPF(bin[m]) > minPts and LPF(bin[m+i]) < minPts (i = 1,2,..,ε) ···(8)

[0255] Then, the point detection unit 42 detects the first class to the last class of the cluster as (classes belonging to) the same cluster.

[0256] When the point detection unit 42 scans the classes from the lowest height class to the highest height class, it detects the class with the highest frequency in the first detected cluster as the peak class.

[0257] Then, the point detection unit 42 detects the processing target points belonging to the peak class as points on the floor surface (inlier points), and supplies them to the plane parameter calculation unit 43.

[0258] When detecting horizontal surfaces other than the floor, after the first cluster is detected, the detection of clusters continues toward higher height classes up to the highest class. Then, for each cluster, the class with the highest frequency is detected as the peak class.

[0259] <Description of the computer to which this technology is applied>

[0260] Next, the above-described series of processes can be performed by hardware or software. When the series of processes is performed by software, the programs that make up the software are installed on a general-purpose computer or the like.

[0261] FIG. 14 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0262] The program can be pre-recorded on the hard disk 905 or ROM 903 as a recording medium built into the computer.

[0263] Alternatively, the program can be stored (recorded) on a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as a so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0264] The program can be installed into the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 905. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0265] The computer includes a CPU (Central Processing Unit) 902 , to which an input / output interface 910 is connected via a bus 901 .

[0266] When a user inputs a command via an input / output interface 910 by operating an input unit 907, the CPU 902 executes a program stored in a read-only memory (ROM) 903 in accordance with the command. Alternatively, the CPU 902 loads a program stored on a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0267] As a result, the CPU 902 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 902 outputs the processing results from the output unit 906 via the input / output interface 910, or transmits them from the communication unit 908, or further records them on the hard disk 905, as necessary.

[0268] The input unit 907 is made up of a keyboard, a mouse, a microphone, etc. The output unit 906 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0269] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

[0270] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0271] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0272] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0273] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0274] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0275] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0276] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0277] The present technology can have the following configurations.

[0278] <1> a first point detection unit that detects a first point on a predetermined direction plane whose normal direction is a predetermined direction from a point on the object; a second point detection unit that detects a second point on a specific plane that is a specific predetermined direction plane from the first point; a specific surface detection unit that detects the specific surface using the second point; An information processing device comprising: <2> the predetermined direction plane is a horizontal plane whose normal direction is parallel to the direction of gravity, The first point detection unit detects the first point on the horizontal plane. <1> The information processing device described in <3> the specific surface is a floor surface, The second point detection unit detects the second point on the floor surface. <2> The information processing device described in <4> The second point detection unit generates a histogram for the first point, in which the height of the horizontal plane including the first point is used as a class, and detects the second point using the histogram. <3> The information processing device described in <5> The second point detection unit generates the histogram by setting a height that is lower than the height of the wearable device that is the target of self-position estimation by a value that is set according to the height of the part of the user on which the wearable device is worn as the minimum value of the height of the horizontal plane. <4> The information processing device described in <6> The second point detection unit performs filtering on the histogram using a low pass filter (LPF) and detects the second point using the histogram after the filtering. <4> or <5> The information processing device described in <7> the second point detection unit detects a peak class that is the first peak in the histogram when viewed from the lower side of the horizontal plane; The first point belonging to the peak class is detected as the second point. <4> Or <6> 10. An information processing device according to claim 9, wherein: <8> The second point detection unit detects the peak class from classes whose frequencies exceed a threshold in the histogram. <7> The information processing device described in <9> The second point detection unit detects the second point from the first point that belongs to a class included in a first cluster when counted from the lower side of the horizontal plane, when the classes of the histogram are clustered into clusters including one or more consecutive classes according to frequency. <4> Or <6> 10. An information processing device according to claim 9, wherein: <10> The second point detection unit detects the first point that belongs to the class with the highest frequency among the classes included in the first cluster as the second point. <9> The information processing device described in <11> the second point detection unit detects, from the first point, the second point on each of a plurality of horizontal planes as a plurality of specific planes; The specific plane detection unit detects each of the plurality of horizontal planes using the second point. <2> Or <6> 10. An information processing device according to claim 9, wherein: <12> an estimation unit that estimates the direction of gravity; a generation unit that generates point information including a position on a three-dimensional coordinate system and the normal direction for each point on the object; Furthermore, The first point detection unit detects, as the first point, a point on the object whose normal direction is parallel to the direction of gravity. <2> Or <11> 10. An information processing device according to claim 9, wherein: <13> a generating unit that generates point information for each point on the object, the point information including a position on a three-dimensional coordinate system and the normal direction; The first point detection unit detects a point on the object whose normal direction is in the predetermined direction as the first point. <1> Or <11> 10. An information processing device according to claim 9, wherein: <14> Detecting a first point on a predetermined direction plane whose normal direction is in a predetermined direction from a point on the object; Detecting a second point on a specific plane, which is a specific predetermined direction plane, from the first point; detecting the specific surface using the second point; and An information processing method including: <15> a first point detection unit that detects a first point on a predetermined direction plane whose normal direction is a predetermined direction from a point on the object; a second point detection unit that detects a second point on a specific plane that is a specific predetermined direction plane from the first point; a specific surface detection unit that detects the specific surface using the second point; A program that makes a computer function. [Explanation of symbols]

[0279] 10 Information processing system, 11 HMD, 12 Information processing device, 21 Camera, 22 IMU, 23 Distance measurement sensor, 24 Display unit, 31 Self-position estimation unit, 32 Point information generation unit, 33 Plane estimation unit, 41, 42 Point detection unit, 43 Plane parameter calculation unit, 51 Histogram generation unit, 52 Peak detection unit, 53 Inlier point extraction unit, 901 Bus, 902 CPU, 903 ROM, 904 RAM, 905 Hard disk, 906 Output unit, 907 Input unit, 908 Communication unit, 909 Drive, 910 Input / output interface, 911 Removable recording medium

Claims

1. a first point detection unit that detects, from a point on an object, a first point on a predetermined direction plane whose normal direction is a predetermined direction; a second point detection unit that detects, from the first point, a second point on a specific plane that is the specific predetermined direction plane; a specific surface detection unit that detects the specific surface using the second point; An information processing device comprising:

2. the predetermined direction plane is a horizontal plane whose normal direction is parallel to the direction of gravity, The first point detection unit detects the first point on the horizontal plane. The information processing device according to claim 1 .

3. the specific surface is a floor surface, The second point detection unit detects the second point on the floor surface. The information processing device according to claim 2 .

4. The second point detection unit generates a histogram for the first point, in which heights of the horizontal plane including the first point are used as classes, and detects the second point using the histogram. The information processing device according to claim 3 .

5. The second point detection unit generates the histogram by setting a height that is lower than the height of the wearable device that is the target of self-position estimation by a value that is set according to the height of the part of the user on which the wearable device is worn as the minimum value of the height of the horizontal plane. The information processing device according to claim 4 .

6. The second point detection unit performs filtering on the histogram using a low pass filter (LPF) and detects the second point using the histogram after the filtering. The information processing device according to claim 4 .

7. the second point detection unit detects a peak class that is the first peak in the histogram when viewed from the lower side of the horizontal plane; The first point belonging to the peak class is detected as the second point. The information processing device according to claim 4 .

8. The second point detection unit detects the peak class from classes whose frequencies exceed a threshold in the histogram. The information processing device according to claim 7 .

9. The second point detection unit detects the second point from the first point that belongs to a class included in a first cluster when counted from the lower side of the horizontal plane, when the classes of the histogram are clustered into clusters including one or more consecutive classes according to frequency. The information processing device according to claim 4 .

10. The second point detection unit detects the first point that belongs to the class with the highest frequency among the classes included in the first cluster as the second point. The information processing device according to claim 9 .

11. the second point detection unit detects, from the first point, the second point on each of a plurality of horizontal planes as a plurality of specific planes; The specific plane detection unit detects each of the plurality of horizontal planes using the second point. The information processing device according to claim 2 .

12. an estimation unit that estimates the direction of gravity; a generation unit that generates point information including a position on a three-dimensional coordinate system and the normal direction for each point on the object; Furthermore, The first point detection unit detects, as the first point, a point on the object whose normal direction is parallel to the direction of gravity. The information processing device according to claim 2 .

13. a generating unit that generates point information for each point on the object, the point information including a position on a three-dimensional coordinate system and the normal direction; The first point detection unit detects a point on the object whose normal direction is in the predetermined direction as the first point. The information processing device according to claim 1 .

14. Detecting a first point on a predetermined direction plane whose normal direction is in a predetermined direction from points on the object; Detecting a second point on a specific plane, which is the specific predetermined direction plane, from the first point; detecting the specific surface using the second point; and An information processing method including:

15. a first point detection unit that detects, from a point on an object, a first point on a predetermined direction plane whose normal direction is a predetermined direction; a second point detection unit that detects, from the first point, a second point on a specific plane that is the specific predetermined direction plane; a specific surface detection unit that detects the specific surface using the second point; A program that makes a computer function.

Images