Method for determining a floor plane and host computer

Abstract

A method and a host for determining a floor plane are disclosed. The method comprises determining a first plane as a reference floor plane based on motion data; in response to determining that the host has moved a predetermined distance after the first plane is determined as the reference floor plane, determining a plurality of first feature points on the reference floor plane; determining a second plane based on the first feature points on the reference floor plane; and in response to determining that the second plane satisfies a predetermined condition, determining the second plane as the reference floor plane.

Description

TECHNICAL FIELD

[0001] The present application generally relates to a tracking mechanism, and in particular, to a method for determining a floor plane and a host. BACKGROUND

[0002] Referring to Figure 1A and Figure 1B wherein Figure 1A a schematic diagram of a skewed coordinate system is shown, and Figure 1B a schematic diagram of a skewed floor plane is shown.

[0003] Conventionally, when tracking the pose of a device (e.g., a head-mounted display (HMD)) using a tracking system that utilizes a tracking mechanism such as inside-out tracking, the tracking system needs to first determine a coordinate system 10 and in turn determine the pose of the tracked device, where the y-z plane can be understood as a floor plane corresponding to the coordinate system 10.

[0004] However, over time, the tracked device (e.g., the HMD) can move around, and the coordinate system under consideration (e.g., coordinate system 10') can deviate from the correct coordinate system (e.g., coordinate system 10) due to, for example, accumulated tracking errors.

[0005] In Figure 1A , the x-axis of the coordinate system 10' can be skewed from the x-axis of the coordinate system 10 by a skew angle (referred to as φ), such that the floor plane (e.g., the y-z plane) corresponding to the coordinate system 10' is also skewed from the floor plane corresponding to the coordinate system 10 by the skew angle.

[0006] In such a scenario, the tracked height of the HMD can be different from the height of the HMD perceived by the user. Since the visual content provided by the HMD (e.g., virtual reality (VR) visual content) is rendered based on the pose of the HMD as tracked, the user can feel disoriented when viewing the visual content. SUMMARY

[0007] Accordingly, the present application relates to a method for determining a floor plane and a host that can be used to solve the above technical problems.

[0008] The embodiment of the present application provides a method for determining a floor plane suitable for a host, which comprises: determining a first plane as a reference floor plane based on motion data; determining a plurality of first feature points on the reference floor plane in response to determining that the host has moved a predetermined distance after determining the first plane as the reference floor plane; determining a second plane based on the first feature points on the reference floor plane; and determining the second plane as the reference floor plane in response to determining that the second plane satisfies a predetermined condition.

[0009] The embodiment of the present application provides a host comprising a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the non-transitory storage circuit and accesses the program code to perform: determining a first plane as a reference floor plane based on motion data; determining a plurality of first feature points on the reference floor plane in response to determining that the host has moved a predetermined distance after determining the first plane as the reference floor plane; determining a second plane based on the first feature points on the reference floor plane; and determining the second plane as the reference floor plane in response to determining that the second plane satisfies a predetermined condition. BRIEF DESCRIPTION OF DRAWINGS

[0010] The present application includes drawings to provide further understanding of the present application, and the drawings are incorporated into the present specification and form part of the present specification. The drawings show embodiments of the present application, and are used together with the present specification to explain the principles of the present application.

[0011] Figure 1A is a schematic diagram of a skewed coordinate system;

[0012] Figure 1B is a schematic diagram of a skewed floor plane;

[0013] Figure 2 is a schematic diagram of a host of the embodiment of the present application;

[0014] Figure 3 is a flowchart of a method for determining a floor plane of the embodiment of the present application;

[0015] Figure 4 is a schematic diagram of determining feature points on a reference floor plane of the embodiment of the present application;

[0016] Figure 5 is a schematic diagram of an application scenario of the embodiment of the present application.

[0017] Explanation of symbols

[0018] 10, 10', 510, 520, 530: coordinate system

[0019] 200: host

[0020] 202: storage circuit

[0021] 204: processor

[0022] 400, IM1, IM2, IM3: image

[0023] 410: image region

[0024] 500: field

[0025] S310, S320, S330, S340: step

[0026] x, y, z: direction axis

[0027] φ: skew angle DETAILED DESCRIPTION

[0028] Reference will now be made in detail embodiments of the application, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

[0029] Referring to Figure 2 , which shows a schematic diagram of a host according to an embodiment of the application. In Figure 2 , the host 200 can be any device capable of tracking the pose of the host 200 by implementing an inside-out tracking mechanism (e.g., Simultaneous Localization and Mapping (SLAM)), but the application is not limited thereto. In one embodiment, the host 200 can be an HMD that provides augmented reality (AR) / VR services / content.

[0030] In some embodiments, the HMD can include a (front) camera with a field of view (FOV), and the HMD can implement a tracking function based on images captured by the camera, but the application is not limited thereto. In the present application, the details of the inside-out tracking mechanism can refer to related prior art, and will not be described here.

[0031] In Figure 2 , the host 200 includes a storage circuit 202 and a processor 204. The storage circuit 202 is one of or a combination of a fixed random access memory (RAM) or a mobile RAM, a read-only memory (ROM), a flash memory, a hard disk, or any other similar device, and the storage circuit 202 records a plurality of modules executable by the processor 204.

[0032] The processor 204 can be coupled to the storage circuit 202 and can be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like.

[0033] In embodiments of the present application, the processor 204 can access modules stored in the storage circuit 202 to implement the method of determining a floor plane provided in the present application, which will be discussed further below.

[0034] Referring to Figure 3 , a flowchart of a method of determining a floor plane according to embodiments of the present application is shown. Figure 2 The host 200 in Figure 2 may perform the method of the present embodiments, and details of each step in Figure 3 will be set forth below in conjunction with the components shown in .

[0035] First, in step S310, the processor 204 determines a first plane as a reference floor plane based on motion data.

[0036] In one embodiment, the host 200 can include a motion detection circuit (e.g., an inertial measurement unit (IMU)) for collecting motion data, and the collected motion data can be 6 degrees-of-freedom (DOF) data, but the present application is not limited thereto.

[0037] In the present embodiment, how the processor 204 determines a floor plane based on motion data (e.g., IMU data) can refer to related prior art, which will not be discussed further herein.

[0038] In embodiments of the present application, the processor 204 can establish a coordinate system based on motion data and accordingly determine a floor plane as a first plane (e.g., a reference floor plane), but the present application is not limited thereto.

[0039] In an embodiment of the present application, the processor 204 can determine the pose of the host 200 based on the reference floor plane, and details thereof can also be referred to the related prior art, which will not be further discussed herein. That is, the processor 204 can determine the pose of the host 200 based on the first plane.

[0040] In one embodiment, after determining the first plane as the reference floor plane, the processor 204 can determine whether the host 200 has moved a predetermined distance (e.g., 1 meter or any other distance preferred by the designer).

[0041] In one embodiment, if the processor 204 determines that the host 200 has not moved the predetermined distance after determining the first plane as the reference floor plane, the processor 204 can maintain the first plane as the reference floor plane. That is, in the case where the host 200 has not moved the predetermined distance after determining the first plane as the reference floor plane, the processor 204 can still determine the pose of the host 200 based on the first plane.

[0042] In another embodiment, if the processor 204 determines that the host 200 has moved the predetermined distance after determining the first plane as the reference floor plane, the processor 204 can perform step S320.

[0043] In step S320, in response to determining that the host has moved the predetermined distance after determining the first plane as the reference floor plane, the processor 204 determines a plurality of first feature points on the reference floor plane.

[0044] Referring to Figure 4 , a schematic diagram of determining feature points on a reference floor plane according to an embodiment of the present application is shown. In Figure 4 , the processor 204 can obtain an image 400 corresponding to a front view angle of the host 200, for example, by using a front camera of the host 200. Next, the processor 204 can identify an image region 410 corresponding to the reference floor plane and determine a plurality of feature points within the image region 410 as the plurality of first feature points on the reference floor plane.

[0045] From another perspective, the processor 204 can determine feature points (which are exemplarily shown as black dots in Figure 4 ) in the image 400 based on, for example, SLAM, and the processor 204 can determine the image region 410 corresponding to the reference floor plane. In Figure 4 , image regions corresponding to non-floor objects (e.g., cars, trees, houses, etc.) will not be considered as the image region 410. Next, the processor 204 determines the feature points within the image region 410 as the plurality of first feature points.

[0046] In step S330, the processor 204 determines a second plane based on the first feature points on the reference floor plane.

[0047] In one embodiment, the processor 204 can determine whether the number of the first feature points is higher than a predetermined threshold (e.g., 3 or more). If the number of the first feature points is higher than the predetermined threshold, the processor 204 implements step S330; if the number of the first feature points is not higher than the predetermined threshold, the processor 204 can not implement step S330 until the number of the first feature points is higher than the predetermined threshold, but the present application is not limited thereto.

[0048] In one embodiment, in determining the second plane based on the first feature points on the reference floor plane, the processor 204 can determine at least one candidate plane, wherein the at least one candidate plane comprises a first candidate plane. In one embodiment, the processor 204 can determine the at least one candidate plane in a coordinate system used to track the pose of the host 200, but the present application is not limited thereto.

[0049] Next, the processor 204 can determine an error characteristic corresponding to each candidate plane. For example, the error characteristic corresponding to the first candidate plane can comprise an average distance between each first feature point and the first candidate plane, but the present application is not limited thereto.

[0050] In one embodiment, in response to determining that the error characteristic corresponding to the first candidate plane is the smallest among the at least one candidate plane, the processor 204 can determine the first candidate plane as the second plane.

[0051] That is, the processor 204 can determine the candidate plane having the smallest average distance from each first feature point as the second plane, but the present application is not limited thereto.

[0052] After determining the second plane, the processor 204 can determine whether the second plane satisfies a predetermined condition. For example, the processor 204 can determine whether the average distance corresponding to the second plane is within a predetermined range. If the average distance corresponding to the second plane is within the predetermined range, the processor 204 can determine that the second plane satisfies the predetermined condition; if the average distance corresponding to the second plane is not within the predetermined range, the processor 204 can determine that the second plane does not satisfy the predetermined condition, but the present application is not limited thereto.

[0053] In step S340, in response to determining that the second plane satisfies the predetermined condition, the processor 204 determines the second plane as the reference floor plane. That is, the processor 204 can take the second plane as a new reference floor plane and track the pose of the host 200 accordingly, but the present application is not limited thereto.

[0054] In one embodiment, after step S340, the processor 204 can further determine whether the host 200 has moved a predetermined distance after determining the second plane as the reference floor plane. If the host 200 has not moved the predetermined distance, the processor 204 can maintain the second plane as the reference floor plane; if the host 200 has moved the predetermined distance, the processor 204 can determine a plurality of second feature points on the reference floor plane (i.e., the second plane).

[0055] In the present embodiment, how the processor 204 determines the second feature points on the reference floor plane can refer to the description of how the processor 204 determines the first feature points on the first plane, which is not repeated here. Figure 4

[0056] Next, the processor 204 can determine a third plane based on the second feature points on the reference floor plane, and the details can refer to the description of how the processor 204 determines the second plane based on the first feature points, which is not repeated here.

[0057] After determining the third plane, the processor 204 can determine whether the third plane satisfies a predetermined condition. For example, the processor 204 can determine whether an average distance between the third plane and each second feature point is within a predetermined range. If the average distance between the third plane and each second feature point is within the predetermined range, the processor 204 can determine that the third plane satisfies the predetermined condition; if the average distance between the third plane and each second feature point is not within the predetermined range, the processor 204 can determine that the third plane does not satisfy the predetermined condition, but the present application is not limited thereto.

[0058] In one embodiment, in response to determining that the third plane satisfies the predetermined condition, the processor 204 can determine the third plane as the reference floor plane.

[0059] From another perspective, the processor 204 can be regarded as monitoring whether the host 200 has moved a predetermined distance after determining a plane as the reference floor plane. If the host 200 has moved the predetermined distance, the processor 204 finds feature points on the reference floor plane and determines a new plane accordingly. Next, if the new plane satisfies a predetermined condition, the processor 204 can determine the new plane as the reference floor plane and accordingly track the pose of the host 200, and the processor 204 can start monitoring whether the host 200 has moved the predetermined distance again.

[0060] Therefore, the reference floor plane used to track the pose of the host 200 can be updated as the host 200 moves around, so that the tracking accuracy does not decrease as the host 200 moves further.

[0061] Referring to Figure 5 , which shows an application scenario according to an embodiment of the present application. In Figure 5 ​In this case, assume that the host 200 (e.g., an HMD) moves around in the field 500.

[0062] First, the host 200 can sequentially perform: (1) determining a reference floor plane based on feature points corresponding to a floor plane within a captured image IM1 (this can also be regarded as determining a coordinate system 510, since the reference floor plane corresponds to the y-z plane of the coordinate system 510); (2) after the host 200 moves a predetermined distance, determining another reference floor plane based on feature points corresponding to a floor plane within a captured image IM2 (this can also be regarded as determining a coordinate system 520, since the another reference floor plane corresponds to the y-z plane of the coordinate system 520); (3) after the host 200 moves a predetermined distance, determining yet another reference floor plane based on feature points corresponding to a floor plane within a captured image IM3 (this can also be regarded as determining a coordinate system 530, since the yet another reference floor plane corresponds to the y-z plane of the coordinate system 530).

[0063] In summary, embodiments of the present application provide a solution of adaptively updating a considered reference floor plane based on feature points corresponding to a floor plane. Thus, since the reference floor plane (and / or coordinate system) is more accurate, the tracking accuracy of the host pose can be improved. In another perspective, the tracking error does not accumulate as the distance of host movement increases, and thus the problem of skewed floor plane can be avoided. In this case, the tracked height of the host can be more consistent with the height perceived by a user of the host, so that the user does not feel strange when viewing the visual content provided by the host.

[0064] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the scope or spirit of the application. Accordingly, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method of determining a floor plane to correct for accumulated tracking errors of a host in a tracking system, the method comprising: comprises: determining a first plane as a reference floor plane based on motion data; in response to determining that the host has moved a predetermined distance after determining the first plane as the reference floor plane, determining a plurality of first feature points on the reference floor plane; determining a second plane based on the plurality of first feature points on the reference floor plane; and in response to determining that the second plane satisfies a predetermined condition, determining the second plane as an updated reference floor plane to replace the first plane, thereby correcting the accumulated tracking error.

2. The method of claim 1, wherein the motion data is collected by a motion detection circuit on the host.

3. The method of claim 1, wherein determining the second plane based on the plurality of first feature points on the reference floor plane comprises: determining at least one candidate plane, wherein the at least one candidate plane comprises a first candidate plane; determining an error characteristic corresponding to each of the candidate planes; in response to determining that the error characteristic corresponding to the first candidate plane is the smallest among the at least one candidate plane, determining the first candidate plane as the second plane.

4. The method of claim 3, wherein the error characteristic corresponding to the first candidate plane comprises an average distance between each of the first feature points and the first candidate plane.

5. The method of claim 1, further comprising: determining an average distance between each of the first feature points and the second plane; in response to determining that the average distance corresponding to the second plane is within a predetermined range, determining that the second plane satisfies the predetermined condition; in response to determining that the average distance corresponding to the second plane is not within the predetermined range, determining that the second plane does not satisfy the predetermined condition.

6. The method of claim 1, wherein prior to determining the second plane based on the plurality of first feature points on the reference floor plane, the method further comprises: in response to determining that a number of the plurality of first feature points on the reference floor plane is higher than a predetermined threshold, determining the second plane based on the plurality of first feature points.

7. The method of claim 1, wherein after determining the second plane as the updated reference floor plane, the method further comprises: in response to determining that the host has moved the predetermined distance after determining the second plane as the updated reference floor plane, determining a plurality of second feature points on the updated reference floor plane; determining a third plane based on the second feature points on the updated reference floor plane; and in response to determining that the third plane satisfies the predetermined condition, determining the third plane as the reference floor plane.

8. The method of claim 1, wherein determining the plurality of first feature points on the reference floor plane comprises: obtaining an image corresponding to a front view angle of the host; identifying an image region corresponding to the reference floor plane; determining a plurality of feature points within the image region as the plurality of first feature points on the reference floor plane.

9. The method of claim 1, further comprising: determining a pose of the host based on the reference floor plane.

10. A host, characterized by, comprising: a storage circuit that stores program code; a processor coupled to the storage circuit that accesses the program code to correct a cumulative tracking error of the host in a tracking system and to perform: determining a first plane as a reference floor plane based on motion data; in response to determining that the host has moved a predetermined distance after determining the first plane as the reference floor plane, determining a plurality of first feature points on the reference floor plane; determining a second plane based on the plurality of first feature points on the reference floor plane; and in response to determining that the second plane satisfies a predetermined condition, determining the second plane as an updated reference floor plane to replace the first plane, thereby correcting the cumulative tracking error.

Images