Pose tracking device and method

Abstract

A pose tracking device and method are disclosed. The device captures an image of one eye of a user. Based on the eye image, the device calculates the pose of the eye. The device captures a reflection image from the eye image. Based on the reflection image and the eye pose, the device calculates the pose of the user's head. The pose tracking method disclosed herein can calculate a more accurate head pose.

Description

Technical Field

[0001] This disclosure relates to a pose tracking device and method, and more particularly to a pose tracking device and method based on eye images. Background Technology

[0002] In existing technologies, head-mounted devices typically need to perform calculations based on the scene image and the camera's own parameters (e.g., the relative relationships between multiple shooting perspectives, image distortion parameters) in order to track the user's head pose and construct a scene. On the other hand, head-mounted devices also need to be equipped with an image capturing unit facing the user in order to track the user's facial and / or eye movements.

[0003] Therefore, to achieve the aforementioned functions, the head-mounted device needs to be equipped with multiple cameras facing different directions, resulting in limitations in hardware specifications, increased manufacturing costs, and increased weight. Furthermore, for privacy or confidentiality reasons, the cameras used for capturing scenes may capture images that the user does not wish to be photographed.

[0004] In view of this, providing more convenient head pose tracking technology is an urgent goal that the industry needs to strive for. Summary of the Invention

[0005] To address the aforementioned problems, this disclosure proposes a pose tracking device comprising a camera and a processor. The camera is used to capture an image of one eye of a user. The processor is electrically connected to the camera and is used to perform the following operations: calculating the pose of one eye based on the eye image; capturing a reflection image from the eye image; and calculating the pose of one head of the user based on the reflection image and the eye pose.

[0006] In one embodiment of the present invention, the operation of calculating the eye pose further includes: calculating a pupil position and a gaze direction based on the eye image; and generating the eye pose based on the pupil position and the gaze direction.

[0007] In one embodiment of the present invention, the operation of calculating the eye pose further includes: calculating a binocular distance based on a plurality of corrected eye images captured from a plurality of different viewpoints; and calculating the eye pose based on the binocular distance and the eye images.

[0008] In one embodiment of the present invention, the operation of capturing the reflected image further includes: generating an eye background image based on a plurality of first reference eye images; and removing the eye background image from the eye image to generate the reflected image.

[0009] In one embodiment of the present invention, the operation of calculating the head pose further includes a plurality of iterative operations, and each of the plurality of iterative operations includes the following operations: calculating a candidate head pose and a loss value based on the reflection image and the eye pose; wherein the operation of calculating the head pose further includes: selecting the head pose from the candidate head poses corresponding to the plurality of iterative operations based on the loss value corresponding to the plurality of iterative operations.

[0010] In one embodiment of the present invention, the operation of calculating the head pose further includes a plurality of iterative operations, and each of the plurality of iterative operations includes the following operations: calculating a candidate head pose and an optimized parameter based on an initial parameter; and using the optimized parameter as the initial parameter for the next iteration in the plurality of iterative operations.

[0011] In one embodiment of the present invention, the operation of calculating the head pose further includes: acquiring a second reference eye image, wherein the second reference eye image is captured by the camera when the user looks at a preset image; calculating a deformation parameter based on the second reference eye image and the preset image, wherein the deformation parameter is used to represent a deformation state of the preset image in the second reference eye image; adjusting the reflected image based on the deformation parameter; and calculating the head pose based on the adjusted reflected image and the eye pose.

[0012] In one embodiment of the present invention, it further includes: an inertial measurement unit for measuring inertial data; wherein the operation of calculating the head pose further includes: calculating the head pose based on the inertial data corresponding to the eye pose, the reflected image, and the eye pose.

[0013] In one embodiment of the present invention, the processor further performs the following operations: constructing a scene image based on the reflected image and the eye pose, wherein the scene image corresponds to a line of sight of the eye pose.

[0014] In one embodiment of the present invention, the device further includes: an inertial measurement unit for measuring inertial data; wherein the processor further performs the following operations: determining whether the eye image contains a valid pupil image; and in response to the eye image not containing the valid pupil image, calculating the head pose corresponding to the inertial data based on the inertial data.

[0015] This disclosure also provides a pose tracking method applicable to an electronic device, the steps of which include: capturing an image of one eye of a user; calculating the pose of one eye based on the image of the eye; capturing a reflection image in the image of the eye; and calculating the pose of one head of the user based on the reflection image and the pose of the eye.

[0016] It should be understood that the foregoing general description and the following specific description are merely exemplary and explanatory, and are intended to provide further explanation of the claimed disclosure. Attached Figure Description

[0017] To make the above and other objects, features, advantages and embodiments of this disclosure more apparent and understandable, the accompanying drawings are described below: Figure 1 This is a schematic diagram of an environment in which a user wears a head-mounted device, as described in some embodiments of this disclosure. Figure 2 This is a schematic diagram of the pose tracking device in the first embodiment of this disclosure; Figure 3 A schematic diagram of a pose tracking device capturing reflected images from eye images; and Figure 4 This is a flowchart of the pose tracking method in the second embodiment of this disclosure.

[0018] Symbol explanation: E: Environment S: Specific object G: Headset U: User 1: Pose tracking device 12: Processor 14: Camera EI: Eye Imaging RI: Reflected image BI: Eye Background Image 200: Pose Tracking Methods S201~S204: Steps Detailed Implementation

[0019] To make the description of this disclosure more detailed and complete, reference may be made to the accompanying drawings and the various embodiments described below, in which the same numbers represent the same or similar elements.

[0020] Please refer to Figure 1 This is a schematic diagram of an environment E in which a user U wears a head-mounted device G, as described in some embodiments of this disclosure. Figure 1 As shown, user U wears a head-mounted device G. Environment E is an environment in which user U operates the head-mounted device G. Environment E includes a specific object S, which is a real object. In one embodiment, the head-mounted device G may be a device in a mixed reality (MR) system or an augmented reality (AR) system, and this disclosure is not limited thereto.

[0021] Please refer to Figure 2This is a schematic diagram of a pose tracking device 1 according to a first embodiment of the present disclosure. The pose tracking device 1 includes a processor 12 and a camera 14, wherein the processor 12 is electrically connected to the camera 14. The pose tracking device 1 is used to determine the user's head pose. In some embodiments, the pose tracking device 1 is located on... Figure 1 The head-mounted device G shown is used to determine the head position of the user U. In some embodiments, the pose tracking device 1 is disposed in the glasses and is used to determine the user's head position.

[0022] In some embodiments, processor 12 may include a central processing unit (CPU), a graphics processing unit (GPU), a multiprocessor, a distributed processing system, an application-specific integrated circuit (ASIC), and / or a suitable computing unit.

[0023] Camera 14 includes one or more image capturing units for capturing an image of one eye of a user. In some embodiments, camera 14 includes an image capturing unit disposed on pose tracking device 1 and facing the user to capture images. In some embodiments, camera 14 includes a depth image capturing unit for capturing a depth image of the user's eye.

[0024] It should be noted that the eye images captured by camera 14 may include a single image captured by a single image capturing unit, or multiple images captured by multiple image capturing units from multiple different perspectives. This disclosure does not limit the number of eye images or the shooting perspective.

[0025] The pose tracking device 1 uses the user's eyes as the medium for capturing the scene outwards, and analogizes the user's eyes to a camera lens. To achieve this, the pose tracking device 1 calculates the eye pose to obtain the lens's viewpoint, and captures the image reflected by the pupil as the scene image captured by the lens. Furthermore, after obtaining the lens viewpoint and the scene image, the pose tracking device 1 can then track the user's head pose.

[0026] Specifically, the processor 12 performs the following operations: calculating the pose of one eye based on the eye image; capturing a reflection image from the eye image; and calculating the pose of one head of the user based on the reflection image and the eye pose.

[0027] For example, after obtaining the user's eye pose and reflection image, the pose tracking device 1 calculates the user's head pose based on algorithms such as simultaneous localization and mapping (SLAM), where the head pose is the user's head six degrees of freedom (6DoF) data.

[0028] In some embodiments, the eye images include multiple images of the user's eyes captured by the camera 14 over a period of time. Accordingly, the pose tracking device 1 can determine the user's head movements based on changes in the pupil reflection images in the multiple images, and then calculate the head pose.

[0029] For example, the pose tracking device 1 uses the earliest captured image among the multiple images as a reference to identify objects and / or blocks presented in the reflected image. Based on this, the pose tracking device 1 determines the positional changes of objects and / or blocks in the reflected image in subsequent images, and calculates the head pose by combining the eye pose.

[0030] In some embodiments, since partial images of the environment and the pose of the user's head have been obtained, the pose tracking device 1 can further utilize reflected images and eye poses to construct a scene in three-dimensional space.

[0031] Specifically, the processor 12 further performs the following operations: constructing a scene image based on the reflected image and the eye pose, wherein the scene image corresponds to a line of sight in the eye pose.

[0032] For example, when the pose tracking device 1 calculates the head pose based on the simultaneous localization and mapping algorithm, it can also construct a scene image at the same time. The scene image includes images of the user's surrounding environment and the relative positional relationship between the image and the user's head in three-dimensional space.

[0033] In some embodiments, to increase the accuracy of pose tracking, the pose tracking device 1 further incorporates inertial data measured by an inertial measurement unit (IMU) to calculate head pose.

[0034] Specifically, the pose tracking device 1 further includes an inertial measurement unit (not shown in the figure), wherein the inertial measurement unit is used to measure inertial data. Correspondingly, the operation of the processor 12 to calculate the head pose further includes: calculating the head pose based on the inertial data corresponding to the eye pose, the reflected image, and the eye pose.

[0035] For example, when calculating head pose based on the Simultaneous Localization and Mapping (SLAM) algorithm, the pose tracking device 1 uses reflected images and eye pose as parameters for calculation, and also incorporates inertial data such as acceleration, angular velocity, and orientation for calculation. The inertial data is data captured simultaneously with the eye images.

[0036] In some embodiments, when calculating eye pose, the pose tracking device 1 calculates the position of the user's pupil and the direction of the pupil's gaze from the eye image, and determines the direction of the user's gaze (i.e., eye pose) as the basis for subsequent calculation of head pose.

[0037] Specifically, the operation of the processor 12 in calculating the eye pose further includes: calculating a pupil position and a gaze direction based on the eye image; and generating the eye pose based on the pupil position and the gaze direction.

[0038] For example, the pose tracking device 1 calculates the coordinates of the pupil in the image (i.e., the pupil position) based on the eye image and calculates the vector of the user's gaze (i.e., the gaze direction) based on the position of the pupil, and uses it as the eye pose.

[0039] In some embodiments, before calculating the head pose based on the user's eye pose, the pose tracking device 1 needs to obtain the distance between the user's eyes as an initialization parameter.

[0040] Specifically, the operation of the processor 12 to calculate the eye pose further includes: calculating a binocular distance based on multiple corrected eye images captured from multiple different viewpoints; and calculating the eye pose based on the binocular distance and the eye images.

[0041] The pose tracking device 1 can obtain the user's eye distance in many different ways. For example, when the camera 14 includes image capturing units located at multiple different angles, the pose tracking device 1 captures multiple images of the user's eyes from different angles while the user is wearing the device, and further calculates the user's eye distance based on the relative positional relationship of the image capturing units.

[0042] In other examples, when camera 14 includes a depth image capturing unit, pose tracking device 1 captures a depth image of the user's eyes while the user is wearing the device, and then calculates the user's binocular distance based on the depth information in the depth image.

[0043] In other examples, when the pose tracking device 1 includes an inertial measurement unit, the pose tracking device 1 captures images of the user's eyes as the user wears the device and moves their head, and then calculates the user's eye distance based on the inertial data measured by the inertial measurement unit and the eye images.

[0044] In some embodiments, in order to capture the reflected image, the pose tracking device 1 captures an eye background image from multiple eye images, wherein the eye background image includes parts of the eye image that are not part of the pupil reflection scene, such as skin, sclera, and pupil. Further, when capturing the reflected image, the pose tracking device 1 removes the eye background image from the eye image, thereby obtaining the reflected image.

[0045] Please refer to Figure 3 This is a schematic diagram of the pose tracking device 1 capturing the reflected image RI from the eye image EI. As shown in the figure, the pupil of the user U in the eye image EI reflects the image of a specific object S in the environment E. Through the aforementioned operation, the pose tracking device 1 can capture the reflected image RI from the eye image EI based on the pre-generated eye background image BI, wherein the reflected image RI presents the image reflected by the pupil.

[0046] Specifically, the operation of the processor 12 to capture the reflected image further includes: generating an eye background image based on a plurality of first reference eye images; and removing the eye background image from the eye image to generate the reflected image.

[0047] For example, the first reference eye image can be multiple eye images captured by camera 14 and corresponding eye background images BI generated by processor 12 when the user wears pose tracking device 1. Subsequently, when calculating head pose, pose tracking device 1 can generate reflection image RI based on eye background image BI.

[0048] It should be noted that this disclosure does not limit the timing or subject of the first reference eye image capture; in fact, the pose tracking device 1 can be adjusted as needed. For example, the pose tracking device 1 can capture multiple eye images as the first reference eye image while calculating the head pose, or it can use the eye images of other users as the first reference eye image.

[0049] In some embodiments, when calculating head pose, the pose tracking device 1 gradually calculates a more accurate head pose through multiple iterative calculations.

[0050] Specifically, the operation of processor 12 in calculating the head pose further includes multiple iterative operations, and each of the multiple iterative operations includes the following operations: calculating a candidate head pose and a loss value based on the reflection image and the eye pose. Furthermore, the operation of processor 12 in calculating the head pose further includes: selecting a head pose from the candidate head poses corresponding to each of the multiple iterative operations based on the loss value corresponding to each of the multiple iterative operations.

[0051] For example, pose tracking device 1 uses a simultaneous localization and mapping (SMR) algorithm to calculate candidate head poses and corresponding loss values ​​in each iteration. Furthermore, pose tracking device 1 corrects the parameters based on the gradient direction of the loss value, and in the next iteration, calculates a head pose with a lower loss value (i.e., a more accurate head pose) based on the corrected parameters.

[0052] Generally speaking, when a camera captures an image, different lens designs will produce different image distortions. When calculating head pose or constructing scene images, it is necessary to correct the image distortion using the camera's intrinsic parameters. On the other hand, since the eyeball is roughly spherical, the reflected image will also be distorted. Therefore, corresponding intrinsic parameters are also needed to correct the reflected image as a reference for subsequent calculations of head pose or construction of scene images.

[0053] The pose tracking device 1 can obtain internal parameters in different ways. In some embodiments, the pose tracking device 1 first sets the internal parameters required for iterative calculation to initial values, and then corrects the internal parameters by referencing the loss value during multiple iterative calculations.

[0054] Specifically, the operation of the processor 12 in calculating the head pose further includes multiple iterative operations, and each of the multiple iterative operations includes the following operations: calculating a candidate head pose and an optimized parameter based on an initial parameter; and using the optimized parameter as the initial parameter for the next iteration in the multiple iterative operations.

[0055] For example, similar to the aforementioned embodiments, when the pose tracking device 1 calculates the head pose using the simultaneous localization and mapping algorithm, it first performs the first iteration calculation with initial parameters, and then in multiple iteration calculations, it corrects the internal parameters based on the gradient direction of the loss value to obtain more accurate internal parameters.

[0056] In another embodiment, the pose tracking device 1 requires the user to look at a specific image. Since the specific image is known, the pose tracking device 1 can calculate the internal parameters by judging the deformation state of the specific image in the image reflected by the user's pupil.

[0057] Specifically, the operation of the processor 12 in calculating the head pose further includes: acquiring a second reference eye image, wherein the second reference eye image is captured by the camera when the user looks at a preset image; calculating a deformation parameter based on the second reference eye image and the preset image, wherein the deformation parameter is used to represent a deformation state of the preset image in the second reference eye image; adjusting the reflected image based on the deformation parameter; and calculating the head pose based on the adjusted reflected image and the eye pose.

[0058] When the user closes their eyes, the pose tracking device 1 cannot calculate the head pose from the eye images. Therefore, in some embodiments, the pose tracking device 1 calculates the head pose based on inertial data when the user closes their eyes.

[0059] Specifically, the pose tracking device 1 further includes an inertial measurement unit (not shown in the figure), wherein the inertial measurement unit is used to measure inertial data. The processor 12 further performs the following operations: determining whether the eye image contains a valid pupil image; and in response to the eye image not containing the valid pupil image, calculating the head pose corresponding to the inertial data based on the inertial data.

[0060] For example, similar to the aforementioned embodiments, the pose tracking device 1 also includes an inertial measurement unit. When the pose tracking device 1 determines that the eye image captured by the camera 14 does not contain a pupil, the pose tracking device 1 calculates the head pose using the inertial data measured by the inertial measurement unit.

[0061] In summary, the pose tracking device 1 proposed in this disclosure can calculate the user's head pose by capturing images of the user's eyes. Besides reducing specification requirements and costs, the pose tracking device 1 can also avoid capturing images that the user does not wish to be photographed. Furthermore, the pose tracking device 1 can further construct scene images. Moreover, through initialization, correction, and multiple iterative calculations, the pose tracking device 1 can calculate a more accurate head pose.

[0062] Please refer to Figure 4 This is a flowchart of the pose tracking method 200 in the second embodiment of this disclosure. The pose tracking method 200 includes steps S201 to S204. The pose tracking method 200 is used to determine the user's head pose. The pose tracking method 200 can be executed by an electronic device (e.g., the pose tracking device 1 in the first embodiment).

[0063] First, in step S201, the electronic device captures an image of one of a user's eyes.

[0064] Next, in step S202, the electronic device calculates the pose of one eye based on the eye image.

[0065] Next, in step S203, the electronic device captures a reflected image from the eye image.

[0066] Finally, in step S204, the electronic device calculates the head position pose of the user based on the reflected image and the eye position pose.

[0067] In some embodiments, step S202 further includes the electronic device calculating a pupil position and a gaze direction based on the eye image; and the electronic device generating the eye pose based on the pupil position and the gaze direction.

[0068] In some embodiments, step S202 further includes the electronic device calculating a binocular distance based on multiple corrected eye images captured from multiple different viewpoints; and the electronic device calculating the eye pose based on the binocular distance and the eye images.

[0069] In some embodiments, step S203 further includes the electronic device generating an eye background image based on a plurality of first reference eye images; and the electronic device removing the eye background image from the eye image to generate the reflected image.

[0070] In some embodiments, step S204 further includes multiple iterative operations, and each of the multiple iterative operations includes the following steps: the electronic device calculates a candidate head pose and a loss value based on the reflected image and the eye pose. Step S204 further includes the electronic device selecting the head pose from the candidate head poses corresponding to the multiple iterative operations based on the loss value corresponding to each of the multiple iterative operations.

[0071] In some embodiments, step S204 further includes multiple iterative operations, and each of the multiple iterative operations includes the following steps: the electronic device calculates a candidate head pose and an optimized parameter based on an initial parameter; and the electronic device uses the optimized parameter as the initial parameter for the next iteration in the multiple iterative operations.

[0072] In some embodiments, step S204 further includes the electronic device acquiring a second reference eye image, wherein the second reference eye image is captured when the user looks at a preset image; the electronic device calculating a deformation parameter based on the second reference eye image and the preset image, wherein the deformation parameter is used to represent a deformation state of the preset image in the second reference eye image; the electronic device adjusting the reflected image based on the deformation parameter; and the electronic device calculating the head pose based on the adjusted reflected image and the eye pose.

[0073] In some embodiments, step S204 further includes the electronic device calculating the head pose based on inertial data corresponding to the eye pose, the reflected image, and the eye pose.

[0074] In some embodiments, the pose tracking method 200 further includes the electronic device constructing a scene image based on the reflected image and the eye pose, wherein the scene image corresponds to a gaze direction in the eye pose.

[0075] In some embodiments, the pose tracking method 200 further includes the electronic device determining whether the eye image contains a valid pupil image; and in response to the eye image not containing the valid pupil image, the electronic device calculating the head pose corresponding to the inertial data based on inertial data.

[0076] In summary, the pose tracking method 200 proposed in this disclosure can calculate the user's head pose by capturing the user's eye images. Besides reducing specification requirements and costs, the pose tracking method 200 can also avoid capturing images that the user does not want to be photographed. Furthermore, the pose tracking method 200 can further construct scene images. Moreover, through initialization, correction, and multiple iterative calculations, the pose tracking method 200 can calculate a more accurate head pose.

[0077] Although several embodiments have been described in detail above as examples, the pose tracking device and method proposed in this disclosure can also be implemented in other systems, hardware, software, storage media, or combinations thereof. Therefore, the scope of protection of this disclosure should not be limited to the specific implementations described in the embodiments of this disclosure, but should be determined by the appended claims.

[0078] It will be apparent to those skilled in the art to which this disclosure pertains that various modifications and variations can be made to the structure of this disclosure without departing from its scope or spirit. In view of the foregoing, the scope of protection of this disclosure also covers modifications and variations made within the scope of the appended claims.

Claims

1. A pose tracking device, characterized by, Include: A camera used to capture an image of one of a user's eyes; and A processor, electrically connected to the camera, is used to perform the following operations: Based on this eye image, calculate the pose of one eye. Capture a reflected image from the eye image; and Based on the reflected image and the eye position pose, the pose of one of the user's head positions is calculated.

2. The pose tracking device as described in claim 1, characterized in that, The operation for calculating the eye pose further includes: Based on the eye image, calculate the position of a pupil and the direction of a gaze; and The eye pose is generated based on the pupil position and the direction of the gaze.

3. The pose tracking device as described in claim 1, characterized in that, The operation for calculating the eye pose further includes: Based on multiple corrected eye images captured from multiple different viewpoints, the binocular distance is calculated; and The eye pose is calculated based on the binocular distance and the eye image.

4. The pose tracking device as described in claim 1, characterized in that, The operation of capturing the reflected image further includes: A background image of one eye is generated based on multiple first reference eye images; and The background image of the eye is removed from the eye image to generate the reflected image.

5. The pose tracking device as described in claim 1, characterized in that, The operation of calculating the head pose further includes multiple iterative operations, and each of these multiple iterative operations includes the following operations: Based on the reflected image and the eye pose, a candidate head pose and a loss value are calculated. The operation of calculating the head pose further includes: Based on the loss value corresponding to each of the multiple iterative operations, the head pose is selected from the candidate head poses corresponding to each of the multiple iterative operations.

6. The pose tracking device as described in claim 1, characterized in that, The operation of calculating the head pose further includes multiple iterative operations, and each of these multiple iterative operations includes the following operations: Based on an initial parameter, calculate a candidate head pose and an optimized parameter; and The optimized parameter is used as the initial parameter for the next iteration in the multiple iterations.

7. The pose tracking device as described in claim 1, characterized in that, The operation of calculating the head pose further includes: A second reference eye image is acquired, wherein the second reference eye image is captured by the camera when the user looks at a preset image; Based on the second reference eye image and the preset image, a deformation parameter is calculated, wherein the deformation parameter is used to represent a deformation state of the preset image in the second reference eye image; Based on this deformation parameter, adjust the reflected image; and The head pose is calculated based on the adjusted reflected image and the eye pose.

8. The pose tracking device as described in claim 1, characterized in that, Further includes: An inertial measurement unit (IMU) is used to measure inertial data. The operation of calculating the head pose further includes: The head pose is calculated based on the inertial data, the reflected image, and the eye pose corresponding to the eye pose.

9. The pose tracking device as described in claim 1, characterized in that, The processor further performs the following operations: Based on the reflected image and the eye pose, a scene image is constructed, wherein the scene image corresponds to a line of sight for the eye pose.

10. The pose tracking device as described in claim 1, characterized in that, Further includes: An inertial measurement unit (IMU) is used to measure inertial data. The processor further performs the following operations: Determine whether the eye image contains a valid pupil image; and In response to the absence of a valid pupil image in the eye image, the head pose corresponding to the inertial data is calculated based on the inertial data.

11. A pose tracking method, characterized in that, Applicable to an electronic device, the steps include: Capture an image of one of a user's eyes; Based on this eye image, calculate the pose of one eye. Capture a reflected image from the eye image; and Based on the reflected image and the eye position pose, the pose of one of the user's head positions is calculated.

Images