Joint bending estimation

Abstract

A system for correcting for frame bending of an augmented reality system is provided. A combination of strain gauges and visual inertial odometry is used to determine strains in the frame. An initial model between strain gauge measurements and actual frame spatial relationships is based on finite element analysis or calibration. During an initial visual inertial odometry data calculation phase, the augmented reality system calculates bending or strains of the frame using strain data from the strain gauges mounted to the frame. Subsequent visual inertial odometry data calculations are used to generate a corrected frame model of the frame. The corrected frame model is used for calculating corrected tracking data and corrected virtual overlays that are used to generate virtual overlays used in an AR experience provided by the augmented reality system.

Description

This abstract provides a system for correcting frame bending in an augmented reality (AR) system. The system combines strain gauges and visual inertial odometry (VIO) to determine the strain within the frame. An initial model relating the strain gauge measurements to the actual spatial relationship of the frame can be established based on finite element analysis or calibration. In the initial VIO data calculation phase, the AR system uses strain data acquired by the strain gauges mounted on the frame to calculate the degree of bending or strain of the frame. Subsequent VIO data calculations generate a corrected frame model. This corrected frame model is used to calculate corrected tracking data and a corrected virtual overlay image, and the data and image are used to generate a virtual overlay image for display in the AR experience provided by the AR system.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: capturing, by one or more processors, using one or more imaging devices of an Augmented Reality (AR) system, tracking video frame data of a real-world scene; measuring, by one or more processors, using one or more strain gauges of the AR system, strain data of strains of a frame of the AR system as the tracking video frame data is captured; generating, by the one or more processors, a corrected frame model of the frame based on the strain data, the tracking video frame data, and a frame model of the frame; and generating, by the one or more processors, corrected tracking data based on the corrected frame model and the tracking video frame data.

2. The computer-implemented method of claim 1, further comprising: generating, by the one or more processors, virtual overlay data based on the corrected tracking data; generating, by the one or more processors, corrected virtual overlay video frame data based on the corrected frame model and the virtual overlay data; and providing, by the one or more processors, using an optical engine of the AR system, a virtual overlay to a user of the AR system based on the corrected virtual overlay video frame data.

3. The computer-implemented method of claim 1, wherein the operation of generating the corrected frame model of the frame further comprises: generating visual inertial odometry data based on the tracking video frame data, the strain data, and the frame model; and generating the corrected frame model based on the visual inertial odometry data and the frame model.

4. The computer-implemented method of claim 3, wherein the operation of generating the visual inertial odometry data further comprises: generating yaw bending data of the frame based on the strain data; andgenerating the visual inertial odometry data based on the tracking video frame data, the yaw bending data, and the frame model.

5. The computer-implemented method of claim 3, wherein the operation of generating the visual inertial odometry data further comprises: measuring, by the one or more processors, using an inertial measurement unit of the frame, inertial movement data of the frame as the tracking video frame data is captured; generating initial inertial odometry data based on the inertial movement data; and generating the visual inertial odometry data based on the initial inertial odometry data, the tracking video frame data, the strain data, and the frame model of the frame.

6. The computer-implemented method of claim 1, wherein at least one of the one or more strain gauges are mounted on a bridge portion of the frame.

7. The computer-implemented method of claim 1, wherein the AR system comprises a head-worn AR apparatus.

8. An AR system comprising: a frame; one or more strain gauges operable to measure strains of the frame; one or more imaging devices mounted to the frame; one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the AR system to perform operations comprising: capturing, using the one or more imaging devices, tracking video frame data of a real-world scene; measuring, by one or more processors, using the one or more strain gauges of the AR system, strain data of the strains of the frame of the AR system as the tracking video frame data is captured; generating, by the one or more processors, a corrected frame model of the frame based on the strain data, the tracking video frame data, and a frame model of the frame; and generating, by the one or more processors, corrected tracking data based on the corrected frame model and the tracking video frame data.

9. The AR system of claim 8, wherein the AR system further comprises an optical engine mounted to the frame, and wherein the instructions when executed by the one or more processors further cause the AR system to perform operations comprising: generating virtual overlay data based on the corrected tracking data; generating corrected virtual overlay video frame data based on the corrected frame model and the virtual overlay data; and providing, using the optical engine, a virtual overlay to a user of the AR system based on the corrected virtual overlay video frame data.

10. The AR system of claim 8, wherein the instructions that, when executed by the one or more processors, cause the AR system to perform operations of generating the corrected frame model of the frame further cause the AR system to perform operations comprising: generating visual inertial odometry data based on the tracking video frame data, the strain data, and the frame model; and generating the corrected frame model based on the visual inertial odometry data and the frame model.

11. The AR system of claim 10, wherein the instructions that, when executed by the one or more processors, cause the AR system to perform operations of generating the visual inertial odometry data further cause the AR system to perform operations comprising: generating yaw bending data of the frame based on the strain data; and generating the visual inertial odometry data based on the tracking video frame data, the yaw bending data, and the frame model.

12. The AR system of claim 11, wherein the instructions that, when executed by the one or more processors, cause the AR system to perform operations comprising generating the visual inertial odometry data further cause the AR system to perform operations comprising: determining, by the one or more processors, using an inertial measurement unit of the frame, inertial movement data of the frame as the tracking video frame data is captured; generating initial inertial odometry data based on the inertial movement data; andgenerating the visual inertial odometry data based on the initial inertial odometry data, the tracking video frame data, the strain data, and the frame model of the frame.

13. The AR system of claim 8, wherein at least one of the one or more strain gauges are mounted on a bridge portion of the frame.

14. The AR system of claim 8, wherein the AR system comprises a head-worn AR apparatus.

15. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising: capturing, using one or more imaging devices of an AR system, tracking video frame data of a real-world scene; measuring, by one or more processors, using one or more strain gauges of the AR system, strain data of strains of a frame of the AR system as the tracking video frame data is captured; generating, by the one or more processors, a corrected frame model of the frame based on the strain data, the tracking video frame data, and a frame model of the frame; and generating, by the one or more processors, corrected tracking data based on the corrected frame model and the tracking video frame data.

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions when executed by the computer further cause the computer to perform operations comprising: generating virtual overlay data based on the corrected tracking data; generating corrected virtual overlay video frame data based on the corrected frame model and the virtual overlay data; and providing, using an optical engine of the AR system, a virtual overlay to a user of the AR system based on the corrected virtual overlay video frame data.

17. The non-transitory computer-readable storage medium of claim 15, wherein the instructions that, when executed by the computer, cause the computer to perform operations of generating the corrected frame model of the frame further cause the computer to perform operations comprising:generating visual inertial odometry data based on the tracking video frame data, the strain data, and the frame model; and generating the corrected frame model based on the visual inertial odometry data and the frame model.

18. The non-transitory computer-readable storage medium of claim 17, wherein the instructions that, when executed by the computer, cause the computer to perform operations of generating the visual inertial odometry data further cause the computer to perform operations comprising: generating yaw bending data of the frame based on the strain data; and generating the visual inertial odometry data based on the tracking video frame data, the yaw bending data, and the frame model.

19. The non-transitory computer-readable storage medium of claim 18, wherein the instructions that, when executed by the computer, cause the computer to perform operations comprising generating the visual inertial odometry data further cause the computer to perform operations comprising: determining, using an inertial measurement unit of the frame, inertial movement data of the frame as the tracking video frame data is captured; generating initial inertial odometry data based on the inertial movement data; and generating the visual inertial odometry data based on the initial inertial odometry data, the tracking video frame data, the strain data, and the frame model of the frame.

20. The non-transitory computer-readable storage medium of claim 15, wherein the AR system comprises a head-worn AR apparatus.

Images