Ar system burging correction

Abstract

A system for distortion or bend correction in an augmented reality (AR) system. A sensor is positioned in a frame of the head-mounted AR system to sense a force or pressure acting on the frame through temples attached to the frame. The sensed force or pressure is used in conjunction with a model of the frame to determine a corrected model of the frame. The corrected model is used to correct video data captured by the AR system and to correct a video virtual overlay provided to a user wearing the head-mounted AR system.

Description

[0001] Priority claim

[0002] This application claims the benefit of U.S. patent application serial number 17 / 937,950, filed on October 4, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to user interfaces, and more particularly to user interfaces used in augmented reality and virtual reality. Background Art

[0004] The head-mounted device can be implemented by a transparent or translucent display, through which the user of the head-mounted device can view the surrounding environment or real-world scene. Such a device enables the user to view the real-world scene through the transparent or translucent display, and can also see the objects generated for display (e.g., virtual objects such as renderings of 2D or 3D graphic models, images, videos, text, etc.), which appear as part of the real-world scene and / or are superimposed on the real-world scene. This is generally referred to as "augmented reality" or "AR". The head-mounted device can also completely block the user's field of view and display a virtual environment through which the user can move or be moved. This is generally referred to as "virtual reality" or "VR". In a hybrid form, an imaging device is used to capture a view of the real-world scene, and then the view, together with the enhancement, is displayed to the user on a display that blocks the user's eyes. As used herein, unless the context indicates otherwise, the term AR refers to augmented reality, virtual reality, and any mixture of these technologies.

[0005] A user of a head mounted device can access and use computer software applications to perform various tasks or participate in entertainment activities. Performing tasks or participating in entertainment activities may require inputting various commands into the head mounted device. Therefore, it is desirable to have a mechanism for inputting commands. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] To easily identify the discussion of any particular element or action, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0007] Figure 1 is a stereoscopic diagram of a head mounted device according to some examples.

[0008] Figure 2 According to some examples Figure 1 Another view of the headset.

[0009] Figure 3AMisalignment errors caused by yaw bending of a head-mounted AR system in response to lateral forces acting on a frame of the head-mounted AR system are shown, according to some examples.

[0010] Figure 3B Misalignment errors caused by pitch movement of a head-mounted AR system are shown according to some examples.

[0011] Figure 4 is an illustration of the arrangement of sensors on a frame of a head-mounted AR system according to some examples.

[0012] Figure 5 is an illustration of a sensor arrangement on a frame of a head-mounted AR system during vertical loading by temple pieces attached to the frame, according to some examples.

[0013] Figure 6 is an illustration of a sensor arrangement on a frame of a head-mounted AR system during lateral loading of temple pieces attached to the frame, according to some examples.

[0014] Fig. 7A is a flow chart of a frame curvature correction method used by a head-mounted AR system according to some examples.

[0015] Figure 7B is a collaboration diagram of components of an AR system according to some examples.

[0016] Figure 8 is a diagrammatic representation of a machine according to some examples within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

[0017] Fig. 9 is a block diagram showing a networked system including details of a head-mounted AR system according to some examples.

[0018] Fig.10 is a block diagram illustrating a software architecture within which the present disclosure may be implemented, according to some examples.

[0019] Fig.11 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network according to some examples. DETAILED DESCRIPTION

[0020] Understanding the spatial relationships of the system components of a head-mounted AR device is useful for generating accurate virtual overlays for the AR experience. An ergonomic and visually appealing head-mounted AR device frame design provides lightweight glasses. However, such a design may be less rigid, and this may cause the spatial relationships between different components of the head-mounted AR device (such as a display, an imaging device, an inertial measurement unit, and a projector) to change over time. During normal operation, such relationships may also change by the user simply putting on the head-mounted AR device, walking, or touching the frame of the head-mounted device. This may result in an incorrect sensing of the surrounding world (such as stereo depth estimation), which leads to an unrealistic AR experience. The spatial relationships between components are determined during factory calibration, which allows for the creation of accurate enhancements for the AR experience; however, if the spatial relationships between components change during operation, the factory calibration becomes invalid, and the quality of the enhancement is reduced.

[0021] In some examples, one or more force or pressure sensors are mounted on the temple hinges of the head-mounted AR system to measure the forces acting on the frame of the head-mounted AR system. This allows for the direct measurement of the AR system frame dynamics to infer spatial relationships. This allows for the compact integration of sensors, enabling a more flexible and ergonomic frame design while maintaining a near-optimal AR experience.

[0022] In some examples, the AR system includes: a frame; one or more sensors operable to sense forces acting on the frame; one or more imaging devices mounted to the frame; an attitude component that measures the attitude and position of the frame; and an optical engine mounted to the frame. During operation, the AR system captures sensor data of the forces acting on the frame and generates a corrected frame model of the frame based on the sensor data and a physical model of the frame. The AR system uses the imaging devices to capture tracking video frame data of one or more physical objects in the real-world scene being viewed or interacted with by the user of the AR system. While the AR system captures the tracking video frame data, the AR system uses the attitude component to capture attitude and position data. The AR system generates tracking data based on the corrected frame model, the tracking video frame data, and the attitude and position data.

[0023] In some examples, the attitude component includes one or more inertial measurement units (IMUs) for determining attitude and position data.

[0024] In some examples, the attitude component includes a global positioning system (GPS) sensor and one or more IMUs. The AR system combines data from the GPS sensor and the one or more IMUs to generate attitude and position data.

[0025] In some examples, the pose component includes one or more IMUs for generating pose and position data. The AR system generates tracking data based on the calibrated frame model and the pose and position data without using tracking video frame data.

[0026] In some examples, the AR system generates virtual overlay data based on the tracking data, generates virtual overlay video frame data based on the corrected frame model and the virtual overlay data, and provides virtual overlay to the user based on the virtual overlay video frame data using an optical engine.

[0027] Other technical features may be readily apparent to those skilled in the art from the following drawings, descriptions and claims.

[0028] Figure 1 is a head mounted AR system according to some examples (e.g., Figure 1 A stereoscopic image of glasses 100).

[0029] As used herein, unless an alternative meaning is indicated, directional terms such as, but not limited to, “up,” “upper,” “down,” “lower,” “vertical,” “horizontal,” “lateral,” “left,” “right,” “forward,” and “backward” are to be interpreted based on the perspective of a user wearing a head-mounted AR system (such as glasses 100).

[0030] The glasses 100 may include a frame 102 made of any suitable material (e.g., plastic or metal, including any suitable shape memory alloy). In one or more examples, the frame 102 includes a first or left optical element holder 104 (e.g., a display or lens holder) and a second or right optical element holder 106 connected by a bridge 112. A first or left optical element 108 and a second or right optical element 110 may be disposed within the respective left and right optical element holders 104, 106. The right and left optical elements 110, 108 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 100.

[0031] The frame 102 additionally includes a left arm or temple piece 122 and a right arm or temple piece 124. In some examples, the frame 102 can be formed from a single piece of material to have a unitary or unitary construction.

[0032] The glasses 100 may include a computing system such as a computer 120, which may be of any suitable type so as to be carried by the frame 102, and in one or more examples, the computer 120 may be of a suitable size and shape to be partially disposed in one of the left temple piece 122 or the right temple piece 124. The computer 120 may include multiple processors, memories, and various communication components that share a common power source. As discussed below, the various components of the computer 120 may include low-power circuitry, high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of various aspects of the computer 120 may be implemented as shown in the data processor 902 discussed below.

[0033] The computer 120 additionally includes a battery 118 or other suitable portable power supply. In some examples, the battery 118 is disposed in the left temple piece 122 and is electrically coupled to the computer 120 disposed in the right temple piece 124. The glasses 100 may include a connector or port (not shown) suitable for charging the battery 118, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0034] The glasses 100 include one or more imaging devices, such as, but not limited to, a first or left imaging device 114 and a second or right imaging device 116. In some examples, the one or more imaging devices of the glasses 100 include an imaging sensor and an optical component, such as, but not limited to, a camera, etc. In some examples, the imaging sensor senses electromagnetic radiation in the visible spectrum. In some examples, the imaging sensor senses electromagnetic radiation in the infrared spectrum. In some examples, the glasses 100 also include one or more light sources, such as light emitting diodes (LEDs). In some examples, one or more LEDs of the AR system operate in the infrared light frequency range.

[0035] In some examples, one or more imaging devices of glasses 100 include one or more laser imaging, detection, and ranging (LIDAR) devices.

[0036] In one or more examples, the glasses 100 include any number of input sensors or other input / output devices in addition to the left imaging device 114 and the right imaging device 116. Such sensors or input / output devices may additionally include biometric sensors, positioning sensors, motion sensors, etc.

[0037] In some examples, left imaging device 114 and right imaging device 116 provide video frame data for use by glasses 100 to extract 3D information from a real-world scene being viewed by a user of glasses 100 .

[0038] The glasses 100 may also include a touchpad 126 mounted to or integrated with one or both of the left temple piece 122 and the right temple piece 124. The touchpad 126 is typically arranged vertically, and in some examples, the touchpad is approximately parallel to the temple of the user. As used herein, generally vertically aligned means that the touchpad is within a vertical range that is more vertical than horizontal. Additional user input may be provided by one or more buttons 128, which in the example shown are disposed on the outer upper edges of the left optical element holder 104 and the right optical element holder 106. The one or more touchpads 126 and buttons 128 provide a means by which the glasses 100 can receive input from a user of the glasses 100.

[0039] The glasses 100 may also include one or more IMUs 134 configured to measure the physical orientation (e.g., attitude and position) of the frame 102 and generate IMU data including the measured attitude and position. In some examples, the IMU 134 is operable to measure the rotation angles of the frame 102 around the pitch rotation axis, the roll rotation axis, and the yaw rotation axis. In some examples, the IMU 134 is operable to measure the rotational movement of the frame 102 around the pitch rotation axis, the roll rotation axis, and the yaw rotation axis and the translational movement of the frame 102 in a 3D space such as a real-world scene.

[0040] The glasses 100 may also include a GPS sensor 136 configured to receive and process global positioning signals to determine the physical location of the frame 102 and generate GPS location data based on the physical location.

[0041] Figure 2 The glasses 100 are shown from the perspective of a user wearing the glasses 100. For clarity, Figure 1 Several elements shown in FIG. have been omitted. Figure 1 As described in Figure 2 The eyeglasses 100 shown in FIG. 1 include a left optical element 108 and a right optical element 110 secured within a left optical element holder 104 and a right optical element holder 106, respectively.

[0042] The glasses 100 include a forward optical assembly 202 including a right projector 204 and a right near-eye display 206 , and a forward optical assembly 210 including a left projector 212 and a left near-eye display 216 .

[0043] In some examples, the right near-eye display 206 and the left near-eye display 216 are waveguides. The waveguides include reflective structures or diffractive structures (e.g., gratings and / or optical elements such as mirrors, lenses, or prisms). Light 208 emitted by the right projector 204 encounters the diffractive structure of the waveguide of the right near-eye display 206, which directs the light toward the right eye of the user to provide an image on or in the right optical element 110 that is superimposed with the view of the real-world scene seen by the user. Similarly, light 214 emitted by the left projector 212 encounters the diffractive structure of the waveguide of the left near-eye display 216, which directs the light toward the left eye of the user to provide an image on or in the left optical element 108 that is superimposed with the view of the real-world scene seen by the user. The combination of the GPU, the forward optical assembly 202, the left optical element 108, and the right optical element 110 provides the optical engine of the glasses 100. The glasses 100 use an optical engine to generate an overlay of the user's view of a real-world scene to a user of the glasses 100 , including a display of a user interface.

[0044] However, it will be appreciated that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's field of view. For example, an LCD, LED or other display panel or surface may be provided instead of the right projector 204 and waveguide.

[0045] In use, information, content, and various user interfaces will be presented to the user of the glasses 100 on the near-eye display. As described in more detail herein, the user can then use the touch pad 126 and / or buttons 128, associated devices (e.g., Fig. 9 The user may interact with the glasses 100 through voice input or touch input on a client device 926 shown in FIG. 1 , and / or hand movements, positions, and gestures detected by the glasses 100 .

[0046] In some examples, the glasses 100 are operably connected to a client device, such as a computer, smart phone, etc., which provides additional computing resources that provide additional functionality to the glasses 100. For example, the glasses 100 recognize simple user interactions by the user, but utilize the computing resources of the client device to recognize more complex interactions, such as swipe gestures on a touch screen or hand gestures captured by an imaging device.

[0047] Figure 3A and Figure 3BDepth misalignment errors caused by yaw bending and pitch movement of a head-mounted AR system (e.g., glasses 100) in response to lateral and vertical forces acting on the frame of the head-mounted AR system are shown according to some examples. When worn by a user, the head-mounted AR system (e.g., glasses 100) experiences optical misalignment errors caused by deformation or bending of the frame 102 of the head-mounted AR system. When the user places the head-mounted AR system on their head, the temple pieces (e.g., left temple piece 122 and right temple piece 124) are tightened by opposing lateral forces 308 and lateral forces 310, causing the frame 102 to bend along its length, which is referred to as "yaw bending" in this article, as indicated by bend lines 312a, 312b, 312c, and 312d. In addition, when the glasses 100 are subjected to vertical forces on one or both of the temple pieces of the glasses 100, the glasses 100 are subjected to pitch movement. Yaw bending and pitch movement can cause misalignment errors in the optical components of the AR glasses 100. These misalignment errors can cause tracking errors when the AR system generates tracking data, and cause misalignment between the virtual overlays that the AR system provides to the user and the physical objects and features of the real-world scene that the user is viewing when wearing the head-mounted AR system.

[0048] The yaw bending may cause the left imaging device 114 and the right imaging device 116 to experience yaw motion. The yaw motion may cause the left optical axis 314 of the left imaging device 114 to become misaligned, as indicated by the misaligned left optical axis 318. The yaw motion may also cause the right optical axis 316 of the right imaging device 116 to become misaligned, as indicated by the misaligned right optical axis 320. When using the video frame data of the imaging devices to stereoscopically determine the position of the physical feature 302 in the real-world scene, the system introduces a depth or Z error 306 on the Z axis because the physical feature 302 was determined to be at a different position when the AR system generated tracking data for the feature in the real-world scene, and thus appeared to be a distinct physical feature 304. In a similar manner, when a virtual object of a virtual overlay of an AR experience is rendered in the video frame data and provided to a user of the head-mounted AR system, the user will experience a misalignment of the provided virtual object with the real-world scene.

[0049] In the correctly aligned video frame 322 of the virtual overlay, the AR system correctly displays the virtual object 324 aligned with the real world scene feature 326. The misaligned video frame 330 causes the AR system to display the virtual object 328 in an incorrect position (with respect to the real world scene feature 326).

[0050] Figure 4is an illustration of a sensor arrangement on a frame 102 of eyeglasses 100 according to some examples. The frame 102 includes a forward-facing left imaging device 114 having an optical axis 412 extending forward from the left imaging device 114. The frame 102 is attached to a left temple piece 122 by a left hinge spine 410, thereby forming a left hinge 132, wherein the frame 102 acts as a first leaf or frame hinge leaf of the left hinge 132, and the left temple piece 122 acts as a second leaf or temple hinge leaf of the left hinge 132.

[0051] One or more sensors (e.g., upper sensor 402 and lower sensor 404) are mounted at a location on the left outer portion of frame 102 and have one or more corresponding sensing surfaces aligned with frame hinge leaf surface 406, so that when left hinge 132 is closed, temple hinge leaf surface 408 of left temple piece 122 impacts on the corresponding sensing surfaces of upper sensor 402 and lower sensor 404 and exerts a force or pressure on lower sensor 404 and upper sensor 402. In some examples, at least two sensors are used. In some examples, at least two sensors are mounted in a vertically spaced arrangement.

[0052] In some examples, the temple hinge leaf surface 408 includes corresponding one or more protrusions that impinge on a sensing surface of the one or more sensors. In some examples, the one or more sensing surfaces of the one or more sensors are convex. In some examples, an outermost portion of the one or more sensing surfaces of the one or more sensors extends beyond the frame hinge leaf surface 406.

[0053] In some examples, one or more sensors (e.g., an upper sensor and a lower sensor) are mounted in the left temple piece 122 and have one or more corresponding sensing surfaces aligned with the temple hinge leaf surface 408, such that when the left hinge 132 is closed, one or more portions of the frame hinge leaf surface 406 of the left temple piece 122 impact on the corresponding sensing surfaces of the upper sensor and the lower sensor and exert a force or pressure on the one or more sensors. In some examples, the frame hinge leaf surface 406 includes one or more corresponding protrusions that impact on the sensing surface of one or more sensors mounted in the left temple piece 122. In some examples, the one or more sensing surfaces of the one or more sensors are convex. In some examples, the outermost portion of the one or more sensing surfaces of the one or more sensors extends beyond the temple hinge leaf surface 408.

[0054] In some examples, the right outer portion of the frame 102 includes a forward-facing right imaging device 116 having an optical axis extending forward from the right imaging device 116. (See Figure 2 and Figure 3A ). The frame 102 is attached to the right temple piece 124 by a right hinge ridge (not shown), thereby forming a right hinge 130, wherein the frame 102 acts as a first leaf or frame hinge leaf of the right hinge 130, and the right temple piece 124 acts as a second leaf or temple hinge leaf of the right hinge 130. One or more sensors (not shown) are mounted at a position on the right outer portion of the frame 102 and have one or more corresponding sensing surfaces aligned with the frame hinge leaf surface (not shown), so that when the right hinge 130 is closed, the temple hinge leaf surface (not shown) of the right temple piece 124 impacts the corresponding sensing surface of the one or more sensors and exerts a force or pressure on the one or more sensors. In some examples, the temple hinge leaf surface includes corresponding one or more protrusions that impact the corresponding sensing surface of the one or more sensors. In some examples, at least two sensors are used. In some examples, at least two sensors are mounted in a vertically spaced arrangement.

[0055] In some examples, one or more sensors (e.g., an upper sensor and a lower sensor) are mounted in the right temple piece 124 and have one or more corresponding sensing surfaces aligned with the temple hinge leaf surface such that when the right hinge 130 is closed, one or more portions of the frame hinge leaf surface of the frame 102 impact against the corresponding sensing surfaces of the one or more sensors and exert a force or pressure on the one or more sensors.

[0056] In some examples, one or more sensors are mounted in the right temple piece 124 and / or the left temple piece 122 of the eyeglasses 100 and one or more sensors are mounted in the frame.

[0057] In some examples, a sensor in the one or more sensors includes a force sensor (eg, a load cell, etc.) that senses a force acting on a sensing surface of the force sensor.

[0058] In some examples, a sensor in the one or more sensors includes a pressure sensor that senses pressure acting on a sensing surface of the pressure sensor.

[0059] In some examples, a sensor in the one or more sensors includes: a force bearing member (eg, a diaphragm or a cantilever beam) integral to the frame 102; and a strain gauge operably connected to a surface of the force bearing member.

[0060] In some examples, a mix of sensor types are used.

[0061] In some examples, one or more sensors are mounted on a temple piece of the head-mounted AR system and have corresponding sensing surfaces aligned with temple hinge leaf surfaces of the temple piece. When the corresponding hinge is closed, the corresponding frame hinge leaf surface impacts the corresponding sensing surface of the one or more sensors.

[0062] In some examples, a portion of one or more sensors is mounted on a frame of the AR system, and a portion of one or more sensors is mounted on a temple piece of the AR system.

[0063] In some examples, one or more sensors are mounted on an interior portion of frame 102. In some examples, one or more sensors are mounted on a middle portion of frame 102.

[0064] Figure 5 1 is an illustration of a sensor arrangement on the frame 102 during vertical loading of the temple piece 122 of the frame 102 attached to a head-mounted AR system (such as glasses 100) according to some examples. When a force or pressure acts on the left temple piece 122 in a vertical direction (e.g., vertical force 504), the vertical force 504 causes unequal forces to act on the upper sensor 402 and the lower sensor 404. For example, when the vertical force 504 acts on the left temple piece 122, the left temple piece 122 behaves similar to a first lever arm of a lever having a fulcrum at the left hinge ridge 410, so that the temple hinge leaf surface 408 applies unequal forces or pressures to the sensing surfaces of the upper sensor 402 and the lower sensor 404. The unequal forces cause the frame 102 to experience a pitch movement 502. The pitch movement 502 causes a change in the pitch angle of the left imaging device 114 and the optical axis 412 of the left imaging device 114.

[0065] In some examples, in the case of an upward vertical force, upper sensor 402 will experience a higher upper sensor force or pressure 506 than lower sensor force or pressure 508 experienced by lower sensor 404. In the case of a downward vertical force, upper sensor 402 will experience a lower upper sensor force or pressure 506 than lower sensor 404 will experience a lower lower sensor force or pressure 508.

[0066] In some examples, the direction of the pitch movement 502 is determined by determining the value of the force or pressure difference resulting from subtracting the value of the lower sensor force or pressure 508 captured by the lower sensor 404 from the value of the upper sensor force or pressure 506 captured by the upper sensor 402. If the resulting value is greater than zero, the pitch movement 502 is in the downward direction (i.e., the change in the pitch angle of the optical axis 412 of the left imaging device 114 is a negative value). In a similar manner, if the resulting value is less than zero, the pitch movement 502 is in the upward direction (i.e., the change in the pitch angle of the optical axis 412 of the left imaging device 114 is a positive value).

[0067] In some examples, the magnitude of the change in the pitch angle of the optical axis 412 of the left imaging device 114 is determined based on the absolute value of the force or pressure difference between the value of the upper sensor force or pressure 506 captured by the upper sensor 402 and the value of the lower sensor force or pressure 508 captured by the lower sensor 404.

[0068] In some examples, the right temple piece 124 of the eyeglasses 100 and the right portion of the frame 102 include a sensor and hinge arrangement similar to the frame 102 and the left temple piece 122. Thus, the direction and magnitude of the pitch movement can be determined based on the value of the lower sensor force or pressure captured by the lower right sensor (not shown) and the value of the upper sensor force or pressure captured by the upper right sensor (not shown). In some examples, the magnitude of the change in the pitch angle of the optical axis of the right imaging device 116 is determined based on the absolute value of the force or pressure difference between the value of the upper sensor force or pressure captured by the upper right sensor and the value of the lower sensor force or pressure captured by the lower right sensor.

[0069] Figure 6 1 is an illustration of a sensor arrangement on a frame 102 during lateral loading of a temple piece 122 of a frame 102 attached to a head-mounted AR system (such as glasses 100) according to some examples. When a force or pressure acts on the left temple piece 122 in a lateral direction (e.g., lateral force 602), and the frame 102 is prevented from rotating, for example, by an opposing lateral force acting on a right temple piece (not shown) attached to the frame 102, the lateral force 602 causes an upper sensor force or pressure 506 and a lower sensor force or pressure 508 to act on the upper sensor 402 and the lower sensor 404, respectively. Such opposing lateral forces may occur when a user wears the head-mounted AR system. For example, when the lateral force 602 acts on the left temple piece 122, the left temple piece 122 behaves similarly to a first lever arm of a lever having a fulcrum at the left hinge ridge 410, such that the temple hinge leaf surface 408 applies a force or pressure to the sensing surfaces of the upper sensor 402 and the lower sensor 404. This results in a torque force about the vertical axis of the left hinge spine 410, which causes the left portion of the frame 102 to experience a yaw motion 604 as the frame 102 deforms or bends in response to the lateral forces acting on the left temple piece 122. The yaw motion 604 results in a change in the yaw angle of the optical axis 412 and the left imaging device 114.

[0070] In some examples, the magnitude of the change in the yaw angle of the optical axis 412 of the left imaging device 114 is determined based on the sum of: a value of an upper sensor force or pressure 506 captured by the upper sensor 402, a value of a lower sensor force or pressure 508 captured by the lower sensor 404, and a physical model of the frame 102, which includes values ​​of the stiffness of the frame 102 and the length of the frame 102.

[0071] In some examples, the magnitude of the yaw force or pressure of frame 102 is determined based on the sum of the value of upper sensor force or pressure 506 captured by upper sensor 402 and the value of lower sensor force or pressure 508 captured by lower sensor 404 .

[0072] In some examples, the right portion of the frame 102 of the glasses 100 and the right temple piece 124 include a sensor and hinge arrangement similar to the frame 102 and the left temple piece 122. Thus, the magnitude of the yaw force or pressure on the frame 102 is determined based on the sum of the value of the upper sensor force or pressure captured by the upper right sensor (not shown) and the value of the lower sensor force or pressure captured by the lower right sensor (not shown).

[0073] Fig. 7A is a flow chart of a frame curvature correction method 700 used by a head-mounted AR system (such as glasses 100) to correct for curvature of a frame (e.g., frame 102) of the AR system, and Figure 7B 744 is a collaboration diagram of components of an AR system according to some examples. The AR system uses the components shown in the collaboration diagram 744 to perform the frame warp correction method 700 to generate tracking data 740 and virtual overlay video frame data 736 for an AR application 718.

[0074] In operation 702, the AR system captures sensor data 734 from one or more sensors 730 (e.g., one or more of the upper sensor 402, the lower sensor 404, the upper right sensor, and the lower right sensor) mounted on the frame 102 of the glasses 100. The sensor data 734 includes data of measured values ​​of force or pressure acting on the one or more sensors 730 by the corresponding left temple piece 122 or right temple piece 124 when the user wears the head-mounted AR system.

[0075] In operation 704, the AR system uses the frame model correction component 722 to generate corrected frame model data 738 based on the sensor data 734 and the frame physical model 716. The frame physical model 716 includes data of a frame model, which includes a geometric model of the frame that defines the positions and geometric relationships between the various components of the frame 102. For example, the frame physical model 716 includes data of the geometric relationships between one or more imaging devices 724 (e.g., the left imaging device 114 and the right imaging device 116 of the glasses 100). The frame physical model 716 also includes data of the positions and geometric relationships between components of the optical engine 726 (e.g., the left projector 212 and the left optical element 108 of the glasses 100), and data of the positions and geometric relationships between the right projector 204 and the right optical element 110 of the glasses 100. The frame physical model 716 also includes data of the positions and geometric relationships of the following devices, which include posture components, such as one or more IMUs, GPS sensors, etc. The frame physical model 716 also includes data on frame bending stiffness values ​​or stiffness values ​​of the frame 102 at various locations along the frame (e.g., the bending stiffness of the frame when yaw bending force and pitch movement force act on the frame 102 through the left temple piece 122 and the right temple piece 124 of the glasses 100).

[0076] In some examples, the data of the frame physical model 716 (including the frame bending stiffness values ​​and the frame model) is generated based on a finite element analysis of the frame as designed. In some examples, the data of the frame physical model 716 (including the frame bending stiffness values ​​and the frame model) is determined by testing and calibration of the frame as constructed. In some examples, the data of the frame physical model 716 (including the frame bending stiffness values ​​and the frame model) is generated based on a combination of a finite element analysis of the frame as designed and testing and calibration of the frame as constructed.

[0077] The frame model correction component 722 generates corrected frame model data 738 by calculating the bending of the frame 102 using data of the value of the force or pressure acting on the one or more sensors 730 (which is included in the sensor data 734) and data of the frame bending stiffness value of the frame physical model 716. The corrected frame model data 738 includes data of the corrected geometric relationship of the components of the glasses 100. In some examples, the corrected frame model data 738 includes data of the corrected geometric relationship between one or more imaging devices 724 (e.g., the left imaging device 114 and the right imaging device 116 of the glasses 100) when the frame 102 is subjected to the measured force or pressure. In some examples, the corrected frame model data 738 includes data of the corrected geometric relationship between the left projector 212 and the left optical element 108, and data of the corrected geometric relationship between the right projector 204 and the right optical element 110 of the glasses 100. In some examples, corrected frame model data 738 includes data of corrected geometric relationships between components of frame 102 and devices including attitude components (e.g., one or more IMUs 134, GPS sensors 136, etc.).

[0078] In some examples, the frame physics model 716 includes a function for calculating a corrective spatial relationship between two or more components of the frame 102 based on the sensor data 734. For example, the two or more components may include any combination of frame components (e.g., one or more imaging devices, one or more IMUs, one or more projectors, one or more optical elements, one or more GPS sensors, etc.). In some examples, the function is encoded in the frame physics model 716.

[0079] In operation 706, the AR system uses one or more imaging devices 724 to capture tracking video frame data 732 of the real world scene being viewed by the user of the AR system, and uses a pose component 748 to capture pose and position data 746 of the frame 102. The tracking video frame data 732 includes video frame data of physical objects in the real world scene being viewed by the user of the AR system. The pose and position data 746 includes pose data of the frame 102, which includes the orientation of the frame 102 and the position information of the frame.

[0080] In some examples, the attitude component 748 includes one or more IMUs 134. The attitude component 748 uses one or more IMUs 134 to generate inertial motion data. The attitude component 748 generates attitude and position data 746 based on the inertial motion data. In some examples, the attitude component 748 receives tracking video frame data 750 from the imaging device 724, and uses the inertial motion data from one or more IMUs 134 of the framework 102, and uses a visual inertial odometer (VIO) method to generate attitude and position data 746 based on the tracking video frame data 732 and the inertial motion data. In some examples, the attitude component includes a GPS sensor that generates GPS position data. The attitude component 748 generates attitude and position data 746 based in part on the GPS position data.

[0081] In operation 708, the AR system generates tracking data 740 based on the corrected frame model data 738, the tracking video frame data 732, and the pose and position data 746 using the tracking component 720. For example, the tracking component 720 extracts features of the physical object in the tracking video frame data 732. In some examples, the tracking component 720 extracts features from the tracking video frame data 732 using a computer vision method, including but not limited to Harris corner detection, Shi-Tomasi corner detection, scale-invariant feature transform (SIFT), speeded up robust features (SURF), accelerated segment test features (FAST), oriented FAST, and rotational BRIEF (ORB), etc.

[0082] The tracking component 720 determines the relative distance between the feature extracted from the tracking video frame data 732 and the frame 102 of the AR system based on the tracking video frame data 732 and the corrected frame model data 738. For example, when two imaging devices are used to capture the tracking video frame data 732, the distance between the imaging devices mounted on the frame 102 and the angle of the optical axes of the two imaging devices are determined based on the corrected frame model data 738. The distance between the frame 102 and the feature can be determined by triangulation based on the distance between the imaging devices and the tracking video frame data 732. The corrected frame model data 738 includes the corrected distance between the imaging devices mounted on the frame and the angle of the corrected optical axis of the imaging devices. Using the corrected distance and optical axis angle of the imaging devices reduces the error of the determined distance from the feature to the frame 102 of the AR system. The AR system maps the feature to a 3D model of the real world scene according to a 3D coordinate system (e.g., a 3D Cartesian coordinate system or a 3D polar coordinate system) based on the pose and position data 746 and the relative distance between the frame 102 and the mapped feature.

[0083] In some examples, when an imaging device is used to capture tracking video frame data 732 , the relative distance between the feature and frame 102 can be determined using the angle of the imaging device's optical axis and the assumed physical size of the feature.

[0084] In operation 710, an AR application 718 of the AR system generates virtual overlay data 742 using tracking data 740. The virtual overlay data 742 includes data of virtual objects generated by the AR application 718, which are used to create a virtual overlay that is provided to a user of the AR system using the optical engine 726. The virtual objects are mapped into a 3D model of a real-world scene using the tracking data 740, so that when the virtual objects rendered into video frame data are provided to the user in a display by the optical engine 726, the virtual objects will appear to be located in the real-world scene, with a specific relationship to the features of the physical objects identified in the tracking data 740. For example, the virtual overlay may include a user interface composed of virtual objects, and the user interacts with these virtual objects using the user's hands. The virtual objects will appear in the virtual overlay and in obvious locations that are near the location of the user's hands as determined from the tracking data 740, so that the user can reach the virtual objects and interact with them.

[0085] In operation 712, the virtual overlay rendering component 728 of the AR system generates virtual overlay video frame data 736 by rendering the virtual objects of the virtual overlay into the video frame data using the virtual overlay data 742 and the corrected frame model data 738. The virtual overlay video frame data 736 will be provided to the user in the display through the optical engine 726. The optical engine 726 includes a projector (e.g., the left projector 212 and the right projector 204 of the glasses 100) to project the image of the virtual overlay video frame data 736 onto the optical elements (e.g., the left optical element 108 and the right optical element 110 of the glasses 100). When force or pressure acts on the temple pieces and the frame of the glasses 100, misalignment may occur. The virtual overlay rendering component 728 uses the corrected frame model data 738 to correct the video frame data generated by rendering the virtual objects of the virtual overlay data 742 to solve the misalignment between the projector and the optical elements of the glasses 100.

[0086] In operation 714, the AR system uses the optical engine 726 to provide a virtual overlay to the user based on the virtual overlay video frame data 736. For example, one or more projectors of the glasses 100 project the image included in the virtual overlay video frame data 736 onto one or more optical elements of the glasses 100, and the user can see the virtual overlay superimposed on the real world scene viewable by the user through the optical elements.

[0087] Figure 8is a diagrammatic representation of a machine 800 within which instructions 810 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 800 to perform any one or more of the methodologies discussed herein. The machine 800 may be used as an AR system (such as Figure 1 The computer 120 of the glasses 100). For example, the instructions 810 can cause the machine 800 to perform any one or more of the methods described herein. The instructions 810 convert a general-purpose, unprogrammed machine 800 into a specific machine 800 that is programmed to perform the functions described and shown in the described manner. The machine 800 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 can operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 in conjunction with other components of the AR system may function as, but is not limited to: a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a head-mounted device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 810 specifying actions to be taken by the machine 800. In addition, while a single machine 800 is shown, the term "machine" may also be taken to include a collection of machines that individually or jointly execute instructions 810 to perform any one or more of the methodologies discussed herein.

[0088] The machine 800 may include a processor 802, a memory 804, and an I / O device interface 806 that may be configured to communicate with each other via a bus 844. In an example, the processor 802 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 808 and a processor 812 that execute instructions 810. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 8Multiple processors 802 are shown, but machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0089] The memory 804 includes a main memory 814, a static memory 816, and a storage unit 818, which are all accessible by the processor 802 via the bus 844. The main memory 804, the static memory 816, and the storage unit 818 store instructions 810 that implement any one or more of the methods or functions described herein. The instructions 810 may also reside, completely or partially, within the main memory 814, within the static memory 816, within a non-transitory machine-readable medium 820 within the storage unit 818, within one or more of the processors 802 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 800.

[0090] The I / O device interface 806 couples the machine 800 to the I / O devices 846. One or more of the I / O devices 846 may be components of the machine 800 or may be separate devices. The I / O device interface 806 may include various interfaces to the I / O devices 846 that the machine 800 uses to receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O device interface 806 included in a particular machine will depend on the type of machine. It should be understood that the I / O device interface 806, the I / O devices 846 may include Figure 8 806. In various examples, the I / O device interface 806 may include an output component interface 828 and an input component interface 832. The output component interface 828 may include an interface to a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component interface 832 may include an interface to an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides the location and / or force of a touch or touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0091] In another example, the I / O device interface 806 may include a biometric component interface 834, a motion component interface 836, an environment component interface 838, or a positioning component interface 840, as well as various other component interfaces. For example, the biometric component interface 834 may include an interface to a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component interface 836 may include an interface to an inertial measurement unit (IMU), an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component interface 838 may include, for example, an interface to an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that may provide indications, measurements, or signals associated with the surrounding physical environment. The positioning component interface 840 includes an interface to a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0092] A variety of technologies may be used to achieve communication. The I / O device interface 806 also includes a communication component interface 842 that is operable to couple the machine 800 to a network 822 or a device 824 via corresponding couplings 830 and couplings 826. For example, the communication component interface 842 may include an interface to a network interface component or another suitable device that interfaces with the network 822. In another example, the communication component interface 842 may include an interface to a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a Bluetooth component (e.g., Bluetooth low energy), a Wi-Fi component, and other communication components that provide communication via other modalities. The device 824 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0093] In addition, the communication component interface 842 may include an interface to a component operable to detect an identifier. For example, the communication component interface 842 may include an interface to a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a quick response (QR) code, an Aztec code, a data matrix, a data symbol (Dataglyph), a maximum code (MaxiCode), PDF417, Ultra Code, UCC RSS-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information may be obtained via the communication component interface 842, such as a location obtained via an Internet Protocol (IP) geolocation, a location obtained via Wi-Fi signal triangulation, a location obtained via detecting an NFC beacon signal that may indicate a specific location, and the like.

[0094] Various memories (e.g., memory 804, main memory 814, static memory 816, and / or memory of processor 802) and / or storage unit 818 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 810) when executed by processor 802 cause various operations to implement the disclosed examples.

[0095] The instructions 810 may be transmitted or received over the network 822 using a transmission medium via a network interface device (e.g., a network interface component included in the communication component interface 842) and using any one of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 810 may be transmitted or received via a coupling 826 (e.g., a peer-to-peer coupling) with a device 824 using a transmission medium.

[0096] Fig. 9is a block diagram showing a networked system 900 including details of the glasses 100 according to some examples. The networked system 900 includes the glasses 100, a client device 926, and a server system 932. The client device 926 may be a smart phone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 100 using a low-power wireless connection 936 and / or a high-speed wireless connection 934. The client device 926 is connected to the server system 932 via a network 930. The network 930 may include any combination of wired connections and wireless connections. The server system 932 may be one or more computing devices that are part of a service or network computing system. The client device 926 and any elements of the server system 932 and the network 930 may be connected using a wireless network, such as a wireless network or a wireless network. Fig.10 and Figure 8 The details of the software architecture 1004 or machine 800 described in the embodiment of the present invention are implemented.

[0097] The glasses 100 include a data processor 902, a display 910, one or more imaging devices 908, and additional input / output elements 916. The input / output elements 916 may include microphones, audio speakers, biometric sensors, additional sensors, pressure or force sensors, or additional display elements integrated with the data processor 902. Fig.10 and Figure 8 Examples of input / output elements 916 are further discussed. For example, input / output elements 916 may include any of the I / O device interfaces 806, including output component interface 828, motion component interface 836, etc. Figure 2 Examples of display 910 are discussed in . In the specific examples described herein, display 910 includes displays for the user's left eye and right eye.

[0098] Data processor 902 includes image processor 906 (eg, video processor), GPU and display driver 938, gesture component 940, interface 912, low power circuitry 904, and high speed circuitry 920. The components of data processor 902 are interconnected by bus 942.

[0099] The interface 912 refers to any source of user commands provided to the data processor 902. In one or more examples, the interface 912 is a physical button that sends a user input signal from the interface 912 to the low-power processor 914 when pressed. Immediate release after pressing such a button can be processed by the low-power processor 914 as a request to capture a single image, and vice versa. Pressing such a button for a first period of time can be processed by the low-power processor 914 as a request to capture video data when the button is pressed and end video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for a long period of time can capture a still image. In some examples, the interface 912 can be any mechanical switch or physical interface capable of accepting user input associated with a data request from the imaging device 908. In other examples, the interface 912 can have a software component, or can be associated with a command received wirelessly from another source, such as from the client device 926.

[0100] The image processor 906 includes circuitry for receiving signals from the imaging device 908 and processing those signals from the imaging device 908 into a format suitable for storage in the memory 924 or for transmission to the client device 926. In one or more examples, the image processor 906 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from the imaging device 908, and volatile memory used by the microprocessor in operation.

[0101] The low-power circuit system 904 includes a low-power processor 914 and a low-power wireless circuit system 918. These elements of the low-power circuit system 904 can be implemented as separate elements or can be implemented as part of a single system on a chip on a single IC. The low-power processor 914 includes logic for managing other elements of the glasses 100. As described above, for example, the low-power processor 914 can accept user input signals from the interface 912. The low-power processor 914 can also be configured to receive input signals or command communications from the client device 926 via the low-power wireless connection 936. The low-power wireless circuit system 918 includes circuit elements for implementing a low-power wireless communication system. Bluetooth Smart (also known as Bluetooth Low Energy) is a standard implementation of a low-power wireless communication system that can be used to implement the low-power wireless circuit system 918. In other examples, other low-power communication systems can be used.

[0102] High-speed circuit system 920 includes a high-speed processor 922, a memory 924, and a high-speed wireless circuit system 928. High-speed processor 922 can be any processor capable of managing high-speed communications and operations for any general-purpose computing system for data processor 902. High-speed processor 922 includes processing resources for managing high-speed data transmission over high-speed wireless connection 934 using high-speed wireless circuit system 928. In some examples, high-speed processor 922 executes an operating system such as a LINUX operating system or a program such as a UNIX operating system. Fig.10 The high-speed processor 922, which executes the software architecture of the data processor 902, manages data transmission with the high-speed wireless circuit system 928, in addition to any other duties. In some examples, the high-speed wireless circuit system 928 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit system 928 can implement other high-speed communication standards.

[0103] The memory 924 includes any storage device capable of storing imaging device data generated by the imaging device 908 and the image processor 906. Although the memory 924 is shown as being integrated with the high-speed circuitry 920, in other examples, the memory 924 may be a separate, independent element of the data processor 902. In some such examples, electrical wiring may provide a connection from the image processor 906 or the low-power processor 914 to the memory 924 through a chip including the high-speed processor 922. In other examples, the high-speed processor 922 may manage the addressing of the memory 924 so that the low-power processor 914 will initiate the high-speed processor 922 any time a read or write operation involving the memory 924 is needed.

[0104] The pose component 940 estimates the physical orientation or pose of the glasses 100. For example, the pose component 940 uses image data from the imaging device 908 and associated inertial data captured using the positioning component interface 840 and GPS location data to track the position and determine the pose of the glasses 100 relative to a reference frame (e.g., a real-world scene). The pose component 940 continuously collects and uses updated sensor data describing the movement of the glasses 100 to determine an updated three-dimensional pose of the glasses 100, which indicates changes in relative position and orientation relative to physical objects in the real-world scene. The pose component 940 allows the glasses 100 to perform visual placement of virtual objects relative to physical objects within the user's field of view via the display 910.

[0105] The GPU and display driver 938 may use the pose of the glasses 100 to generate frames of virtual content or other content to be presented on the display 910 when the glasses 100 are operating in an augmented reality mode. In this mode, the GPU and display driver 938 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 100, which reflects changes in the user's position and orientation relative to physical objects in the user's real-world scene.

[0106] One or more functions or operations described herein may also be performed in an application resident on the glasses 100 or on the client device 926 or on a remote server. For example, one or more functions or operations described herein may be performed by one of the applications 1006 (e.g., the messaging application 1046).

[0107] Fig.10 1000 is a block diagram illustrating a software architecture 1004 that may be installed on any one or more of the devices described herein. The software architecture 1004 is supported by hardware such as a machine 1002 including a processor 1020, a memory 1026, and an I / O component interface 1038. In this example, the software architecture 1004 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1004 includes layers such as an operating system 1012, a library 1008, a framework 1010, and an application 1006. In operation, the application 1006 invokes an API call 1050 through the software stack and receives a message 1052 in response to the API call 1050.

[0108] The operating system 1012 manages hardware resources and provides public services. The operating system 1012 includes, for example, a kernel 1014, a service 1016, and a driver 1022. The kernel 1014 is used as an abstraction layer between the hardware and other software layers. For example, the kernel 1014 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. The service 1016 can provide other public services for other software layers. The driver 1022 is responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, the driver 1022 may include a display driver, an imaging device driver, a Bluetooth or Bluetooth low energy driver, a flash drive, a serial communication driver (e.g., a universal serial bus (USB) driver), a WI-FI driver, an audio driver, a power management driver, etc.

[0109] The library 1008 provides a low-level common infrastructure used by the application 1006. The library 1008 may include a system library 1018 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1008 may include an API library 1024, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting graphics content in two dimensions (2D) and three dimensions (3D) on a display, GLMotif for implementing a user interface), an image feature extraction library (e.g., OpenIMAJ), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1008 may also include various other libraries 1028 to provide many other APIs to the application 1006 .

[0110] The framework 1010 provides a high-level common infrastructure used by the applications 1006. For example, the framework 1010 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 1010 can provide a wide range of other APIs that can be used by the applications 1006, some of which can be specific to a particular operating system or platform.

[0111] In an example, applications 1006 may include home applications 1036, contact applications 1030, browser applications 1032, book reader applications 1034, location applications 1042, media applications 1044, messaging applications 1046, game applications 1048, and various other applications such as third-party applications 1040. Applications 1006 are programs that execute functions defined in programs. Various programming languages ​​may be used to create one or more of the applications 1006 constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C language or assembly language). In a specific example, third-party applications 1040 (e.g., applications developed using ANDROID or IOS software development kits (SDKs) by entities other than the vendor of a specific platform) may be mobile software running on a mobile operating system such as IOS, ANDROID, WINDOWSPhone, or another mobile operating system. In this example, third party application 1040 may invoke API calls 1050 provided by operating system 1012 to facilitate the functionality described herein.

[0112] Fig.11 1 is a block diagram illustrating an example messaging system 1100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 1100 includes multiple instances of a client device 926 that host multiple applications including a messaging client 1102 and other applications 1104. The messaging client 1102 is communicatively coupled to other instances of the messaging client 1102 (e.g., hosted on respective other client devices 926), a messaging server system 1106, and a third-party server 1108 via a network 930 (e.g., the Internet). The messaging client 1102 may also communicate with a local host application 1104 using an application program interface (API).

[0113] The messaging clients 1102 are able to communicate and exchange data with other messaging clients 1102 and messaging server systems 1106 via the network 930. The data exchanged between the messaging clients 1102 and between the messaging clients 1102 and messaging server systems 1106 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0114] The messaging server system 1106 provides server-side functionality to a particular messaging client 1102 via the network 930. Although some functionality of the messaging system 1100 is described herein as being performed by the messaging client 1102 or by the messaging server system 1106, it may be a design choice whether some functionality is located within the messaging client 1102 or within the messaging server system 1106. For example, it may be technically preferred to initially deploy some technologies and functionality within the messaging server system 1106, but later migrate the technologies and functionality to the messaging client 1102 where the client device 926 has sufficient processing power.

[0115] The messaging server system 1106 supports various services and operations provided to the messaging clients 1102. Such operations include transmitting data to the messaging clients 1102, receiving data from the messaging clients 1102, and processing data generated by the messaging clients 1102. As examples, the data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 1100 is stimulated and controlled by functions available via the user interface (UI) of the messaging client 1102.

[0116] Turning now specifically to the messaging server system 1106, an application program interface (API) server 1110 is coupled to and provides a programming interface to an application server 1114. The application server 1114 is communicatively coupled to a database server 1116, which facilitates access to a database 1120 that stores data associated with messages processed by the application server 1114. Similarly, a web server 1124 is coupled to and provides a web-based interface to the application server 1114. To this end, the web server 1124 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0117] The application program interface (API) server 1110 receives and transmits message data (e.g., commands and message payloads) between the client device 926 and the application server 1114. Specifically, the application program interface (API) server 1110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 1102 to activate the functions of the application server 1114. The application program interface (API) server 1110 exposes various functions supported by the application server 1114, including: account registration; login functionality; sending messages from a particular messaging client 1102 to another messaging client 1102 via the application server 1114; sending media files (e.g., images or videos) from a messaging client 1102 to the messaging server 1112 and for possible access by another messaging client 1102; setting up of media data collections (e.g., stories); retrieving a friend list of a user of the client device 926; retrieving such collections; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends within a social graph; and opening application events (e.g., related to the messaging client 1102).

[0118] The application server 1114 hosts several server applications and subsystems, including, for example, a messaging server 1112, an image processing server 1118, and a social network server 1122. The messaging server 1112 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 1102. As will be described in more detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 1102. In view of the hardware requirements for other processor- and memory-intensive data processing, such processing can also be performed on the server side by the messaging server 1112.

[0119] The application server 1114 also includes an image processing server 1118 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 1112.

[0120] The social network server 1122 supports various social networking functions and services and makes these functions and services available to the messaging server 1112. To this end, the social network server 1122 maintains and accesses an entity graph within the database 1120. Examples of functions and services supported by the social network server 1122 include identifying other users in the messaging system 1100 with whom a particular user has a relationship or who the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0121] The messaging client 1102 may notify the user of the client device 926 or other users related to such a user (e.g., "friends") of activities taking place in a shared or shareable session. For example, the messaging client 1102 may provide notifications to participants in a conversation (e.g., a chat session) in the messaging client 1102 regarding current or recent use of a game by one or more members of a user group. One or more users may be invited to join an active session or initiate a new session. In some examples, a shared session may provide a shared augmented reality experience in which multiple people may collaborate or participate.

[0122] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media that facilitates communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0123] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0124] A "communication network" refers to one or more portions of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi network, another type of network, or a combination of two or more such networks. For example, a network or a portion of a network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling can implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0125] "Machine-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier waves / modulated data signals. The terms "machine-readable medium," "machine-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0126] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the term includes, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term "signal media."

[0127] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces associated output signals that are applied to operate a machine. A processor may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0128] "Signal medium" refers to any intangible medium that can store, encode or carry instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" may be considered to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0129] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A computer-implemented method, include: capturing, by the one or more processors, sensor data of forces acting on a frame of an augmented reality (AR) system using one or more sensors of the AR system; generating, by the one or more processors, a corrected frame model of the frame based on the sensor data and a physical model of the frame; capturing, by the one or more processors using one or more imaging devices mounted on the frame, tracking video frame data of one or more physical objects in a real-world scene viewed by a user of the AR system; capturing, by the one or more processors, using a gesture component of the AR system, gesture and position data of the AR system when the AR system is capturing the tracking video frame data; as well as Tracking data is generated by the one or more processors based on the corrected frame model, the tracking video frame data, and the pose and position data.

2. The computer-implemented method of claim 1 , further comprising: include: generating, by the one or more processors, virtual overlay data based on the tracking data; generating, by the one or more processors, virtual overlay video frame data based on the corrected frame model and the virtual overlay data; as well as A virtual overlay is provided to the user based on the virtual overlay video frame data using an optical engine of the AR system by the one or more processors.

3. The computer-implemented method of claim 1, in, The one or more sensors are mounted on an exterior portion of the frame.

4. The computer-implemented method of claim 3, in, One or more respective sensing surfaces of the one or more sensors are aligned with a frame hinge leaf surface of the frame.

5. The computer-implemented method of claim 4, in, The one or more sensors include at least two sensors mounted in a vertically spaced apart arrangement.

6. The computer-implemented method of claim 4, in, A temple hinge leaf of a temple piece attached to the frame by a hinge ridge impinges upon the one or more sensors.

7. The computer-implemented method of claim 1, in, The AR system includes a head-mounted AR system.

8. An AR system, include: frame; one or more sensors operable to sense forces acting on the frame; one or more imaging devices mounted to the frame; an optical engine mounted to the frame; one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the AR system to perform operations comprising: capturing sensor data of forces acting on the frame using the one or more sensors; generating a corrected frame model of the frame based on the sensor data and the physical model of the frame; Using the one or more imaging devices, capturing tracking video frame data of one or more physical objects in a real-world scene being viewed by a user of the AR system; using a gesture component of the AR system, capturing gesture and position data of the AR system while the AR system is capturing the tracking video frame data; and Tracking data is generated by the one or more processors based on the rectified frame model, the tracking video frame data, and the pose and position data.

9. The AR system according to claim 8, in, When executed by the AR system, the instructions further cause the AR system to perform operations including: generating virtual overlay data based on the tracking data; generating virtual overlay video frame data based on the corrected framework model and the virtual overlay data; as well as Using the optical engine, a virtual overlay is provided to the user based on the virtual overlay video frame data.

10. The AR system according to claim 8, in, The one or more sensors are mounted on an exterior portion of the frame.

11. The AR system according to claim 10, in, One or more respective sensing surfaces of the one or more sensors are aligned with a frame hinge leaf surface of the frame.

12. The AR system according to claim 11, in, The one or more sensors include at least two sensors mounted in a vertically spaced apart arrangement.

13. The AR system according to claim 11, in, A temple hinge leaf of a temple piece attached to the frame by a hinge ridge impinges upon the one or more sensors.

14. The AR system according to claim 8, in, The AR system includes a head-mounted AR system.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations, the operations include: Using one or more sensors of the head-mounted AR system, capturing sensor data of forces acting on a frame of the AR system; generating a corrected frame model of the frame based on the sensor data and the physical model of the frame; capturing, using one or more imaging devices mounted on the frame, tracking video frame data of one or more physical objects in a real-world scene being viewed by a user of the AR system; as well as capturing, by one or more processors, using a gesture component of the AR system, gesture and position data of the AR system while the AR system is capturing the tracking video frame data; as well as Tracking data is generated, by the one or more processors, based on the rectified frame model, the tracking video frame data, and the pose and position data.

16. The non-transitory computer-readable storage medium of claim 15, in, When executed by the computer, the instructions further cause the computer to perform operations including: generating virtual overlay data based on the tracking data; generating virtual overlay video frame data based on the corrected framework model and the virtual overlay data; as well as Using an optical engine of the AR system, a virtual overlay is provided to the user based on the virtual overlay video frame data.

17. The non-transitory computer-readable storage medium of claim 15, in, The one or more sensors are mounted on an exterior portion of the frame.

18. The non-transitory computer-readable storage medium of claim 17, in, One or more respective sensing surfaces of the one or more sensors are aligned with a frame hinge leaf surface of the frame.

19. The non-transitory computer-readable storage medium of claim 15, in, The one or more sensors include at least two sensors mounted in a vertically spaced apart arrangement.

20. The non-transitory computer-readable storage medium of claim 15, in, The AR system includes a head-mounted AR system.

Images