Goggles with asynchronous rolling shutter camera

Abstract

Eyewear having asynchronous rolling shutter (RS) cameras such that the images produced by each camera are misaligned and a state including velocity and gravity direction in the eyewear frame of reference is computed. The state is not limited to these two parameters as other parameters such as acceleration bias, gyroscope bias, or both can be included. A mathematical solver is used such that the processing time to compute the velocity and gravity direction is acceptable. Arranging the RS cameras in an asynchronous configuration allows the motion of the eyewear to be estimated from only one stereo pair of images and does not require more images to be processed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 021,023, filed on September 15, 2020, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present subject matter relates to eyewear devices, such as smart glasses. Background Art

[0004] Portable eyewear devices available today, such as smart glasses, headgear, and head caps, integrate cameras and see-through displays. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings depict one or more implementations by way of example only and not limitation.In the accompanying drawings, like reference numerals represent like or similar elements.

[0006] Figure 1A is a side view of an example hardware configuration of an eyewear apparatus showing a right optical assembly having an image display and applying field of view adjustments to a user interface presented on the image display based on detected head or eye movement of a user;

[0007] Figure 1B yes Figure 1A a top cross-sectional view of a temple of an eyewear device depicting a visible light camera, a head motion tracker for tracking head motion of a user of the eyewear device, and a circuit board;

[0008] Figure 2A is a rear view of an example hardware configuration of an eyewear apparatus including an eye scanner on a frame for use in a system for identifying a user of the eyewear apparatus;

[0009] Figure 2B is a rear view of another example hardware configuration of an eyewear device including an eye scanner on a temple for use in a system for identifying a user of the eyewear device;

[0010] Figure 2C and 2D is a rear view of an example hardware configuration for a goggle apparatus including two different types of image displays.

[0011] Figure 3 Shown Figure 2A a rear perspective view of the goggle assembly, depicting an infrared emitter, an infrared camera, a front portion of the frame, a rear portion of the frame, and a circuit board;

[0012] Figure 4 is through Figure 3 A cross-sectional view of an infrared emitter and a frame of a goggle device;

[0013] Figure 5 Detecting eye gaze direction is shown;

[0014] Figure 6 Detecting eye position is shown;

[0015] Figure 7 Depicts an example of visible light captured by a left visible light camera as a left original image and visible light captured by a right visible light camera as a right original image;

[0016] Figure 8A Frames from a synchronized rolling shutter camera are shown;

[0017] Figure 8B and Figure 8C shows example equations used by the solver;

[0018] Figure 8D shows frames from an asynchronous rolling shutter camera where the image from one camera is rotated;

[0019] Figure 8E shows frames from an asynchronous rolling shutter camera where the images of one camera are shifted in time;

[0020] Figure 8F shows frames from an asynchronous rolling shutter camera where the image of one camera is spatially displaced;

[0021] Figure 9 A block diagram showing the electronic components of the eyewear device; and

[0022] Figure 10 is a flow chart for determining the velocity and gravity direction of goggles using an asynchronous rolling shutter camera. DETAILED DESCRIPTION

[0023] The present disclosure includes an example of goggles having asynchronous rolling shutter (RS) cameras so that the images generated by each camera are not aligned and a state including velocity and direction of gravity in the goggles' reference frame is calculated. The state is not limited to these two parameters, as other parameters such as acceleration bias, gyroscope bias, or both may be included. A mathematical solver is used so that the processing time for calculating velocity and direction of gravity is acceptable. Arranging the RS cameras in an asynchronous configuration allows the motion of the goggles to be estimated from only one stereo image pair, without the need to process more images.

[0024] Additional objects, advantages, and novel features of the examples will be set forth in part in the following description and will become apparent to those skilled in the art upon study of the following and the accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized and obtained by the methods, means, and combinations particularly pointed out in the appended claims.

[0025] In the following detailed description, numerous specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the present teachings can be practiced without such details. In other instances, well-known methods, procedures, components, and circuits have been described in a relatively high-level manner, without detail, to avoid unnecessarily obscuring various aspects of the present teachings.

[0026] As used herein, the term "coupled" refers to any logical, optical, physical, or electrical connection, link, or the like that transmits signals or light generated or provided by one system element to another coupled element. Unless otherwise specified, coupled elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or propagation media that can modify, manipulate, or transport light or signals.

[0027] The orientations of the eyewear assembly, associated components, and any complete assembly incorporating an eye scanner and camera (such as shown in any of the accompanying figures) are provided by way of example only for purposes of illustration and discussion. In operation for a particular variable optical processing application, the eyewear assembly may be oriented in any other orientation suitable for the particular application of the eyewear assembly, such as up, down, sideways, or any other orientation. Moreover, as used herein, any directional terms, such as front, back, inside, outside, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used by way of example only and do not limit the direction or orientation of any optical member or component of an optical assembly constructed as otherwise described herein.

[0028] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0029] Figure 1A is a side view of an example hardware configuration of the goggle device 100, which includes an image display 180D ( Figure 2A ) of the right optical assembly 180B. The eyewear device 100 includes a plurality of visible light cameras 114A-B ( Figure 7 ), wherein the right visible light camera 114B is located on the right temple 110B.

[0030] Left and right visible light cameras 114A-B have image sensors sensitive to wavelengths in the visible light range. Each of visible light cameras 114A-114B has a different forward-facing angle of coverage. For example, visible light camera 114B has depicted angle of coverage 111B. The angle of coverage is the angular range over which the image sensors of visible light cameras 114A-B receive electromagnetic radiation and generate an image. Examples of such visible light cameras 114A-B include high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640×480 pixels, for a total of 300,000 pixels), 720p, or 1080p. Image sensor data from visible light cameras 114A-B is captured along with geolocation data, digitized by an image processor, and stored in memory.

[0031] To provide stereoscopic vision, visible light cameras 114A-B may be coupled to an image processor ( Figure 9 Image processor 912 includes components for receiving signals from visible light cameras 114A-B and processing those signals from visible light cameras 114A-B into a form suitable for storage in memory ( Figure 9 The timestamp may be added by the image processor 912 or other processor that controls the operation of the visible light cameras 114A-B. The visible light cameras 114A-B allow the stereo camera to simulate human binocular vision. The stereo camera provides a time stamp based on two captured images ( Figure 7 Components 758A-B) to reproduce a three-dimensional image ( Figure 7 715). This three-dimensional image 715 allows for an immersive, realistic experience, such as for use in virtual reality or video games. For stereoscopic vision, a pair of images 758A-B is generated at a given moment—one for each of the left and right visible light cameras 114A-B. When the pair of generated images 758A-B from the forward-facing coverage angles 111A-B of the left and right visible light cameras 114A-B are stitched together (e.g., by image processor 912), depth perception is provided by optical components 180A-B.

[0032] In an example, the user interface field of view adjustment system includes a goggle device 100. The goggle device 100 includes a frame 105, a right temple 110B extending from a right outer side 170B of the frame 105, and a see-through image display 180D ( Figure 2A -B), the see-through image display 180D includes an optical assembly 180B for presenting a graphical user interface to the user. The eyewear device 100 includes a left visible light camera 114A connected to the frame 105 or the left temple 110A to capture a first image of a scene. The eyewear device 100 also includes a right visible light camera 114B connected to the frame 105 or the right temple 110B to capture a second image of the scene (e.g., simultaneously with the left visible light camera 114A), the second image partially overlapping the first image. Although in Figure 1A -B, but the user interface field of view adjustment system also includes: a processor 932, which is connected to the goggle device 100 and connected to the visible light cameras 114A-B; a memory 934, which is accessible to the processor 932; and programming in the memory 934, such as programming in the goggle device 100 itself or in another part of the user interface field of view adjustment system.

[0033] Although Figure 1A Not shown, the goggle device 100 also includes a head motion tracker ( Figure 1B Element 109) or eye movement tracker ( Figure 2A -B element 213). The eyewear device 100 also includes: a see-through image display 180C-D of the optical assembly 180A-B for presenting a series of displayed images; and an image display driver ( Figure 9 The eyewear device 100 further includes a memory 934 and a processor 932 that can access the image display driver 942 and the memory 934. The eyewear device 100 also includes programming ( Figure 9 Element 934). The programming executed by the processor 932 configures the eyewear device 100 to perform functions including the function of presenting an initial display image of the above series of display images via the see-through image display 180C-D, the initial display image having an initial field of view corresponding to the initial head orientation or the initial eye gaze direction ( Figure 5 element 230).

[0034] Programming executed by the processor 932 further configures the eyewear device 100 to detect movement of the eyewear device user by: (i) detecting movement of the eyewear device user via a head motion tracker ( Figure 1B 109) tracking the head movement of the user's head, or (ii) via an eye movement tracker ( Figure 2A -B. Figure 5Element 213 of the eyewear device 100 tracks eye movements of the user's eyes. Processor 932 is programmed to further configure the eyewear device 100 to determine a field of view adjustment for an initial field of view of an initial display image based on the detected user movement. The field of view adjustment includes a continuous field of view corresponding to continuous head orientations or continuous eye orientations. Processor 932 is programmed to further configure the eyewear device 100 to generate a continuous display image of the aforementioned series of display images based on the field of view adjustment. Processor 932 is programmed to further configure the eyewear device 100 to present the continuous display image via see-through image displays 180C-D of optical assemblies 180A-B.

[0035] Figure 1B yes Figure 1A FIG1 is a top cross-sectional view of a temple of the eyewear device 100 depicting the right visible light camera 114B, the head motion tracker 109, and the circuit board. The construction and placement of the left visible light camera 114A are substantially similar to the right visible light camera 114B, except that the connection means and the coupling are both located on the left outer side 170A. As shown, the eyewear device 100 includes the right visible light camera 114B and the circuit board, which may be a flexible printed circuit board (PCB) 140. The right hinge 126B connects the right temple 110B to the right temple 125B of the eyewear device 100. In some examples, components or contacts of the right visible light camera 114B, the flexible PCB 140, or other electrical connectors may be located on the right temple 125B or the right hinge 126B.

[0036] As shown, the eyewear device 100 has a head motion tracker 109, which includes, for example, an inertial measurement unit (IMU). An IMU is an electronic device that uses a combination of an accelerometer and a gyroscope, and sometimes a magnetometer, to measure and report specific forces, angular rates, and sometimes the magnetic field around the body. The IMU works by detecting linear acceleration using one or more accelerometers and detecting rotation rate using one or more gyroscopes. A typical configuration of an IMU includes an accelerometer, a gyroscope, and a magnetometer for each of the three axes, the horizontal axis (X) for left and right movement, the vertical axis (Y) for up and down movement, and the depth or distance axis (Z) for up and down movement. The accelerometer detects the gravity vector. The magnetometer defines rotation in the magnetic field (e.g., facing south, north, etc.), just like a compass that generates a directional reference. The three accelerometers are used to detect acceleration along the horizontal, vertical, and depth axes defined above, which may be defined relative to the ground, the eyewear apparatus 100 , or a user wearing the eyewear apparatus 100 .

[0037] The goggle device 100 detects movement of the user of the goggle device 100 by tracking head movement of the user's head via the head movement tracker 109. The head movement includes a change in head orientation on a horizontal axis, a vertical axis, or a combination thereof from an initial head orientation during presentation of an initial display image on the image display. In one example, tracking head movement of the user's head via the head movement tracker 109 includes measuring an initial head orientation (e.g., lateral or diagonal movement) on a horizontal axis (e.g., an X-axis), a vertical axis (e.g., a Y-axis), or a combination thereof via the inertial measurement unit 109. Tracking head movement of the user's head via the head movement tracker 109 also includes measuring continuous head orientations on a horizontal axis, a vertical axis, or a combination thereof via the inertial measurement unit 109 during presentation of the initial display image.

[0038] Tracking the head movement of the user's head via the head movement tracker 109 also includes determining a change in head orientation based on both the initial head orientation and the continuous head orientation. Detecting the movement of the user of the eyewear device 100 also includes determining that the change in head orientation exceeds a deviation angle threshold on the horizontal axis, the vertical axis, or a combination thereof in response to tracking the head movement of the user's head via the head movement tracker 109. The deviation angle threshold is between approximately 3° and 10°. As used herein, when referring to an angle, the term "approximately" refers to ±10% of the specified amount.

[0039] Changes along the horizontal axis slide three-dimensional objects such as characters, Bitmojis, app icons, etc., into and out of view by, for example, hiding, unhiding, or otherwise adjusting the visibility of the three-dimensional objects. In one example, when the user looks up, changes along the vertical axis display weather information, time of day, date, calendar appointments, etc. In another example, when the user looks down on the vertical axis, the eyewear device 100 can be powered off.

[0040] The right temple 110B includes a temple body 211 and a temple cap. Figure 1B The temple cap is omitted from the cross section of FIG. 1. Various interconnected circuit boards (e.g., PCBs or flexible PCBs) are disposed inside the right temple 110B, which contain controller circuitry for the right visible light camera 114B, microphone 130, speaker 132, low-power wireless circuitry (e.g., for communicating via Bluetooth). TM wireless short-range network communications), high-speed wireless circuits (for example, for wireless local area network communications via WiFi).

[0041] The right visible light camera 114B is coupled to or disposed on the flexible PCB 240 and covered by a visible light camera cover lens, which is aimed through an opening formed in the right temple 110B. In some examples, the frame 105 coupled to the right temple 110B includes an opening (or openings) for the visible light camera cover lens. The frame 105 includes a front-facing side that is configured to face outward, away from the user's eye. The opening for the visible light camera cover lens is formed on the front side and through the front-facing side. In this example, the right visible light camera 114B has an outward-facing coverage angle 111B relative to the line of sight or viewing angle of the right eye of the user of the eyewear device 100. The visible light camera cover lens can also be adhered to the outward-facing surface of the right temple 110B, wherein the opening is formed with an outward-facing coverage angle, but in a different outward direction. This coupling can also be indirect via an intermediate component.

[0042] Left (first) visible light camera 114A is coupled to left see-through image display 180C of left optical assembly 180A to generate a first background scene for a first continuously displayed image. Right (second) visible light camera 114B is coupled to right see-through image display 180D of right optical assembly 180B to generate a second background scene for a second continuously displayed image. The first and second background scenes partially overlap to present a three-dimensional observable area for the continuously displayed image.

[0043] Flexible PCB 140 is disposed within right temple 110B and is coupled to one or more other components housed within right temple 110B. Although shown as being formed on a circuit board of right temple 110B, right visible light camera 114B may be formed on a circuit board of left temple 110A, temples 125A-B, or frame 105.

[0044] Figure 2A is a rear view of an example hardware configuration of the eyewear apparatus 100 including an eye scanner 213 on the frame 105 for use in a system for determining the eye position and gaze direction of a wearer / user of the eyewear apparatus 100. Figure 2A As shown, the eyewear device 100 is configured to be worn by a user. Figure 2A The example of the eyewear is glasses. The eyewear device 100 can take other forms and can be combined with other types of frames, such as a headgear, headphones, or a helmet.

[0045] In the example of goggles, the goggle device 100 includes a frame 105 including a left frame 107A connected to a right frame 107B via a nose bridge 106 that fits the user's nose. The left and right frames 107A, 107B include corresponding holes 175A-B that hold corresponding optical elements 180A-B (such as lenses and see-through displays 180C-D). As used herein, the term lens is intended to cover a transparent or translucent piece of glass or plastic with curved and flat surfaces that cause light to converge / diverge or cause little or no convergence / divergence.

[0046] Although shown as having two optical elements 180A-B, the eyewear apparatus 100 may include other arrangements, such as a single optical element depending on the application or intended user of the eyewear apparatus 100. As further shown, the eyewear apparatus 100 includes a left temple 110A adjacent to the left outer side 170A of the frame 105 and a right temple 110B adjacent to the right outer side 170B of the frame 105. The temples 110A-B may be integrated into the frame 105 on the respective sides 170A-B (as shown) or implemented as separate components attached to the frame 105 on the respective sides 170A-B. Alternatively, the temples 110A-B may be integrated into temples (not shown) attached to the frame 105.

[0047] exist Figure 2A In the example shown, the eye scanner 213 includes an infrared emitter 115 and an infrared camera 120. Visible light cameras typically include a blue light filter to block infrared light detection. In the example shown, the infrared camera 120 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640X480 pixels for a total of 300,000 pixels), in which the blue filter is removed. The infrared emitter 115 and the infrared camera 120 are co-located on the frame 105. For example, the two are shown as connected only to the upper portion of the left side frame 107A. The frame 105 or one or more of the left and right temples 110A and 110B include a circuit board (not shown) that includes the infrared emitter 115 and the infrared camera 120. The infrared emitter 115 and the infrared camera 120 can be connected to the circuit board by, for example, soldering.

[0048] Other arrangements of the infrared emitter 115 and the infrared camera 120 can be implemented, including arrangements in which the infrared emitter 115 and the infrared camera 120 are both on the right side frame 107B or in different locations on the frame 105, for example, the infrared emitter 115 is on the left side frame 107A and the infrared camera 120 is on the right side frame 107B. In another example, the infrared emitter 115 is on the frame 105 and the infrared camera 120 is on one of the temples 110A-B, or vice versa. The infrared emitter 115 can be attached to substantially any location on the frame 105, the left temple 110A, or the right temple 110B to emit a pattern of infrared light. Similarly, the infrared camera 120 can be attached to substantially any location on the frame 105, the left temple 110A, or the right temple 110B to capture at least one reflected change in the emission pattern of infrared light.

[0049] Infrared emitter 115 and infrared camera 120 are arranged to face inward toward the user's glasses through part or all of the field of view of the glasses to identify the corresponding glasses position and gaze direction. For example, infrared emitter 115 and infrared camera 120 are positioned directly in front of the glasses, on the upper portion of the frame 105, or on the temples 110A-B at either end of the frame 105.

[0050] Figure 2B 2 is a rear view of another example hardware configuration of the eyewear device 200. In this example configuration, the eyewear device 200 is depicted as including an eye scanner 213 on the right temple 210B. As shown, the infrared emitter 215 and the infrared camera 220 are co-located on the right temple 210B. It should be understood that the eye scanner 213 or one or more components of the eye scanner 213 can be located on the left temple 210A and other locations on the eyewear device 200 (e.g., the frame 105). The infrared emitter 215 and the infrared camera 220 are co-located on the right temple 210B. Figure 2A The infrared emitter of is similar to the infrared camera, but the glasses scanner 213 can be changed to be sensitive to different wavelengths of light, as previously described in Figure 2A Described in .

[0051] Similar to Figure 2A , the eyewear device 200 includes: a frame 105, the frame 105 including a left frame 107A connected to a right frame 107B via a nose bridge 106; and the left frame 107A and the right frame 107B include corresponding holes for holding corresponding optical elements 180A-B including see-through displays 180C-D.

[0052] Figure 2C-D is a rear view of an example hardware configuration of the eyewear device 100, including two different types of see-through image displays 180C-D. In one example, these see-through image displays 180C-D of the optical components 180A-B include integrated image displays. Figure 2C As shown, the optical assembly 180A-B includes a suitable display matrix 180C-D of any suitable type, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a waveguide display, or any other such display. The optical assembly 180A-B also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components in any combination. The optical layers 176A-N may include prisms of appropriate size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's goggles. The prisms of the optical layers 176A-N extend over all or at least a portion of the corresponding apertures 175A-B formed in the left and right bezels 107A, 107B to allow the user to see the second surface of the prisms when the user's eyes are looking through the corresponding left and right bezels 107A, 107B. The first surfaces of the prisms of optical layers 176A-N face upward from the frame 105, and the display matrix is ​​positioned above the prisms such that photons and light beams emitted by the display matrix strike the first surface. The prisms are sized and shaped such that light is refracted within the prisms and directed toward the user's eyes by the second surfaces of the prisms of optical layers 176A-N. In this regard, the second surfaces of the prisms of optical layers 176A-N can be convex to direct light toward the center of the goggles. The prisms can optionally be sized and shaped to magnify the image projected by the see-through image displays 180C-D, with light traveling through the prisms such that the image viewed from the second surface is larger in one or more dimensions than the image emitted by the see-through image displays 180C-D.

[0053] In another example, the see-through image displays 180C-D of the optical assemblies 180A-B include: Figure 2D . The optical assembly 180A-B includes a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source (such as the laser projector 150) is disposed in or on one of the temples 125A-B of the eyewear device 100. The optical assembly 180A-180B includes one or more optical strips 155A-155N spaced across the width of the lens of the optical assembly 180A-180B or across the depth between the front and back surfaces of the lens.

[0054] As the photons projected by the laser projector 150 travel through the lenses of the optical assembly 180A-B, the photons encounter the optical bands 155A-N. When a particular photon encounters a particular optical band, the photon is either redirected toward the user’s eye or it is passed to the next optical band. The combination of the modulation of the laser projector 150 and the modulation of the optical bands can control particular photons or beams of light. In an example, the processor controls the optical bands 155A-N by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical assemblies 180A-B, the eyewear device 100 can include other arrangements, such as a single or three optical assemblies, or the optical assemblies 180A-B can have arranged differently depending on the application or intended user of the eyewear device 100.

[0055] As further shown in Figures 2C to 2D The eyewear device 100 includes a left temple 110A adjacent to a left outer side 170A of the frame 105 and a right temple 110B adjacent to a right outer side 170B of the frame 105, as further shown in

[0056] In one example, the see-through image display includes a first see-through image display 180C and a second see-through image display 180D. The eyewear device 100 includes a first aperture 175A and a second aperture 175B that hold respective first and second optical assemblies 180A and 180B. The first optical assembly 180A includes the first see-through image display 180C (e.g., a display matrix or optical bands 155A-N’ and a projector 150A). The second optical assembly 180B includes the second see-through image display 180D (e.g., a display matrix or optical bands 155A-N” and a projector 150B). The continuous field of view of the continuous display image includes a horizontal, vertical, or diagonal measurement of between about 15° to 30°, and more particularly 24°. The continuous display image having the continuous field of view represents a combined three-dimensional observable area visible by stitching together the two display images presented on the first and second image displays. Figure 2C Figure 2C The continuous display image having the continuous field of view represents a combined three-dimensional observable area visible by stitching together the two display images presented on the first and second image displays.

[0057] ​As used herein, “angle of view” describes the angular range of the field of view associated with a displayed image presented on each of the left and right image displays 180C-D of the optical assembly 180A-B. “Covering angle” describes the angular range that a lens of the visible light cameras 114A-B or infrared camera 220 can image. Typically, the image circle produced by a lens is large enough to completely cover the film or sensor, possibly including some vignetting (i.e., a decrease in brightness or saturation of an image toward the periphery as compared to the center of the image). If the covering angle of the lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the borders, and the effective angle of view will be limited to the covering angle. “Field of view” is intended to describe the field of a viewable area that a user of the eyewear device 100 can see through his or her eyes via a displayed image presented on the left and right image displays 180C-D of the optical assembly 180A-B. The image displays 180C of the optical assembly 180A-B can have a field of view with a covering angle between 15° and 30° (e.g., 24°), and a resolution of 480 x 480 pixels.

[0058] Figure 3 A rear perspective view of the eyewear device of Figure 2A is shown. The eyewear device 100 includes an infrared emitter 215, an infrared camera 220, a front frame portion 330, a rear frame portion 335, and a circuit board 340. In Figure 3 the upper portion of the left border of the frame of the eyewear device 100 includes the front frame portion 330 and the rear frame portion 335. An opening for the infrared emitter 215 is formed on the rear frame portion 335.

[0059] As shown in the wrap-around section 4 in the middle upper portion of the left border of the frame, the circuit board, which is a flexible PCB 340, is sandwiched between the front frame portion 330 and the rear frame portion 335. The left temple 110A is also shown in more detail as being attached to the left temple 325A via the left hinge 126A. In some examples, components of the eye movement tracker 213, including the infrared emitter 215, the flexible PCB 340, or other electrical connectors or contacts, can be located on the left temple 325A or the left hinge 126A.

[0060] Figure 4 is a cross-sectional view through the infrared emitter 215 and the frame corresponding to the wrap-around section 4 of the eyewear device of Figure 3 is shown. In Figure 4A cross-section of the eyewear device 100 is shown with a number of layers. As shown, the frame includes a frame front 330 and a frame back 335. A flexible PCB 340 is disposed on the frame front 330 and connected to the frame back 335. The infrared emitter 215 is disposed on the flexible PCB 340 and covered by an infrared emitter cover lens 445. For example, the infrared emitter 215 is reflowed to the back of the flexible PCB 340. Reflow attaches the infrared emitter 215 to contact pads formed on the back of the flexible PCB 340 by subjecting the flexible PCB 340 to controlled heat of a molten solder paste to connect the two components. In one example, reflow is used to surface mount the infrared emitter 215 on the flexible PCB 340 and electrically connect the two components. However, it should be appreciated that through-holes can be used to connect leads from the infrared emitter 215 to the flexible PCB 340, for example, via interconnects.

[0061] The frame back 335 includes an infrared emitter opening 450 for the infrared emitter cover lens 445. The infrared emitter opening 450 is formed on a back-facing side of the frame back 335 that is configured to face inward toward a user’s eye. In this example, the flexible PCB 340 can be connected to the frame front 330 via a flexible PCB adhesive 460. The infrared emitter cover lens 445 can be connected to the frame back 335 via an infrared emitter cover lens adhesive 455. This coupling can also be indirect via an intermediate component.

[0062] In an example, the processor 932 utilizes the eye tracker 213 to determine an eye gaze direction 230 of a wearer’s eye 234 as shown in Figure 5 and an eye position 236 of the wearer’s eye 234 within a human eye window as shown in Figure 6 The eye tracker 213 is a scanner that uses infrared light illumination (e.g., near infrared, short wavelength infrared, mid wavelength infrared, long wavelength infrared, or far infrared) to take images of changes in reflection of infrared light from the eye 234 to determine a gaze direction 230 of a pupil 232 of the eye 234 and an eye position 236 relative to the see-through display 180D.

[0063] Figure 7 An example is depicted of taking visible light with a camera. Visible light is taken by the left visible light camera 114A with the left visible light camera field of view 111A as a left raw image 758A. Visible light is taken by the right visible light camera 114B with the right visible light camera field of view 111B as a right raw image 758B. Based on processing of the left raw image 758A and the right raw image 758B, the processor 932 generates a three-dimensional depth map 715 of a three-dimensional scene, hereinafter referred to as an image.

[0064] A global shutter camera uses a single exposure-then-readout step for the entire image. However, a rolling shutter (RS) camera has a multi-step mechanism that sequentially captures image rows at different times.

[0065] refer to Figure 8A Typically, the left RS visible light camera 114A and the right RS visible light camera 114B are synchronized so that the optical axes of the cameras are parallel and the camera sensors are coplanar. The reason for this synchronization configuration is to achieve as close to rectified stereo as possible, eliminating the need for rectification. This configuration is called RS stereo. A single RS stereo image does not allow estimation of the velocity v0 or the direction of gravity g0 of the eyewear assembly 100 / 200 (also referred to as the equipment) to which the camera is coupled.

[0066] Figure 8B A first linear solver is shown, which is a linear system of mathematical equations showing that the velocity v0 and the direction of gravity g0 of the goggles 100 / 200 in motion cannot be determined using synchronized RS cameras. Figure 8C A second linear solver is shown, which is also a linear mathematical equation, and it is also shown that the velocity v0 and the direction of gravity g0 cannot be determined using synchronized RS cameras.

[0067] Figure 8B and Figure 8C The two linear solutions have the form Ax=b. Matrix A represents a rectangular matrix and b represents a column vector whose elements are set according to the known inertial measurement unit (IMU) readings generated by the IMU 800 ( Figure 9 ), and the corresponding scene point X ( Figure 8D 、 Figure 8E and Figure 8F ). The unknown vector x contains, among other elements, the velocity v0 of the goggle device 100 / 200. The entire vector X is estimated. Estimating the initial velocity v0 is a prerequisite for being able to further use the IMU readings in a Visual-Inertial Odometry (VIO) system. However, (prior) knowledge of the velocity v0 at any frame facilitates many other use cases. As independent knowledge, the magnitude of the velocity v0 indicates whether the goggle device 100 / 200 is in motion. It should be noted that at linear velocity, the IMU 800 outputs zero acceleration and it is not possible to distinguish from the IMU 800 alone whether one is static or in motion.

[0068] For example, a person in a skyscraper elevator doesn't perceive the elevator's motion after its initial acceleration, even though it's moving very quickly. This is because the vestibular system in the human inner ear acts as an IMU. Elevators lack windows, so without visual input, people can't tell whether they're moving at a constant speed or are stationary.

[0069] Visual information from the cameras is a cue that helps estimate velocity. Arranging RS cameras 114A and 114B in an asynchronous manner allows distinguishing / estimating motion from only one stereo image pair 758A and 758B and reduces the need to process more stereo images. Specifically, the time between the scene point X (visual matching) of the corresponding stereo images is TimeL and TimeR ( Figure 10 ), providing a non-zero time interval to integrate the inertial data, which together with the visually matched image coordinates are then Figure 8B and Figure 8C A linear solver is used. When this time difference disappears (standard stereo cameras), there is no contribution from the inertial data. The linear solver is an example, as the same information (integrated inertial data and vision mismatch) can be similarly used with a nonlinear solver / optimizer.

[0070] refer to Figure 8D 、 Figure 8E and Figure 8F In the example of the present disclosure, the RS feature is exploited at the geometric level by asynchronously or misaligning the cameras 114A and 114B so that time ti is not equal to time tj. Figure 10 Referred to as TimeL and TimeR. Figure 8D shows a right camera image 758B rotated compared to the left camera image 758A, Figure 8E A right camera image 758B shifted in time is shown, and Figure 8F A spatially displaced right camera image 758B is shown. Figure 8D 、 Figure 8E and Figure 8F A camera image is shown having features relative to another camera image, i.e., rotation, time shift, or spatial shift. The present disclosure of making the cameras asynchronous or misaligned is related to Figure 8A This allows the goggle assembly 100 / 200 to be observed as well as the camera velocity v0 and the direction of gravity g0 as the goggle assembly 100 / 200 moves. Figure 8B and Figure 8C The linear solver equations of calculate the state including the initial velocity v0 and the gravity direction g0 from the stereo frames of each camera 114A and 114B and provide stable visual-inertial triangulation, such as for visual-inertial odometry (VIO) and for visual-inertial simultaneous localization and mapping (vi-SLAM). The state is not limited to these two parameters, as other parameters such as acceleration bias, gyroscope bias, or both can be included. By reference Figure 10 The processor 932 , described in more detail, handles the camera asynchronous algorithm 945 .

[0071] The motion and asynchrony of cameras 114A and 114B causes any imaged 3D scene point X to be projected in the left and right camera images 758A and 758B, respectively, as shown in different rows in the images, i.e., at different times. In this example, the same 3D scene point is shown as XL in the left camera image 758A and as XR in the right camera image 758B. This allows processor 932 to integrate the data from internal measurement unit IMU 800 ( Figure 9 ) and generate IMU samples Figure 8B and Figure 8C The matrix A shown in is sufficiently constrained to be able to estimate the velocity v0 and gravity g0 under which the eyewear assembly 100 / 200 is currently moving. No further stereo images need to be added in the estimation process.

[0072] Figure 9 A high-level functional block diagram of exemplary electronic components provided in the eyewear device 100 / 200 is depicted. The illustrated electronic components include a processor 932 that executes a camera asynchronous algorithm 945, as will be described with reference to FIG. Figure 10 Descriptive.

[0073] The memory 934 includes instructions for execution by the processor 932 to implement the functions of the goggle device 100 / 200, including instructions for the processor 932 to execute the image correction algorithm 945. The processor 932 receives power from the battery 950 and executes instructions stored in the memory 934 or on a chip integrated with the processor 932 to perform the functions of the goggle device 100 / 200 and communicate with external devices via a wireless connection.

[0074] The user interface adjustment system 900 includes a goggle device. The user interface adjustment system 900 also includes a mobile device 990 and a server system 998 connected via different networks. The mobile device 990 can be a smartphone, a tablet, a laptop, an access point, or any other such device capable of connecting to the goggle device 100 / 200 using both a low-power wireless connection 925 and a high-speed wireless connection 937. The mobile device 990 is connected to the server system 998 and the network 995. The network 995 can include any combination of wired and wireless connections.

[0075] Goggles device 100 / 200 includes at least two visible light cameras 114A-114B (one associated with left side 170A and one associated with right side 170B). Goggles device 100 / 200 also includes two see-through image displays 180C-D of optical assembly 180A-B (one associated with left side 170A and one associated with right side 170B). In the present disclosure, image displays 180C-D are optional. Goggles device 100 / 200 also contains image display driver 942, image processor 912, low-power circuitry 920, and high-speed circuitry 930. Figure 9 The depicted components for goggle device 100 / 200 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the temples. Alternatively or additionally, the depicted components can be located in the temples, frames, hinges, or nose bridge of goggle device 100 / 200. Left and right visible light cameras 114A-114B can include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, lenses, or any other respective visible or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0076] Eye movement tracking program implements user interface field-of-view adjustment instructions, including instructions for causing goggle device 100 / 200 to track eye movements of a user’s eyes of goggle device 100 / 200 via eye movement tracker 213. Other implemented instructions (functions) cause goggle device 100 / 200 to determine a field-of-view adjustment to an initial field-of-view of an initial displayed image based on detected eye movements of the user’s eyes corresponding to successive eye directions. Further implemented instructions generate a successive displayed image of the series of displayed images based on the field-of-view adjustment. The successively displayed image is produced as visible output to the user via a user interface. The visible output appears on see-through image displays 180C-D of optical assembly 180A-B, which are driven by image display driver 942 to present the series of displayed images, including the initially displayed image with the initial field-of-view and the successively displayed image with the successive field-of-view.

[0077] As Figure 9As shown, the high-speed circuitry 930 includes a high-speed processor 932, memory 934, and high-speed wireless circuitry 936. In an example, an image display driver 942 is coupled to the high-speed circuitry 930 and is operated by the high-speed processor 932 to drive the left image display 180C and the right image display 180D of the optical assembly 180A-B. The high-speed processor 932 can be any processor capable of managing high-speed communications and operations required by any general-purpose computing system for the eyewear assembly 100 / 200. The high-speed processor 932 includes the processing resources required to manage high-speed data transmission over a high-speed wireless connection 937 to a wireless local area network (WLAN) using the high-speed wireless circuitry 936. In a specific example, the high-speed processor 932 executes an operating system, such as a Linux operating system or other such operating system for the eyewear assembly 100 / 200, and the operating system is stored in the memory 934 for execution. In addition to any other responsibilities, the high-speed processor 932, which executes the software architecture of the eyewear assembly 100 / 200, is used to manage data transmission using the high-speed wireless circuitry 936. In a specific example, high-speed wireless circuit 936 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by high-speed wireless circuit 936.

[0078] The low power wireless circuit 924 and the high speed wireless circuit 936 of the eyewear device 100 / 200 may include a short range transceiver (Bluetooth TM ) and a wireless wide area, local area network, or wide area network transceiver (e.g., cellular or WiFi). The mobile device 990 (including a transceiver that communicates via a low power wireless connection 925 and a high speed wireless connection 937) can be implemented using details of the architecture of the goggle device 100 / 200, as can the other elements of the network 995.

[0079] Memory 934 includes any storage device capable of storing different data and applications, including color maps, camera data generated by left and right visible light cameras 114A-B and image processor 912, and images generated by image display driver 942 for display on see-through image displays 180C-D of optical assembly 180A-B, among other things. While memory 934 is shown as integrated with high-speed circuitry 930, in other examples, memory 934 can be a separate off-chip element of eyewear device 100 / 200. In a particular such example, electrical routing lines can provide connectivity from image processor 912 or low-power processor 922 to memory 934 through a chip including high-speed processor 932. In other examples, high-speed processor 932 can manage addressing of memory 934 such that low-power processor 922 will direct high-speed processor 932 at any time that a read or write operation involving memory 934 is required.

[0080] Server system 998 can be one or more computing devices included as part of a service or network computing system, for example including a processor, memory, and network communication interface to communicate with mobile device 990 and eyewear device 100 / 200 over network 995. Eyewear device 100 is connected with a host computer. For example, eyewear device 100 / 200 is paired with mobile device 990 via high-speed wireless connection 937, or connected to server system 998 via network 995, such as via high-speed wireless connection 937.

[0081] Output components of eyewear device 100 / 200 include visual components, as in Figures 2C to 2DThe left image display 180C and the right image display 180D of the optical components 180A and 180B described in the accompanying drawings are, for example, displays such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide. The image displays 180C and 180D of the optical components 180A and 180B are driven by an image display driver 942. The output components of the goggle device 100 and 200 also include an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor), other signal generators, and the like. The input components of the goggle device 100 / 200, the mobile device 990, and the server system 998 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides the location and force of a touch or touch gesture, or other tactile input components), and an audio input component including a microphone 938. The microphone 938 captures audio near the goggle device 100 / 200, which can be streamed to the remote operator's mobile device 990. In an example, the microphone 938 can be directional to correspond to the image being viewed.

[0082] The eyewear assembly 100 / 200 may optionally include additional peripheral device elements 919. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the eyewear assembly 100 / 200. For example, the peripheral device elements 919 may include any I / O components, including output components, movement components, position components, or any other such components described herein.

[0083] For example, the biometric components of the user interface field of view adjustment 900 include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal identification, facial identification, fingerprint identification, or identification based on electroencephalogram), etc. The mobile components include an internal measurement unit (IMU) 800, an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The position component includes a position sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a WiFi or Bluetooth receiver for generating positioning system coordinates, etc. TMThe mobile device 990 may also include a transceiver, an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like. Such positioning system coordinates may also be received from the mobile device 990 via the low power wireless circuitry 924 or the high speed wireless circuitry 936 via wireless connections 925 and 937.

[0084] According to some examples, an "application" or "applications" is a program that performs the functions defined in a program. One or more of the various applications structured in different ways can be created using different programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, a third-party application (e.g., an application developed by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Applications developed with a software development kit (SDK) can be developed on mobile operating systems such as IOS TM ANDROID TM 、 In this example, third-party applications can call APIs provided by the operating system to facilitate the functions described herein.

[0085] Figure 10 A flow chart 1000 is shown of a camera asynchronous algorithm 945 executed by the processor 932 for determining the velocity and gravity direction of the goggles.

[0086] At block 1002, processor 932 receives and processes asynchronous camera images 758A and 758B from cameras 114A and 114B, respectively. Figure 8D 、 Figure 8E and Figure 8F The 3D scene point X is projected in the left and right camera images 758A and 758B, respectively, as shown in the different pixel rows in the image, i.e., Figure 8E The midfield attraction X is shifted in time and Figure 8F The midfield point X is spatially displaced. In this example, the 3D scene point is shown as point XL in the left camera image 758A and as point XR in the right camera image 758B. Due to the asynchrony of camera images 758A and 758B, the times TimeL and TimeR will not be equal when the eyewear assembly 100 / 200 and the cameras are in motion.

[0087] At block 1004, the processor 932 integrates the IMU samples from the IMU 800 between these two time instances and generates Figure 8B and Figure 8C The IMU samples are limited, thereby allowing the processor 932 to estimate the velocity v0 and the direction of gravity g0 under which the goggle assembly 100 / 200 is currently moving.

[0088] At block 1006, processor 932 uses Figure 8B and Figure 8C The linear solver equations for either or both of the above are used, and only one image from each camera is used to estimate the initial velocity v0 and gravity g0 under which the goggle assembly 100 / 200 is currently moving. No further stereo images need to be added to the estimation process. The processing of the processor 932 is straightforward and therefore not very complex, thereby saving processing resources and quickly calculating the goggle assembly 100 / 200 velocity and gravity direction.

[0089] It should be understood that the terms and expressions used herein have ordinary meanings, as assigned to these terms and expressions for their corresponding corresponding queries and research fields, unless otherwise specifically defined herein. Relational terms such as first and second, etc., may be used only to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms "comprises," "comprising," "includes," "including," or any other variations thereof are intended to cover non-exclusive inclusions such that a process, method, article, or device that includes or comprises a series of elements or steps includes not only those elements or steps, but may also include other elements or steps that are not explicitly listed or that are inherent to such process, method, article, or device. Without further limitation, an element preceded by "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or device that comprises the element.

[0090] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, sizes, and other specifications set forth in this specification (including in the following claims) are approximate and imprecise. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and with conventional art to which they belong. For example, parameter values, etc. may vary from the stated quantity by as much as ±10% unless otherwise expressly stated.

[0091] Furthermore, in the above detailed description, it can be seen that various features are grouped together in various examples for the purpose of streamlining the disclosure. This approach to the present disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in fewer features than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim independently standing as separately claimed subject matter.

[0092] While the foregoing has described what is considered to be the best mode and other examples, it should be understood that various modifications may be made therein, and that the subject matter disclosed herein may be implemented in different forms and examples, and that it may be used in many applications, only some of which have been described herein. It is intended by the appended claims to claim any and all modifications and variations that fall within the true scope of the present concepts.

Claims

1. Goggles, including: Frames; a pair of cameras, each coupled to the frame and configured to capture a respective first image and a second image, wherein the cameras are asynchronous; as well as An electronic processor configured to: receiving the first image and the second image from the asynchronous camera; processing the first image and the second image based on a characteristic of the first image relative to the second image, wherein the characteristic is selected from the group consisting of: the first image is rotated relative to the second image, the first image is temporally shifted relative to the second image, and the first image is spatially shifted relative to the second image; and The velocity and gravity of the goggles are calculated as a function of the characteristics.

2. The goggles according to claim 1, wherein: The processor is configured to calculate the velocity of the goggles using a linear equation.

3. The goggles according to claim 1, wherein: The camera comprises a rolling shutter camera, and the first image and the second image comprise a common scene point.

4. The goggles according to claim 3, wherein: The processor is configured to process a time difference between the pair of cameras imaging the scene point.

5. The goggles according to claim 1, wherein: The goggles include an inertial measurement unit (IMU) configured to generate IMU readings, wherein the processor is configured to process the IMU readings to determine the velocity.

6. A method for using goggles, the goggles comprising a frame; a pair of cameras, each of the pair of cameras being coupled to the frame and configured to capture a corresponding first image and a second image, wherein: The pair of cameras are asynchronous; and an electronic processor: receiving the first image and the second image from the asynchronous camera; processing the first image and the second image based on characteristics of the first image relative to the second image; as well as Calculating a velocity and a gravity of the goggles as a function of the features, wherein the features are selected from the group consisting of: the first image is rotated relative to the second image, the first image is shifted in time relative to the second image, and the first image is shifted in space relative to the second image.

7. The method according to claim 6, wherein: The camera comprises a rolling shutter camera, and the processor calculates the velocity of the goggles using a linear equation.

8. The method according to claim 6, wherein: The first image and the second image include a common scene point.

9. The method according to claim 8, wherein The processor processes a time difference between the pair of cameras imaging the scene point.

10. The method according to claim 6, wherein: The goggles include an inertial measurement unit (IMU) configured to generate IMU readings, wherein the processor processes the IMU readings to determine the velocity.

11. A non-transitory computer readable medium storing program code, the program code being operable, when executed, to cause a processor of goggles having a frame and a pair of asynchronous cameras, the asynchronous cameras each coupled to the frame and configured to capture a respective first image and a second image, to perform the following steps; receiving the first image and the second image from the asynchronous camera; processing the first image and the second image based on a characteristic of the first image relative to the second image, wherein the characteristic is selected from the group consisting of: the first image is rotated relative to the second image, the first image is temporally shifted relative to the second image, and the first image is spatially shifted relative to the second image; and The velocity and gravity of the goggles are calculated as a function of the characteristics.

12. The non-transitory computer-readable medium of claim 11, further comprising code for calculating the velocity of the goggles using a linear equation.

13. The non-transitory computer-readable medium of claim 11, wherein: The first image and the second image include common object points or scene points.

14. The non-transitory computer-readable medium of claim 13, further comprising code for processing a time difference between the pair of cameras imaging the object point or the scene point.

Images