Eyewear with audio source separation using attitude tracker

Through the microphone array and position tracking technology of the electronic glasses device, audio source separation is realized in a multi-person environment, solving the problem of difficulty in speech demixing in the prior art, and improving the separation accuracy and efficiency of audio signals.

CN119014007APending Publication Date: 2024-11-22SNAP INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380032352.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-01
Filing Date
2023-03-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing electronic glasses devices are difficult to effectively separate and distinguish different audio sources in a multi-person environment, resulting in difficulty in demixing speech.

Method used

Using a simplified audio source separation method, the microphone array of the electronic glasses device and the position tracking technology of the remote user is used to determine the position and direction of the sound source through the microphone array and the processor to achieve audio source separation.

Benefits of technology

It effectively solves the problem of indistinguishable audio signals in multiple people's environments, improves the accuracy and efficiency of voice demixing, and enhances the audio communication experience between users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119014007A_ABST
    Figure CN119014007A_ABST
Patent Text Reader

Abstract

Electronic glasses devices provide simplified audio source separation, also referred to as voice / sound unmixing, using alignment between respective device trajectories. Multiple users of an electronic eyewear device in an environment may simultaneously generate audio signals (e.g., voice / sound) that are difficult to distinguish from each other. The electronic eyewear device tracks the location of a remote electronic eyewear device moving by other users, or an object of other users, such as the face of a remote user, to provide audio source separation using the location of the sound source. Simplified voice de-mixing uses a microphone array of an electronic glasses device and a known location of a remote user's electronic glasses device relative to the user's electronic glasses device to facilitate audio source separation.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 711,494, filed on April 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present subject matter relates to an electronic eyewear device with audio source separation, such as smart glasses with a see-through display. Background Art

[0004] Portable electronic eyewear devices available today (such as smart glasses, headgear, and head caps) integrate microphones, cameras, and see-through displays. The microphones and speakers used by the electronic eyewear devices can be used to enhance the user's experience in the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings depict one or more implementations by way of example only and not limitation.In the accompanying drawings, like reference numerals represent like or similar elements.

[0006] Figure 1A is a side view of an example hardware configuration of an electronic eyewear apparatus showing a right optical assembly with an image display and applying a field of view adjustment to a user interface presented on the image display based on detected head or eye movement of the user;

[0007] Figure 1B yes Figure 1A A top cross-sectional view of a temple of an electronic eyewear device depicting a visible light camera, a head movement tracker for tracking head movement of a user of the electronic eyewear device, and a circuit board;

[0008] Figure 1C is a side view of an example hardware configuration of an electronic eyewear apparatus showing a left optical assembly having an image display;

[0009] Figure 2A is a rear view of an example hardware configuration of an electronic eyewear device for use in a system for identifying a user of the electronic eyewear device, the electronic eyewear device including an eye scanner on a frame;

[0010] Figure 2B is a rear view of another example hardware configuration of an electronic eyeglass device for use in a system for identifying a user of the electronic eyeglass device, the electronic eyeglass device including an eye scanner on a temple;

[0011] Figure 2C and Figure 2D is a rear view of an example hardware configuration of an electronic eyewear device including two different types of image displays.

[0012] Figure 3 Shows Figure 2A a cutaway rear perspective view of an electronic eyewear device depicting an infrared emitter, an infrared camera, a front portion of the frame, a rear portion of the frame, and a circuit board;

[0013] Figure 4 It passes through Figure 3 A cross-sectional view of an infrared emitter and a frame of an electronic eyeglass device;

[0014] Figure 5 is a top view showing detection of eye gaze direction;

[0015] Figure 6 is a top view showing detection of eye positions;

[0016] Figure 7 is a block diagram depicting visible light captured by a visible light camera as a corresponding raw image;

[0017] Figure 8 is a schematic diagram of a user in an example environment;

[0018] Fig. 9 is a block diagram depicting the electronic components of the electronic eyewear device;

[0019] Fig.10 is a flow chart illustrating an example method for displaying a virtual object in a physical environment;

[0020] Fig.11 is a block diagram of a system for aligning an electronic eyewear device;

[0021] Fig.12 is a flow chart showing an apparatus for aligning electronic glasses;

[0022] Fig.13 is a schematic diagram of multiple users in an example environment with aligned tracks and illustrating audio source separation; and

[0023] Fig.14 is a flow chart illustrating separation of audio signals from different audio sources using aligned tracks. DETAILED DESCRIPTION

[0024] The present disclosure is directed to an electronic eyeglass device with simplified audio source separation (also known as speech / sound demixing) using alignment between the corresponding tracks of the electronic eyeglass device and a remote electronic eyeglass device. Multiple users of the electronic eyeglass device in an environment may generate audio signals that are difficult to distinguish from each other. The user's electronic eyeglass device tracks the position of the other user's mobile remote electronic eyeglass device or the other user's object (such as their face) to provide audio source separation using the location of the sound source. The simplified audio source separation uses the microphone array of the electronic eyeglass device and the known position of the remote user's electronic eyeglass device relative to the electronic eyeglass device to facilitate audio source separation.

[0025] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims.

[0026] In the following detailed description, many specific details are set forth by way of example to provide a thorough understanding of the relevant teachings. However, it should be clear to those skilled in the art that the present teachings can be practiced without such details. In other examples, well-known methods, processes, components, and circuits have been described at a relatively high level without detail in order to avoid unnecessarily making aspects of the present teachings unclear.

[0027] As used herein, the term "coupled" refers to any logical, optical, physical or electrical connection, link, etc., whereby a signal or light generated or supplied by one system element is imparted to another coupled element. Unless otherwise described, coupled elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements or communication media that may modify, manipulate or carry light or signals.

[0028] The orientation of the electronic eyeglass device, associated components, and any complete device incorporating an eye scanner and camera, such as shown in any of the accompanying figures, is given by way of example only for purposes of demonstration and discussion. In operation for a particular variable optical processing application, the electronic eyeglass device can be oriented in any other direction suitable for the particular application of the electronic eyeglass device, such as up, down, sideways, or any other orientation. Moreover, within the scope used herein, any directional term (such as, front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side) is used only by way of example and does not limit the direction or orientation of any optical component or optical component constructed as otherwise described herein.

[0029] Reference is now made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0030] Figure 1A and Figure 1C 1 is a side view of an example hardware configuration of the electronic eyeglass apparatus 100, which includes an optical assembly 180A having an image display 180C and another optical assembly 180B having an image display 180D ( Figure 2A The electronic eyewear device 100 includes a plurality of visible light cameras 114A-114B ( Figure 7 ), wherein a first visible light camera 114A is located on the right temple 110A and a second visible light camera is located on the left temple 110B.

[0031] The electronic eyeglass device 100 includes a plurality of microphones 145A-145D forming a microphone array 145, wherein a first (right front) microphone 145A is located on the right temple 110A and a second (right rear) microphone 145B is located on the right temple 110A. A third (left front) microphone 145C is located on the left temple 110B, and a fourth (left rear) microphone 145C is located on the left temple 110B. In order to determine the arrival direction of sounds in the environment, the microphones 145A-145D are coupled to the processor 932 ( Fig. 9 ) for digitally processing the signals from the array. The microphones 145A-145D send data signals to the processor 932, which then determines the direction and intensity of the sound source relative to the electronic eyeglass device 100. In an example, the user of the electronic eyeglass device 100 can talk to a person. The microphone array 145 detects the person's voice and sends a signal to the processor 932. The processor 932 determines the direction of arrival of the person's voice with reference to the electronic eyeglass device 100.

[0032] Visible light cameras 114A and 114B may include image sensors having wavelengths sensitive to the visible light range. Each of visible light cameras 114A and 114B has a different front-facing coverage angle, for example, visible light camera 114A has a depicted coverage angle 111A. The coverage angle is the angular range over which the respective image sensors of visible light cameras 114A and 114B detect incident light and generate image data. Examples of such visible light cameras 114A and 114B include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640×480 pixels, a total of 300,000 pixels), 720p, 1080p, 4K, or 8K. Image sensor data from visible light cameras 114A and 114B may be captured along with geolocation data, digitized by an image processor, and stored in a memory.

[0033] To provide stereoscopic vision, visible light cameras 114A and 114B may be coupled to an image processor ( Fig. 9 Image processor 912 may include a processor for receiving signals from visible light cameras 114A and 114B and processing those signals from visible light cameras 114A and 114B into a format suitable for storage in memory ( Fig. 9 The timestamp may be added by image processor 912 or other processor controlling the operation of visible light cameras 114A and 114B. Visible light cameras 114A and 114B allow the stereo camera to simulate human binocular vision. The stereo camera also provides a method for providing a stereo image based on two captured images (i.e., from visible light cameras 114A and 114B, respectively) having the same timestamp. Figure 7 The image pair 758A and 758B) is used to reproduce the three-dimensional scene ( Figure 7 14A and 14B). Such three-dimensional images allow for immersive virtual experiences that feel very real, such as for use in virtual reality or video games. For stereoscopic vision, a pair of images 758A and 758B may be generated at a given moment—one for each of visible light cameras 114A and 114B. When a pair of generated images 758A and 758B from fields of view (FOVs) 111A and 111B of forward-facing visible light cameras 114A and 114B are stitched together (e.g., by image processor 912), depth perception is provided by optical assemblies 180A and 180B.

[0034] In the example, the electronic eyewear device 100 includes a frame 105, a right side frame 107A, a right temple 110A extending from a right outer side portion 170A of the frame 105, and a see-through image display 180C (including an optical assembly 180A for presenting a graphical user interface to a user). Figure 2A-2B ). The electronic eyeglass device 100 includes a first visible light camera 114A connected to the frame 105 or the right temple 110A to capture a first image of a scene. The electronic eyeglass device 100 further includes a second visible light camera 114B connected to the frame 105 or the left temple 110B to capture a second image of the scene that at least partially overlaps with the first image (e.g., simultaneously with the first visible light camera 114A). Figure 1A and Figure 1B Not shown, the high speed (HS) processor 932 ( Fig. 9 ) is coupled to the electronic eyewear device 100 and is connected to the visible light cameras 114A and 114B; a memory 934 ( Fig. 9 ) and the programs in the memory 934 can be set in the electronic eyeglass device 100 itself.

[0035] Although in Figure 1A As not shown in the figure, the electronic glasses device 100 may also include a head movement tracker ( Figure 1B Element 109) or eye movement tracker ( Figure 2A Element 113 or Figure 2B The electronic eyeglass device 100 may further include a perspective image display 180C and 180D for presenting a sequence of display images, respectively, of the optical components 180A and 180B. The electronic eyeglass device 100 may further include an image display driver ( Fig. 9 The image display driver is coupled to the see-through image displays 180C and 180D to drive the image displays 180C and 180D. The see-through image displays 180C and 180D and the image display driver are described in further detail below. The electronic eyewear device 100 may further include a memory 934 and a processor 932 (which may access the image display driver 942 and the memory 934). Figure 4 ), and the program in the memory 934. The execution of the program by the processor 932 configures the electronic eyewear device 100 to perform functions, including the function of presenting an initial display image of a sequence of display images via the perspective image displays 180C and 180D, the initial display image having an initial field of view ( ) corresponding to the initial head direction or the initial eye gaze direction determined by the eye movement tracker 113 or 213. Figure 5 Component 230).

[0036] The execution of the program by the processor 932 may further configure the electronic eyewear device 100 to detect the movement of the user of the electronic eyewear device by: (i) detecting the movement of the user of the electronic eyewear device via a head movement tracker ( Figure 1B 109) to track the head movements of the user's head, or (ii) via an eye movement tracker ( Figure 2A Element 113 or Figure 2B Element 213 of the electronic eyeglass device 100) tracks eye movements of the eyes of the user of the electronic eyeglass device 100. Execution of the program by the processor 932 may further configure the electronic eyeglass device 100 to determine a field of view adjustment of an initial field of view of an initial display image based on the detected movement of the user. The field of view adjustment may include a continuous field of view corresponding to continuous head directions or continuous eye directions. Execution of the program by the processor 932 may further configure the electronic eyeglass device 100 to generate a continuous display image of a sequence of display images based on the field of view adjustment. Execution of the program by the processor 932 may further configure the electronic eyeglass device 100 to present a continuous display image via the perspective image displays 180C and 180D of the optical components 180A and 180B.

[0037] Figure 1B It depicts Figure 1A 1 is an illustration of a top cross-sectional view of the optical components and electronics in a portion of the electronic eyewear device 100 shown in FIG. 1 , depicting the first visible light camera 114A, the head movement tracker 109, and the circuit board 140. The construction and location of the second visible light camera 114B are substantially similar to the first visible light camera 114A, except that it is connected and coupled to another outer side portion 170B ( Figure 2A ). As shown, the electronic eyeglass device 100 includes a first visible light camera 114A and a circuit board, which may be a flexible printed circuit board (PCB) 140. A first hinge 126A connects the right temple 110A to an articulated arm 125A of the electronic eyeglass device 100. In some examples, the first visible light camera 114A, the flexible PCB 140, or other electrical connectors or contacts may be located on the right temple 110A or the first hinge 126A.

[0038] As shown, the electronic glasses device 100 may include a head movement tracker 109, which includes, for example, an inertial measurement unit (IMU). An inertial measurement unit is an electronic device that uses a combination of an accelerometer and a gyroscope (sometimes also a magnetometer) to measure and report specific forces, angular velocity, and sometimes magnetic fields around the body. The inertial measurement unit works by using one or more accelerometers to detect linear acceleration and one or more gyroscopes to detect rotation rate. A typical configuration of an inertial measurement unit includes an accelerometer, a gyroscope, and a magnetometer for each of the three axes: a horizontal axis (X) for left and right movement, a vertical axis (Y) for top and bottom movement, and a depth or distance axis (Z) for up and down movement. The accelerometer detects the gravity vector. The magnetometer defines the rotation in the magnetic field (e.g., facing south, north, etc.) like a compass that generates a direction reference. The three accelerometers detect acceleration along the horizontal, vertical, and depth axes defined above, which may be defined relative to the ground, the electronic eyeglass apparatus 100 , or a user wearing the electronic eyeglass apparatus 100 .

[0039] The electronic eyeglass device 100 can detect the movement of the user of the electronic eyeglass device 100 by tracking the head movement of the user's head via the head movement tracker 109. The head movement includes a change in the head direction from the initial head direction on the horizontal axis, the vertical axis, or a combination thereof during the presentation of the initial display image on the image display. In one example, tracking the head movement of the user's head via the head movement tracker 109 includes measuring the initial head direction on the horizontal axis (e.g., X-axis), the vertical axis (e.g., Y-axis), or a combination thereof (e.g., lateral or diagonal movement) via an inertial measurement unit. Tracking the head movement of the user's head via the head movement tracker 109 further includes measuring the continuous head direction on the horizontal axis, the vertical axis, or a combination thereof via the inertial measurement unit during the presentation of the initial display image.

[0040] Tracking head movements of the user's head via the head movement tracker 109 may include determining a change in head direction based on both an initial head direction and a continuous head direction. Detecting movement of the user of the electronic eyewear device 100 may further include determining that a change in head direction exceeds a deviation angle threshold on a horizontal axis, a vertical axis, or a combination thereof in response to tracking head movements of the user's head via the head movement tracker 109. In an exemplary configuration, the deviation angle threshold is between approximately 3° and 10°. As used herein, when referring to an angle, the term "approximately" refers to a deviation of ±10% from the stated amount.

[0041] Changes along the horizontal axis slide three-dimensional objects (such as characters, Bitmojis, application icons, etc.) into and out of the field of view by, for example, hiding the three-dimensional objects, showing the three-dimensional objects, or otherwise adjusting the visibility of the three-dimensional objects. In one example, for example, when the user looks up, changes along the vertical axis display weather information, time of day, date, calendar appointments, etc. In another example, when the user looks down on the vertical axis, the electronic eyewear device 100 can be powered off.

[0042] like Figure 1B As shown, the right temple 110A includes a temple body 211 configured to receive a temple cover, wherein Figure 1B The temple cover is omitted in the cross-sectional view of FIG. A plurality of interconnected circuit boards (such as a PCB or a flexible PCB 140) are disposed inside the right temple 110A, the plurality of interconnected circuit boards including a controller circuit for the first visible light camera 114A, a microphone 130, a speaker 132, a low-power wireless circuit (e.g., for communicating with the user via a wireless communication interface). Wireless short-range network communications) and high-speed wireless circuits (e.g., for wireless LAN communication and positioning via GPS).

[0043] The first visible light camera 114A is coupled to or disposed on the flexible PCB 140 and is covered by a visible light camera cover lens that is aimed through an opening formed in the right temple 110A. In some examples, the frame 105 connected to the right temple 110A includes an opening for the visible light camera cover lens. The frame 105 may include a front side configured to face outward away from the user's eyes. The opening for the visible light camera cover lens may be formed on the front side and through the front side. In this example, the first visible light camera 114A has an outward-facing coverage angle 111A that is colinear with the line of sight or viewing angle of the right eye of the user of the electronic eyeglass device 100. The visible light camera cover lens may also be adhered to the outward-facing surface of the right temple 110A, wherein the opening is formed with an outward-facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0044] First visible light camera 114A may be connected to first perspective image display 180C of first optical assembly 180A to generate a first background scene of a first continuously displayed image. Second visible light camera 114B may be connected to second perspective image display 180D of second optical assembly 180B to generate a second background scene of a second continuously displayed image. The first background scene and the second background scene may partially overlap to present a three-dimensional observable area of ​​the continuously displayed image.

[0045] Flexible PCB 140 may be disposed within right temple 110A and coupled to one or more other components housed in right temple 110A. Although shown as being formed on circuit board 140 of right temple 110A, first visible light camera 114A may be formed on another circuit board (not shown) in one of left temple 110B, articulated arm 125A, articulated arm 125B, or frame 105.

[0046] Figure 2A is a diagram depicting a rear view of an exemplary hardware configuration of the electronic eyeglass apparatus 100. Figure 2A As shown, the electronic eyeglass device 100 is configured to be worn by a user. Figure 2A The example of the eyewear device 100 is glasses. The eyewear device 100 may take other forms and may be combined with other types of frames, such as a head cap, headphones, or a helmet.

[0047] In the glasses example, the electronic glasses device 100 includes a frame 105 including a right frame 107A connected to a left frame 107B via a nose bridge 106 configured to receive the user's nose. The right frame 107A and the left frame 107B include corresponding apertures 175A and 175B that hold corresponding optical elements 180A and 180B (such as lenses and see-through displays 180C and 180D). As used herein, the term lens refers to a transparent or translucent sheet of glass or plastic covered with curved and flat surfaces that cause light to converge / diverge or cause little or no convergence / divergence.

[0048] Although shown as having two optical elements 180A and 180B, the electronic eyeglass device 100 may include other arrangements, such as a single optical element, depending on the application or intended user of the electronic eyeglass device 100. As further shown, the electronic eyeglass device 100 includes a right temple 110A adjacent to the right outer side 170A of the frame 105 and a left temple 110B adjacent to the left outer side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective sides 170A and 170B (as shown) or implemented as separate components of the frame 105 attached to the respective sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0049] exist Figure 2A In the example of , the eye scanner 113 is configured to include an infrared emitter 115 and an infrared camera 120. Visible light cameras typically include a blue light filter to block infrared light detection. In the example, the infrared camera 120 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640×480 pixels, a total of 300,000 pixels), in which the blue filter is removed. The infrared emitter 115 and the infrared camera 120 can be co-located on the frame 105. For example, both are shown as being connected to the upper portion of the left frame 107B. The frame 105 or one or more of the temples 110A and 110B include a circuit board (not shown) that includes the infrared emitter 115 and the infrared camera 120. The infrared emitter 115 and the infrared camera 120 can be connected to the circuit board by, for example, soldering.

[0050] Other arrangements of the infrared emitter 115 and the infrared camera 120 may be implemented, including arrangements in which both the infrared emitter 115 and the infrared camera 120 are on the right frame 107A or in different locations on the frame 105. For example, the infrared emitter 115 may be on the left frame 107B and the infrared camera 120 may be on the right frame 107A. In another example, the infrared emitter 115 may be on the frame 105 and the infrared camera 120 may be on one of the temples 110A or 110B, or vice versa. The infrared emitter 115 may be substantially connected to any location on the frame 105, the right temple 110A, or the left temple 110B to emit a pattern of infrared light. Similarly, the infrared camera 120 may be substantially connected to any location on the frame 105, the right temple 110A, or the left temple 110B to capture at least one reflection change in the emission pattern of infrared light.

[0051] Infrared emitter 115 and infrared camera 120 can be arranged to face inwardly toward the user's eyes with part or all of the eye's field of view to identify the corresponding eye position and gaze direction. For example, infrared emitter 115 and infrared camera 120 can be positioned directly in front of the eyes, in the upper portion of frame 105, or in temple 110A or 110B at either end of frame 105.

[0052] Figure 2B is a diagram depicting a rear view of another exemplary hardware configuration of the electronic eyeglass apparatus 200. In this exemplary configuration, the electronic eyeglass apparatus 200 is depicted as including an eye scanner 213 on the right temple 210A. As shown, an infrared emitter 215 and an infrared camera 220 are co-located on the right temple 210A. The eye scanner 213 or one or more components of the eye scanner 213 may be located on the left temple 210B and at other locations of the electronic eyeglass apparatus 200 (e.g., the frame 105). The infrared emitter 215 and the infrared camera 220 are co-located with the right temple 210A. Figure 2A The infrared emitter of is similar to an infrared camera, but the eye scanner 213 can be changed to be sensitive to different wavelengths of light, as previously described in Figure 2A Similar to Figure 2A , the electronic eyeglass device 200 includes a frame 105 including a right frame 107A connected to a left frame 107B via a nose bridge 106. The frames 107A and 107B may include respective apertures for holding respective optical elements 180A and 180B including see-through displays 180C and 180D.

[0053] Figure 2C and Figure 2D1 is a diagram depicting a rear view of an exemplary hardware configuration of an electronic eyeglass apparatus 100 that includes two different types of see-through image displays 180C and 180D. In one example, these see-through image displays 180C and 180D of optical assemblies 180A and 180B include integrated image displays. Figure 2C As shown, optical assemblies 180A and 180B include any suitable type of display matrices 180C and 180D, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a waveguide display, or any other such display.

[0054] The optical components 180A and 180B also include one or more optical layers 176A-N, which may include lenses, optical coatings, prisms, reflectors, waveguides, light strips, and other optical components in any combination. The optical layer 176 may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eyes. The prism of the optical layer 176 may extend over all or at least a portion of the corresponding apertures 175A and 175B formed in the frame 107A and 107B to allow the user to see the second surface of the prism when the user's eyes are viewing through the corresponding frame 107A and 107B. The first surface of the prism of the optical layer 176 faces upward from the frame 105 and the display matrix covers the prism so that the photons and light emitted by the display matrix hit the first surface. The size and shape of the prism can be determined so that light is refracted within the prism and directed to the user's eyes by the second surface of the prism of the optical layer 176. In this regard, the second surface of the prisms of the optical layer 176 can be convex to direct light toward the center of the eye. The prisms can be sized and shaped to magnify the images projected by the see-through image displays 180C and 180D, and the light travels through the prisms so that the image observed from the second surface is larger in one or more dimensions than the image emitted from the see-through image displays 180C and 180D.

[0055] In another example, the see-through image displays 180C and 180D of the optical assemblies 180A and 180B may include Figure 2DThe projected image display shown. The optical assemblies 180A and 180B include a projector 150, which can be a three-color projector, a laser projector, or other type of projector using a scanning mirror and a galvanometer. During operation, a light source (such as a projector 150) is disposed in or on one of the temples 110A or 110B of the electronic eyeglass device 100. The optical assemblies 180A and 180B include one or more light bands 155A-N spaced across the width of the lens of the optical assemblies 180A and 180B or across the depth of the lens between the front surface and the rear surface of the lens.

[0056] As photons projected by projector 150 travel through the lenses of optical components 180A and 180B, the photons encounter light band 155. When a particular photon encounters a particular light band, the photon is either redirected toward the user's eye or it passes to the next light band. The combination of modulation of laser projector 150 and modulation of the light band can control specific photons or light beams. In an example, the processor controls light band 155 by initiating a mechanical, acoustic, or electromagnetic signal. Although shown as having two optical components 180A and 180B, the electronic eyeglass device 100 may include other arrangements, such as a single or three optical components, or the optical components 180A and 180B may have different arrangements depending on the application or intended user of the electronic eyeglass device 100.

[0057] like Figure 2C and Figure 2D As further shown, the electronic eyeglass device 100 includes a right temple 110A adjacent to the right outer side 170A of the frame 105 and a left temple 110B adjacent to the left outer side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective outer sides 170A and 170B (as shown) or implemented as separate components attached to the frame 105 on the respective sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0058] In one example, the perspective image display includes a first perspective image display 180C and a second perspective image display 180D. The electronic eyeglass device 100 may include a first aperture 175A and a second aperture 175B, and the first aperture 175A and the second aperture 175B hold the corresponding first optical component 180A and the second optical component 180B. The first optical component 180A may include a first perspective image display 180C (e.g., a display matrix or light strip and a projector in the right temple 110A). The second optical component 180B may include a second perspective image display 180D (e.g., a display matrix or light strip and a projector in the right temple 110A (shown as a projector 150)). The continuous field of view of the continuous display image may include a viewing angle of about 15° to 30° measured horizontally, vertically or diagonally, and more specifically 24°. The continuous display image with a continuous field of view represents a combined three-dimensional observable area visible by splicing together two display images presented on the first image display and the second image display.

[0059] As used herein, "angle of view" describes the angular range of the field of view associated with the displayed image presented on each of the image displays 180C and 180D of the optical assemblies 180A and 180B. "Coverage angle" describes the angular range of the lens of the visible light camera 114A or 114B or the infrared camera 220 that can image. Typically, the image circle produced by the lens is large enough to completely cover the film or sensor, possibly including some vignetting (i.e., the brightness or saturation of the image decreases toward the periphery compared to the center of the image). If the coverage angle of the lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective angle of view will be limited to the coverage angle. "Field of view" is intended to describe the area of ​​observable area that a user of the electronic eyewear device 100 can see through his or her eyes via the displayed images presented on the image displays 180C and 180D of the optical assemblies 180A and 180B. Image display 180C of optical assemblies 180A and 180B may have a field of view covering an angle between 15° and 30° (eg, 24°) and a resolution of 480×480 pixels (or greater, such as 720p, 1080p, 4K, or 8K).

[0060] Figure 3 Shows Figure 2A The electronic eyeglass device 100 includes an infrared emitter 115, an infrared camera 120, a frame front 330, a frame back 335, and a circuit board 340. Figure 3 As can be seen in FIG. 1 , the upper portion of the left frame of the frame of the electronic eyeglass device 100 includes a frame front 330 and a frame back 335 . An opening for the infrared emitter 115 is formed on the frame back 335 .

[0061] As shown in section 4 encircled in the upper middle portion of the left side of the frame, a circuit board, which is a flexible PCB 340, is sandwiched between the frame front 330 and the frame back 335. Also shown in more detail is the attachment of the left temple portion 110B to the left temple portion 125B via the left hinge 126B. In some examples, components of the eye movement tracker 113, including the infrared emitter 115, the flexible PCB 340, or other electrical connectors or contacts, can be located on the left temple 125B or the left hinge 126B.

[0062] Figure 4 is through and Figure 3 The cross-sectional view of the infrared emitter 115 and the frame corresponding to the surrounding cross-section 4 of the electronic glasses device. Figure 4 The multiple layers of the electronic eyeglass device 100 are shown in the cross-section of FIG. As shown, the frame includes a frame front 330 and a frame back 335. The flexible PCB 340 is disposed on the frame front 330 and connected to the frame back 335. The infrared emitter 215 is disposed on the flexible PCB 340 and covered by the infrared emitter cover lens 445. For example, the infrared emitter 115 is reflowed to the back of the flexible PCB 340. The two components are connected by subjecting the flexible PCB 340 to controlled heat to melt the solder paste, and the reflow attaches the infrared emitter 115 to the contact pads formed on the back of the flexible PCB 340. In one example, the reflow is used to surface mount the infrared emitter 115 on the flexible PCB 340 and electrically connect the two components. However, it should be understood that, for example, through holes can be used to connect the leads from the infrared emitter 115 to the flexible PCB 340 via interconnects.

[0063] The frame back 335 includes an infrared emitter opening 450 for an infrared emitter cover lens 445. The infrared emitter opening 450 is formed on a rearward-facing side of the frame back 335, which is configured to face inwardly toward the user's eyes. In this example, the flexible PCB 340 can be connected to the frame front 330 via a flexible PCB adhesive 460. The infrared emitter cover lens 445 can be connected to the frame back 335 via an infrared emitter cover lens adhesive 455. The coupling can also be indirect via intervening components.

[0064] In the example, the processor 932 uses the eye tracker 113 to determine whether Figure 5 The eye gaze direction 230 of the wearer's eye 234 is shown, and Figure 6The eye position 236 of the wearer's eye 234 within the eye movement range shown. The eye tracker 113 is a scanner that uses infrared light illumination (e.g., near infrared, short wavelength infrared, medium wavelength infrared, long wavelength infrared, or far infrared) to capture images of changes in reflection of infrared light from the eye 234 to determine the gaze direction 230 of the pupil 232 of the eye 234 and the eye position 236 relative to the see-through display 180.

[0065] Figure 7 The block diagram of FIG. 1 shows an example of capturing visible light with cameras 114A and 114B. Visible light is captured by first visible light camera 114A having a circular field of view (FOV) 111A. The selected rectangular first raw image 758A is used by image processor 912 ( Fig. 9 ) for image processing. Visible light is also captured by a second visible light camera 114B having a circular FOV 111B. A rectangular second raw image 758B selected by image processor 912 is used for image processing by processor 912. Raw images 758A and 758B have overlapping fields of view 713. Processor 912 processes raw images 758A and 758B and generates a three-dimensional image 715 for display by display 180.

[0066] Figure 8 An example environment 800 is depicted along with elements useful for natural feature tracking (NFT; for example, tracking applications using simultaneous localization and mapping (SLAM) algorithms). A user 802 of the electronic eyewear device 100 is present in the example physical environment 800 (the physical environment 800 is located in Figure 8 The processor 932 of the electronic eyewear device 100 uses the captured images to determine its position relative to one or more objects 804 in the environment 800, uses the coordinate system (x, y, z) of the environment 800 to construct a map of the environment 800, and determines its position in the coordinate system ( Fig.10 ). In addition, the processor 932 determines the head posture (roll, pitch, and yaw) of the electronic eyeglass device 100 in the environment by using two or more positioning points (e.g., three position points 806a, 806b, and 806c) associated with a single object 804a, or by using one or more position points 806 associated with two or more objects 804a, 804b, 804c. The processor 932 of the electronic eyeglass device 100 may place a virtual object 808 (such as Figure 8 A key (shown) is positioned within environment 800 for augmented reality viewing via image display 180 .

[0067] Fig. 9Depicted is a high-level functional block diagram including exemplary electronic components disposed in the electronic glasses 100, 200. The electronic components shown include a processor 932, a memory 934, and see-through image displays 180C and 180D.

[0068] The memory 934 includes instructions for execution by the processor 932 to implement the functions of the electronic glasses 100 / 200, including instructions for the processor 932 to control the image 715. The processor 932 receives power from a battery (not shown) and executes instructions stored in the memory 934 or integrated on a chip with the processor 932 to perform the functions of the electronic glasses 100 / 200 and communicate with external devices via a wireless connection.

[0069] The user interface adjustment system 900 includes a wearable device having an eye movement tracker 213 (e.g., Figure 2B 925 and 937). The electronic eyeglass device 100 is shown as an infrared emitter 215 and an infrared camera 220. The user interface adjustment system 900 also includes a mobile device 990 and a server system 998 connected via a variety of networks. The mobile device 990 can be a smart phone, a tablet computer, a laptop computer, an access point, or any other such device capable of connecting to the electronic eyeglass device 100 using both a low-power wireless connection 925 and a high-speed wireless connection 937. The mobile device 990 is connected to the server system 998 and the network 995. The network 995 may include any combination of wired connections and wireless connections.

[0070] The electronic eyeglass device 100 includes at least two visible light cameras 114A and 114B (one associated with the left outer side 170B and one associated with the right outer side 170A). The electronic eyeglass device 100 further includes two perspective image displays 180C and 180D (one associated with each side) of the optical assembly 180A and 180B. The electronic eyeglass device 100 also includes an image display driver 942, an image processor 912, a low power circuit 920, and a high speed circuit 930. Fig. 9 The components for the electronic eyeglass apparatus 100, 200 are shown to be located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the temples. Alternatively or additionally, the depicted components may be located in the temples, frames, hinges, or nose bridges of the electronic eyeglass apparatus 100, 200. The visible light camera 114 may include a digital camera element, such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device, a lens, or any other corresponding visible or light capturing element that can be used to capture data, including images of a scene with an unknown object.

[0071] The eye movement tracking program implements user interface field of view adjustment instructions, including instructions for causing the electronic eyeglass device 100 to track eye movements of the user of the electronic eyeglass device 100 via the eye movement tracker 213. Other implemented instructions (functions) cause the electronic eyeglass device 100, 200 to determine field of view adjustments for the initial field of view 111A-111B based on the detected eye movements of the user corresponding to continuous eye directions. Further implemented instructions generate continuous display images of a display image sequence based on the field of view adjustment. The continuous display images are generated as visible outputs to the user via the user interface. The visible output appears on the perspective image displays 180C and 180D of the optical components 180A and 180B, and the perspective image displays 180C and 180D are driven by the image display driver 942 to present a display image sequence, and the display images include an initial display image with an initial field of view and a continuous display image with a continuous field of view.

[0072] like Fig. 9 As shown, the high-speed circuit 930 includes a high-speed processor 932, a memory 934, and a high-speed wireless circuit 936. In an example, an image display driver 942 is coupled to the high-speed circuit 930 and is operated by the high-speed processor 932 to drive the image displays 180C and 180D of the optical components 180A and 180B. The high-speed processor 932 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required for the electronic eyeglass device 100. The high-speed processor 932 includes the processing resources required to manage high-speed data transmission on the high-speed wireless connection 937 to the wireless local area network (WLAN) using the high-speed wireless circuit 936. In some examples, the high-speed processor 932 executes an operating system, such as a LINUX operating system or other such operating system of the electronic eyeglass device 100, and the operating system is stored in the memory 934 for execution. In addition to any other duties, the high-speed processor 932 that executes the software architecture of the electronic eyeglass device 100 is used to manage data transmission using the high-speed wireless circuit 936. In some examples, high-speed wireless circuit 936 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by high-speed wireless circuit 936.

[0073] The low power wireless circuit 924 and high speed wireless circuit 936 of the electronic eyeglass device 100, 200 may include a short range transceiver (e.g., Bluetooth™) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or WiFi). The mobile device 990 (including a transceiver that communicates via the low power wireless connection 925 and the high speed wireless connection 937) may be implemented using details of the architecture of the electronic eyeglass device 100, as may the other elements of the network 995.

[0074] The memory 934 comprises any storage device capable of storing a variety of data and applications, including, among other things, color maps, camera data generated by the visible light cameras 114A and 114B and the image processor 912, and images generated by the image display driver 942 for display on the see-through image displays 180C and 180D of the optical assemblies 180A and 180B. Although the memory 934 is shown as being integrated with the high-speed circuit 930, in other examples, the memory 934 can be a separate, independently operating element of the electronic eyewear device 100. In some such examples, electrical routing lines can provide connections from the image processor 912 or the low-power processor 922 to the memory 934 through a chip including the high-speed processor 932. In other examples, the high-speed processor 932 can manage the addressing of the memory 934 so that the low-power processor 922 will direct the high-speed processor 932 any time a read or write operation involving the memory 934 is required.

[0075] The server system 998 may be one or more computing devices as part of a service or network computing system, the computing device including, for example, a processor, a memory, and a network communication interface for communicating with the mobile device 990 and the electronic eyeglass device 100, 200 via the network 995. The electronic eyeglass device 100, 200 is connected to a host computer. For example, the electronic eyeglass device 100 is paired with the mobile device 990 via a high-speed wireless connection 937, or is connected to the server system 998 via the network 995.

[0076] The output components of the electronic eyewear device 100 include visual components, such as Figure 2C-Figure 2D The image displays 180C and 180D of the optical components 180A and 180B described in the specification (e.g., displays such as liquid crystal displays (LCDs), plasma display panels (PDPs), light emitting diode (LED) displays, projectors, or waveguides). The image displays 180C and 180D of the optical components 180A and 180B are driven by the image display driver 942. The output components of the electronic eyeglass device 100 further include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, and the like. The input components of the electronic eyeglass devices 100 and 200, the mobile device 990, and the server system 998 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other point devices), tactile input components (e.g., physical buttons, touch screens that provide the location and force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), and the like.

[0077] The electronic eyeglass device 100 may optionally include additional peripheral device elements. Such peripheral device elements may include ambient light and spectral sensors, biometric sensors, additional sensors, or display elements integrated with the electronic eyeglass device 100. For example, the peripheral device elements may include any I / O components, including output components, movement components, position components, or any other such components described herein. The electronic eyeglass device 100 may take other forms and may be combined with other types of frames, for example, head caps, headphones, or helmets.

[0078] For example, the biometric components of the visually adjusted user interface 900 include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The mobile components include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. The position components include position sensor components (e.g., global positioning system (GPS) receiver components) for generating position coordinates, WiFi or Bluetooth™ transceivers for generating positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc. Such positioning system coordinates can also be received from the mobile device 990 via the low power wireless circuit 924 or the high speed wireless circuit 936 via the wireless connections 925 and 937.

[0079] According to some examples, an "application" or "applications" is a program that performs functions defined in a program. Different programming languages ​​can be used to create one or more of the applications structured in a variety of ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application developed using an ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of a particular platform) can be a program that runs on a mobile operating system such as IOS™, ANDROID™, Phone, or another mobile operating system). In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described herein.

[0080] In some examples, the electronic glasses device 100 includes a collection of motion sensing components referred to as an inertial measurement unit (IMU) 972. The motion sensing component may be a microelectromechanical system (MEMS) with microscopic moving parts, usually small enough to be part of a microchip. In some exemplary configurations, the IMU 972 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including the acceleration due to gravity) of the device 100 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the device 100 about three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can provide together the position, orientation, and motion data of the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the direction of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor (such as a global positioning system (GPS) unit 973), one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer and other orientation sensors. Such positioning system coordinates may also be received from mobile device 990 via low power wireless circuitry 924 or high speed wireless circuitry 936 over wireless connections 925 , 937 .

[0081] The IMU 972 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates multiple useful values ​​about the position, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometer can be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the device 100 (in linear coordinates x, y, and z). The angular velocity data from the gyroscope can be integrated to obtain the position of the device 100 (in spherical coordinates). The programming for calculating these useful values ​​can be stored in the memory 934 and executed by the high-speed processor 932 of the electronic eyewear device 100.

[0082] The memory 934 further includes a position detection utility 960, a marker registration utility 962, a positioning utility 964, a virtual object rendering utility 966, a physics engine 968, and a prediction engine 970 for execution by the processor 932. The position detection utility 960 configures the processor 932 to determine the position (position and orientation) within the environment, for example, using the positioning utility 964. The marker registration utility 962 configures the processor 932 to register the marker within the environment. The marker can be a predefined physical marker with a known position within the environment or assigned by the processor 932 to a specific position relative to the environment in which the electronic glasses device 100 operates or relative to the electronic glasses themselves. The positioning utility 964 configures the processor 932 to obtain positioning data for determining the position of the electronic glasses device 100, the virtual object presented by the electronic glasses device, or a combination thereof. The position data can be derived from a series of images, an IMU unit 972, a GPS unit 973, or a combination thereof. The virtual object rendering utility 966 configures the processor 932 to render virtual images for display on the image displays 180C-180D under the control of the image display driver 942 and the image processor 912. The physics engine 968 configures the processor 932 to apply the laws of physics (such as gravity and friction) to the virtual world, for example, between virtual game pieces. The prediction engine 970 configures the processor 932 to predict the expected movement of an object (such as the electronic eyewear device 100) based on the current direction of the object, input from a sensor (such as an IMU 972), an image of the environment, or a combination thereof.

[0083] The memory 934 having instructions for execution by the processor 932 further includes a beamforming utility 972 that configures the processor 932 to demix the audio signals from the microphone array 145 using the tracking data of the remote electronic glasses and the signals from the microphone array 145. Other electronic glasses device 100 position and orientation data is provided by six degrees of freedom (6DOF) alignment ( Fig.11 The beamforming utility 972 additionally configures the processor 932 to determine the direction of arrival (DOA) of sounds present in the environment using the microphone array 145 .

[0084] The processor 932 within the electronic eyeglass device 100 can construct a map of the environment surrounding the electronic eyeglass device 100, determine the position of the electronic eyeglass device within the mapped environment, and determine the relative position of the electronic eyeglass device to one or more objects in the mapped environment. The processor 932 can use a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to construct a map and determine location and position information. In the context of augmented reality, the SLAM algorithm is used to build and update a map of the environment while tracking and updating the position of the device (or user) in the mapped environment. The mathematical solution can be approximated using different statistical methods, such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection.

[0085] The sensor data includes images received from one or both of cameras 114A, 114B, distances received from a laser rangefinder, location information received from GPS 973 and IMU 972, location information received from the GPS and IMU of mobile device 990, or a combination of two or more of such sensor data, or location information received from other sensors that provide data useful for determining location information.

[0086] Fig.10 1 is a flowchart 1000 depicting a method for implementing the augmented reality application described herein on a wearable device (e.g., an electronic eyewear device). Although the steps are described with reference to the electronic eyewear device 100 as described herein, those skilled in the art will appreciate from the description herein other implementations of the steps described for other types of devices. In addition, it is contemplated that in Fig.10 One or more of the steps shown in the drawings and in other figures and described herein may be omitted, performed simultaneously or in series, performed in an order other than shown and described, or performed in combination with additional steps.

[0087] At block 1002, the electronic eyeglass device 100 captures one or more input images of the physical environment 800 near the electronic eyeglass device 100. The processor 932 may continuously receive input images from the visible light camera 114 and store those images in the memory 934 for processing. Additionally, the electronic eyeglass device 100 may capture information from other sensors (e.g., location information from the GPS unit 973, orientation information from the IMU 972, or distance information from a laser distance sensor).

[0088] At block 1004, the electronic eyewear device 100 compares the object in the captured image with the object stored in the image library to identify a match. In some embodiments, the processor 932 stores the captured image in the memory 934. A library of images of known objects is stored in the memory 934.

[0089] In one example, the processor 932 is programmed to recognize predefined specific objects (e.g., a specific picture 804a hung in a known location on a wall, a window 804b in another wall, or an object such as a safe 804c positioned on a floor). Other sensor data (such as GPS data) can be used to narrow the number of known objects used in the comparison (e.g., only images associated with a room identified by GPS coordinates). In another example, the processor 932 is programmed to recognize predefined general objects (e.g., one or more trees in a park).

[0090] At box 1006, the electronic eyeglass device 100 determines its position relative to the object. The processor 932 can determine its position relative to the object by comparing and processing the distance between two or more points in the captured image (for example, between two or more location points on one object 804 or between location points 806 on each of two objects 804) with the known distance between corresponding points in the identified object. A distance between points of the captured image that is larger than the points of the identified object indicates that the electronic eyeglass device 100 is closer to the identified object than the imager that captured the image including the identified object. On the other hand, a distance between points of the captured image that is smaller than the points of the identified object indicates that the electronic eyeglass device 100 is farther from the identified object than the imager that captured the image including the identified object. By processing the relative distances, the processor 932 is able to determine the position relative to the object. Alternatively or additionally, other sensor information (such as laser distance sensor information) can be used to determine the position relative to the object.

[0091] At box 1008, the electronic eyeglass device 100 builds a map of the environment 800 surrounding the electronic eyeglass device 100 and determines its position within the environment. In one example, where the identified object (box 1004) has a predefined coordinate system (x, y, z), the processor 932 of the electronic eyeglass device 100 uses the predefined coordinate system to build a map and determines its position within the coordinate system based on the determined position relative to the identified object (box 1006). In another example, the electronic eyeglass device uses an image of a permanent or semi-permanent object 804 within the environment (e.g., a tree in a park or a park bench) to build a map. According to this example, the electronic eyeglass device 100 can define a coordinate system (x', y', z') for the environment.

[0092] At block 1010, the electronic eyeglass device 100 determines the head posture (roll, pitch, and yaw) of the electronic eyeglass device 100 within the environment. The processor 932 determines the head posture by using two or more position points (e.g., three position points 806a, 806b, and 806c) on one or more objects 804 or by using one or more position points 806 on two or more objects 804. Using conventional image processing algorithms, the processor 932 determines roll, pitch, and yaw by comparing the angles and lengths of lines extending between position points of the captured image and the known image.

[0093] At block 1012, the electronic eyeglass device 100 presents visual images to the user. The processor 932 presents images to the user on the image displays 180C-180D using the image processor 912 and the image display driver 942. The processor develops and presents visual images via the image displays in response to the location of the electronic eyeglass device 100 within the environment 800.

[0094] At block 1014 , the steps described above with reference to blocks 1006 - 1012 are repeated to update the position of the electronic eyewear device 100 and what the user 802 sees as the user moves through the environment 800 .

[0095] Reference again Figure 8 , in this example, the method of implementing an interactive augmented reality application described herein includes a virtual marker 810a associated with a virtual object 808 in an environment 800. In an AR system, markers are registered at multiple locations in the environment to assist the device in the task of tracking and updating the locations of users, devices, and objects (virtual and physical) in the mapped environment. Markers are sometimes registered to high contrast physical objects, such as relatively dark objects 804a mounted on lighter colored walls, to assist cameras and other sensors in the task of detecting markers. Markers can be pre-assigned or assigned by the electronic eyewear device 100 when entering the environment.

[0096] The tag may be encoded with information or otherwise linked to information. The tag may include location information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. The data set associated with the tag is stored in the memory 934 of the electronic eyewear device 100. The data set includes information about the tag 810a, the location (position and orientation) of the tag, one or more virtual objects, or a combination thereof. The tag location may include one or more tag landmarks 816a (such as Figure 8The three-dimensional coordinates of the corners of the generally rectangular marker 810a shown in . The marker position can be expressed relative to real-world geographic coordinates, a marker coordinate system, the position of the electronic eyewear device 100, or other coordinate systems. One or more virtual objects associated with the marker 810a may include any of a variety of materials, including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of such materials. In this context, any type of content that can be stored in memory and retrieved when the marker 810a is encountered or associated with an assigned marker can be classified as a virtual object. Figure 8 The key 808 shown in FIG. 8 is, for example, a virtual object displayed as a still image (2D or 3D) at a marked location.

[0097] In one example, the marker 810a can be registered in memory as being located at the physical object 804a (e.g., Figure 8 In another example, the marker may be registered in memory as a specific location relative to the electronic eyewear device 100.

[0098] Fig.11 An example of establishing a collaborative AR experience between users of an electronic eyewear device 100 by using alignment between corresponding tracks (e.g., on 6DOF) generated by a pose tracker 1100 (e.g., a 6DOF pose tracker) is shown. In one example, the pose tracker includes a visual inertial odometry (VIO) pose tracker, but other types of pose trackers may also be used. The electronic eyewear device 100A of user A and the electronic eyewear device 100B of user B track the electronic eyewear device of another user or an object of another user (such as on the face of the user) to provide a collaborative AR experience. This enables sharing common 3D content between multiple electronic eyewear users without using or aligning common image content, such as reference Figure 8 The markings described above, which is a lighter weight solution with a reduced computational burden on the processor 932. Running a local odometer system on each electronic eyeglass device 100 and cross-tracking of the device or face is sufficient to avoid a full global mapping approach, which saves memory and computing resources. This allows users of the electronic eyeglass devices 100 to add virtual 3D content and see the correctly positioned 3D content through their electronic eyeglass devices 100. Each user can modify the virtual 3D content at the same time. The present disclosure also prevents piracy issues because it does not require sharing of images between electronic eyeglass devices 100. Users only need to be able to see other users from a roughly frontal position for a short period of time.

[0099] Since the electronic eyewear devices 100 (or any rigidly mounted parts or features, like the wearer's face) are visible to each other very quickly, the ego-motion alignment method is novel. Ego-motion alignment provides unique information and sufficient constraints to solve the alignment problem because it does not require each electronic eyewear device to see the same part of the scene, such as the reference Figure 8 As described.

[0100] In this example, a system and method for aligning an electronic eyewear apparatus 100 for two users (User A and User B) is described, although any number of users is possible in the present disclosure.

[0101] Each of user A and user B starts her / his electronic glasses device 100A and 100B, wherein the corresponding processor 932 runs the 6DOF attitude tracker 1100 in the background to track the trajectory (position) of the corresponding electronic glasses device 100A and 100B. The 6DOF attitude tracker 1100 is a 3D or VIO attitude tracker that uses the IMU 972 and camera images 758A and 758B to track the device attitude in six degrees of freedom (x, y, z, pitch, yaw, roll) to estimate the attitude of the corresponding IMU 972. The IMU 972 is a reference body, not a camera, and therefore the user's complete attitude is determined at any time (including regular timestamps). The user's complete 6DOF attitude represents the translation and orientation of the IMU 972, and the translation and orientation of the IMU 972 are expressed in a coordinate system whose origin is at the start of operation of the electronic glasses device 100. In the case of smart glasses including the electronic eyewear device 100, the IMU 972 and the one or more cameras 114A and 114B are rigid parts of the electronic eyewear device 100, where the rigid transformations are known via the electronic eyewear device calibration process. This means that the VIO pose reflects the pose (position and orientation) of the wearer's face in 3D space. An off-the-shelf VIO pose tracker can be used for the purpose of the self-motion alignment method. There are commercially available VIO pose trackers, such as those from the NVIDIA Geometrics of Mountain View, California. ARCore TM and ARKit from Apple in Cupertino, California TM .

[0102] The two electronic eyewear devices 100 have synchronized clocks, and therefore the time synchronized VIO posture is synchronized, accomplished, for example, via the Network Time Protocol (NTP) 1102, which is a known networking protocol for clock synchronization between computer systems on a packet-switched, variable latency data network.

[0103] The VIO attitude trackers 1100 are gravity aligned. This means that one of their coordinate axes (usually the z coordinate) is oriented toward the center of the earth. The IMU 972 determines the gravity alignment. Therefore, the remaining rotational ambiguity to be estimated is only one-dimensional. Therefore, only one angle needs to be estimated for the orientation part of the alignment transformation. For the translation part, three numbers (x, y, z) need to be estimated. For the rotation angles, the alignment of the "roll" and "pitch" angles is implicitly done due to the initialization of each VIO attitude tracking system 1100 against gravity. However, each VIO initialization has a "yaw" angle ambiguity, which is resolved as follows.

[0104] The processor 932 of each wearer's electronic eyeglass device 100 begins a face detection method using the face detector 1104. The processor 932 and cameras 114A and 114B may detect the faces of other electronic eyeglass device users using off-the-shelf methods such as computer vision algorithms implementing eigenfacial techniques.

[0105] There are two options for self-motion alignment. According to one option, when the face detector 1104 is running, the processor 932 selects a fixed point P1 on the symmetry plane on the face (e.g., nose, chin, etc.) of the other user, and outputs its x, y coordinates in each image 758A and 758B (e.g., at a rate of 30fps) for further processing. The symmetry plane is important for fully constraining the problem and making it solvable. In this case, an additional unknown is added to the system, the lever arm distance from the IMU 972 of another electronic eyewear device to the point P1, such as the distance from the IMU 972 to the user's nose in this example.

[0106] In a second option, the processor 932 uses the facial detection of the first option (e.g., the bounding box of the face in images 758A and 758B) to initialize a more accurate electronic eyeglass device 100 tracker 1106. A complete 3D model of the electronic eyeglass device 100 (such as a computer-aided design (CAD) model) is known and can be stored in the memory 934 of the electronic eyeglass device 100, for example, by the manufacturer, and the processor 932 runs a more complex and accurate tracking algorithm to track point P2 on the electronic eyeglass device 100 rather than on the face. In this case, since the relationship of the tracking point P2 on the electronic eyeglass device 100 to the IMU 972 is known using the CAD model, the additional unknown lever arm is eliminated. Both options are solvable, but the second option has higher robustness to noise on the tracked point. Again, knowing the x, y coordinates of point P2 in some camera images 758A and 758B is where the detection is successful.

[0107] For the pose of each camera image (in a global shutter camera) or image line (in a rolling shutter camera), the processor 932 detects the facial feature P1 of another user or the electronic glasses device feature P2 of another user together with its x, y image coordinates. This detection is performed by the processor 932 of each electronic glasses device 100 for user A and user B. It is not mandatory for each electronic glasses device 100 to see the other electronic glasses device 100 or user at the same time. This means that user A can see user B (for example, within the first 5 seconds), and then there may be a time interval where they do not see each other at all, and then user B quickly sees user A, and then user A sees the gain of user B again, and so on. The facial / electronic glasses trajectory and its corresponding 6DOF VIO pose as a fragment of the trail followed by the corresponding mobile electronic glasses device 100 are sent to the server system 998, or to another electronic glasses device 100 where processing occurs.

[0108] The processor 932 uses the x, y coordinates of the user's face or electronic eyewear device trajectory generated by the tracker 1106 and their corresponding gravity 6DOF pose aligned and time synchronized VIO pose tracker 1108 to construct a matrix to obtain a quadratic eigenvalue problem (QEP). QEP is a well-understood problem in linear algebra and can be solved in closed form by the processor 932 as a generalized eigenvalue problem. Solving the QEP using the processor 932 only requires a trace with 3 or 4 positions. The minimal solver requires 4 points (in the case of facial feature tracking) or 3 points (in the case of electronic eyewear device tracking), which allows it to be embedded in a robust estimation procedure. QEP allows for over-constrained as well as minimal solutions.

[0109] The output of the QEP is the yaw angle difference and the 3D transform of the alignment transform. In addition, in case of tracking the facial point P1 (eg, nose), the 3D lever arm unknown vector is also estimated for each user.

[0110] The electronic-glasses ego-motion alignment transform 1110 can be estimated within a few seconds, but as more new face / electronic-glasses detections become available, the estimate can be refined and therefore more accurate.

[0111] Refinement can be viewed as solving an over-constrained QEP or more accurately a nonlinear optimization. The nonlinear optimization can be formulated as a gradient descent-based minimizer with a sparse matrix or as a Gaussian Loopy Belief Propagation on a factor graph, both with robust kernels to handle outliers.

[0112] Once the processor 932 estimates the ego-motion alignment transform 1110, it allows the processor 932 to transform the virtual 3D content 1112 from the local coordinate system of one user / electronic eyewear device to the other user / electronic eyewear device at 1114. The same virtual 3D content 1112 can then be projected and properly rendered into both users' electronic eyewear devices 100 so that the projected 3D content is synchronized and properly displayed, i.e., rendered from the correct perspective of each user.

[0113] Fig.12 1200 is a flowchart depicting a method for implementing an augmented reality application, the method comprising aligning the VIO trajectory of an electronic eyeglass device with the VIO trajectory of a remote electronic eyeglass device to generate a self-motion alignment of the electronic eyeglass device. Although the steps are described with reference to the electronic eyeglass device 100 as described herein, other implementations of the steps described for other types of devices will be understood by those skilled in the art from the description herein. In addition, it is contemplated that Fig.12 One or more of the steps shown in and other figures and described herein may be omitted, performed simultaneously or sequentially, performed in an order other than that shown and described, or performed in conjunction with additional steps.

[0114] At box 1202, the processor 932 of the electronic eyeglass device 100A uses the VIO posture tracker 1100 to generate the trajectory of the electronic eyeglass device 100A. The VIO posture tracker 1100 is a 6DOF posture tracker that uses the IMU 972 and camera images 758A and 758B to track the head posture of the user A in six degrees of freedom (x, y, z, pitch, yaw, roll) to estimate the posture of the corresponding IMU 972. The IMU 972 is a reference body rather than a camera, and therefore determines the complete posture of the user at regular moments. The user's complete 6DOF posture represents the translation and orientation of the IMU 972, and the translation and orientation of the IMU 972 are expressed in a coordinate system whose origin is at the beginning of the operation of the electronic eyeglass device 100A. In the case of smart glasses including the electronic eyeglass device 100, the IMU 972 and one or more cameras 114A and 114B are rigid parts of the electronic eyeglass device 100, where the rigid transformation is known via the electronic eyeglass device calibration process. This means that the VIO pose reflects the pose (position and orientation) of the wearer's face in 3D space.

[0115] At box 1204, the processor 932 uses the face detector 1104 of the electronic eyeglass device 100A to detect a remote user B wearing a remote electronic eyeglass device 100B in the environment, wherein the remote electronic eyeglass device 100B is also configured to generate a trajectory using a remote VIO gesture tracker 1100. The processor 932 uses cameras 114A and 114B to detect the face of another electronic eyeglass device user. For the gesture of each camera image (in a global shutter camera) or image row (in a rolling shutter camera), the processor 932 detects facial features P1 or electronic eyeglass device features P2 and their x, y image coordinates. This detection is performed by the processor 932 of each electronic eyeglass device 100A and 100B for user A and user B. It is not mandatory for each electronic eyeglass device 100 to see another electronic eyeglass device 100 or user at the same time. This means that user A may see user B (for example, within the first 5 seconds), then there may be a time interval where they do not see each other at all, then user B quickly sees user A, then user A sees user B again, and so on.

[0116] At box 1206, the processor 932 of the electronic glasses device 100A aligns the VIO trajectory of the electronic glasses device 100A with the VIO trajectory of the remote electronic glasses device 100B to generate a self-motion alignment of the electronic glasses device 100A and the remote electronic glasses device 100B. There are at least two options for self-motion alignment. According to one option, when the face detector 1104 is running, the processor 932 selects a fixed point P1 on the symmetry plane on the face (e.g., nose, chin, etc.) of other users, and outputs its x, y coordinates in each image 758A and 758B (e.g., at a rate of 30fps) for further processing. The symmetry plane is important for fully constraining the problem and making it solvable. In this case, an additional unknown is added to the system, the lever arm distance from the IMU 972 to the point P1, such as the distance from the IMU 972 to the user's nose in this example. It stems from the fact that this lever arm distance is unique but unknown for each user.

[0117] In the second option, the processor 932 uses the facial detection of the first option (e.g., the bounding box of the face in images 758A and 758B) to initialize a more accurate electronic eyeglass device 100 detector 1106. A complete 3D model of the electronic eyeglass device 100 (such as a computer-aided design (CAD) model) is known and stored in the memory 934 because the electronic eyeglass device 100 is manufactured by the same manufacturer, and the processor 932 runs a more complex and accurate tracking algorithm to track point P2 on the electronic eyeglass device 100 rather than on the face. In this case, since the relationship of the tracking point P2 on the electronic eyeglass device 100 to the IMU 972 is known using the CAD model, the additional unknown lever arm is eliminated. Both options are solvable, but the second option has higher robustness to noise on the tracking points. Again, knowing the x, y coordinates of point P2 in some camera images 758A and 758B is where the detection is successful.

[0118] As about Figure 8 and Fig.13 As described, a first electronic eyeglass device 100A operated by user A and a second electronic eyeglass device 100B operated by user B can each use the described SLAM algorithm to determine its (x, y, z) position in the environment 1300 shown as an interior room. Multiple electronic eyeglass devices 100 can determine their positions in the same coordinate system for multi-player interaction, such as user C with a third electronic eyeglass 100C, and no restrictions are inferred on the two electronic eyeglass devices 100 in the environment 1300. Each electronic eyeglass device 100A and 100B can communicate with each other via wireless circuit 924 (such as and WiFi) or via wireless circuit 936 and network 995 ( Fig. 9 ) shares its (x, y, z) location via messaging using short-range communication technology.

[0119] Optionally, to perform self-motion alignment, the processor 932 of each user's electronic eyeglass device 100 also determines physical attributes, such as a face or head, of a user of another electronic eyeglass device 100 in the interior room. The processor 932 is configured to identify physical attributes in multiple frames generated by cameras 114A and 114B. For example, the processor 932 uses facial recognition software to determine the face or mouth of another user. The electronic eyeglass device 100 can also store facial images in an image library (such as a database) to perform automatic recognition of another user by comparing the detected facial image with the image database for matching.

[0120] In one example, when electronic eyeglass devices 100A and 100B are in session and communicate with each other, such as via a network 995 or directly using low-power signals (such as based on BluetoothTM), the corresponding (x, y, z) coordinate position of each electronic eyeglass device 100 is automatically shared with the other electronic eyeglass device.

[0121] In another example, the electronic eyeglass device 100 of one user may share its (x, y, z) coordinate position with another electronic eyeglass device 100 by using the mobile device 990. For example, the electronic eyeglass device 100B of user B may share its (x, y, z) coordinate position with the electronic eyeglass device 100A using the synchronized mobile device 990 of user B.

[0122] Multiple electronic eyeglass devices 100 can determine their positions in the same coordinate system for multi-player interaction, and no limitation on two electronic eyeglass devices 100 in the environment 1100 is inferred. Each electronic eyeglass device 100A and 100B can communicate with each other via wireless circuit 924 (such as and WiFi) or via wireless circuit 936 and network 995 ( Fig. 9 ) shares its (x, y, z) location via messaging using short-range communication technology.

[0123] Fig.13 Further illustrated is the determination of the position of the electronic eyewear apparatus 100A-100C performed using the microphone array 145 in conjunction with the electronic eyewear apparatus 100 by using the beamforming utility 972 ( Fig. 9 ) Demixing speech / sound from the user's surroundings. Many audio and audio-visual applications in AR require demixing of speech / sound and / or estimating the direction of arrival (DOA) of sound. The problem of demixing speech becomes more complicated when the audio source is moving, and its position is unknown. This problem is solved by simplifying the audio source separation model using the known 3D position / orientation of multiple electronic glasses users to assist in speech / sound demixing.

[0124] The 6DOF tracker 1100 runs on each electronic eyewear device and tracks its own ego motion 1320A-1320C. The 3D position and orientation of the electronic eyewear device 100B and the electronic eyewear device 100C are known by the coordinate system 1310A of the electronic eyewear device 100A through the alignment of the coordinate systems 1310A, 1310B, 1310C of the 6DOF tracker. Fig.11The described method performs alignment. In one embodiment, continuous visual tracking of the remote user is not required. Alignment of the coordinate systems of the 6DOF trackers occurs once at the beginning of the AR session. Each 6DOF tracker provides a 6D pose of each user in a common coordinate system. Based on the initial alignment and the ego-motion data, the relative positions of objects are known at all times, even if the objects are moving.

[0125] The microphone array 145 of the electronic glasses device 100A of user A receives a mixture of speech and sound, the speech 1302 of user B, the speech 1304 of user C, and background noise 1306. The 3D position of the sound source (e.g., the lips of user B and user C) is approximated by the position of the corresponding electronic glasses devices 100B and 100C. This is done by using a simple facial model, and the position of the lips relative to the electronic glasses device 100 can be accurately estimated. By utilizing the alignment of the 6DOF tracker, the system knows the position and orientation of the speech sources 1302 and 1304 with high accuracy. This results in the audio separation model used by the beamforming utility 972 being simplified. The model is simplified to audio source separation using the known position and orientation of the sound sources that generate the speech 1302 and 1304. In addition, the known relative motion and position of the source allows the elimination of sounds from distant sources such as determined by the microphone array 145 when isolating the speech of other users of the electronic glasses 100 and users who do not have electronic glasses. In one example, user B is at a 7 o'clock angle relative to user A, and user C is at a 4 o'clock angle relative to user A. If user C is at a distance of 6 meters relative to user A and user B is at a distance of 2 meters relative to user A, the aligned ego motion provides this data to the beamformer utility 972 to apply different focal points to the microphone array 145 to simplify demixing of the audio signals.

[0126] In another example, user A additionally wears a pair of earplugs coupled to the electronic eyewear device 100. User A is interested in hearing the voice from user B, but not from user C or the background. The beamforming direction is dynamically focused in the direction of user B and relays the isolated / purified voice of user B. User A can select the desired direction to isolate the sound by interacting with the temples of the electronic glasses or the paired earplugs. This is similar to the natural behavior of a person trying to listen in a specific direction by placing a hand on the ear.

[0127] Fig.14is a flow chart depicting a method 1400 for performing audio source demixing using aligned VIO tracks of an electronic eyeglass and one or more remote electronic eyeglass devices. Although the steps are described with reference to an electronic eyeglass device 100 as described herein, other implementations of the steps described for other types of devices will be apparent to one skilled in the art from the description herein. Multiple electronic eyeglass devices 100 may determine their positions in the same coordinate system for multi-user interaction, and no limitation to two electronic eyeglass devices 100 used in audio source demixing is inferred. Furthermore, it is contemplated that in Fig.14 One or more of the steps shown in the drawings and in other figures and described herein may be omitted, performed simultaneously or sequentially, performed in an order other than shown and described, or performed in combination with additional steps.

[0128] At box 1402, the processor 932 of the electronic glasses device 100A aligns the VIO trajectory of the electronic glasses device 100A with the VIO trajectory of the remote electronic glasses device 100B to generate the self-motion alignment of the electronic glasses device 100A and the remote electronic glasses device 100B. The VIO posture tracker 1100 is a 6DOF posture tracker that uses the IMU 972 and camera images 758A and 758B to track the head posture of the user of the electronic glasses 100 in six degrees of freedom (x, y, z, pitch, yaw, roll) to estimate the posture of the corresponding IMU 972. The electronic glasses device 100A and the remote electronic glasses device 100B can be aligned by various methods, such as by using points, markers or maps. Multiple remote electronic glasses devices can exist in the system and method.

[0129] At block 1404, the VIO pose trackers 1100 of the electronic eyeglass apparatus 100A and the remote electronic eyeglass apparatus 100B are utilized to approximate the 3D position of the remote electronic eyeglass apparatus 100B worn by the user B that generates the sound (such as speech 1302). For example, the position of the lips of the remote user B is estimated by the position of the electronic eyeglass apparatus 100B. This is accomplished by using a simple facial model, in which the position of the lips relative to the remote electronic eyeglass apparatus can be accurately estimated.

[0130] At block 1406, the processor 932 separates the audio signals (i.e., speech) generated by user B and user C. The processor 932 uses the 3D position of user B generating the speech 1302 to demix and isolate the speech 1302 from other speech and sounds produced by other sound sources in the surrounding environment 1300 of the electronic eyewear device 100A, such as the speech 1304 generated by user C.

[0131] It will be understood that the terms and expressions used herein have ordinary meanings, such as conforming to such terms and expressions for their corresponding relative queries and research fields, except for the specific meanings that have been otherwise set forth herein. Relational terms such as first and second, etc., can be used only to distinguish one entity or action from another entity or action, without requiring or implying any actual such relationship or order between these entities or actions. The terms "comprises", "comprising", "includes", "including" or any other variation thereof are intended to cover non-exclusive inclusions, so that a process, method, article or device that includes or contains a series of elements or steps includes not only those elements or steps, but also other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. Without further limitation, an element preceded by "one" or "a kind of" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0132] Unless otherwise stated, all or any measurements, values, ratings, positions, amplitudes, sizes, and other specifications set forth in this specification (including in the following claims) are approximate and imprecise. Such quantities are intended to have reasonable ranges consistent with the functions to which they are related and with the conventional art to which they belong. For example, parameter values, etc. may deviate from the stated quantity by as much as ±10%, unless otherwise expressly stated.

[0133] In addition, in the above detailed description, it can be seen that different features are grouped together in different examples for the purpose of simplifying the present disclosure. This method of the present disclosure should not be interpreted as reflecting the intention that the examples claimed for protection require more features than the features explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in fewer features than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim independently serving as a separately claimed subject matter.

[0134] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in different forms and examples, and they may be applied to many applications, only some of which are described herein. The appended claims are intended to claim any and all modifications and variations that fall within the practical scope of the present concept.

Claims

1. An electronic glasses device, comprising: Inertial Measurement Unit (IMU); Posture tracker; a microphone array configured to receive sound from audio sources in the environment and generate an audio signal; a head mounted mirror frame, the head mounted mirror frame supporting the inertial measurement unit, the attitude tracker and the microphone array; as well as A processor, the processor being configured to: generating a trajectory of the electronic eyewear device using the gesture tracker; receiving a trajectory of the remote electronic eyeglass device generated by the remote electronic eyeglass device, and determining a position of the remote electronic eyeglass device from the received trajectory; as well as The received sound is demixed based on the determined position of the remote electronic device.

2. The electronic eyeglass device according to claim 1, wherein: The processor is further configured to establish a shared coordinate system between the electronic-glasses device and the remote electronic-glasses device using the trajectory.

3. The electronic eyeglass device according to claim 2, wherein: The processor is also configured to demix the received sound using the trajectory and the microphone array.

4. The electronic eyeglass device according to claim 3, wherein: The processor is further configured to determine a direction of arrival of the received sound relative to the electronic eyewear device.

5. The electronic glasses device according to claim 3, further comprising: At least one speaker is supported by the head mounted frame to present the demixed speech to a user of the electronic eyewear device.

6. The electronic eyeglass device according to claim 1, wherein: The electronic eyeglass device is configured to have an orientation calibrated to the remote electronic eyeglass device.

7. The electronic eyeglass device according to claim 2, wherein: The processor is configured to: identifying a point associated with a remote user wearing the remote electronic eyewear device and using the identified point to align the trajectory of the gesture tracker with the trajectory of the remote electronic eyewear device; as well as The aligned trajectory is used to transform the pose of the electronic eyewear device into the shared coordinate system.

8. The electronic eyeglass device according to claim 1, wherein: The electronic eyeglass device further includes a beamformer configured to separate the audio signal based on the determined position of the remote electronic device.

9. An interactive augmented reality method using an electronic eyewear device having an inertial measurement unit (IMU), a gesture tracker, a microphone array configured to receive sound from an environment and generate an audio signal, a head-mounted frame, and a processor, the head-mounted frame supporting the inertial measurement unit, the gesture tracker, and the microphone array, the processor: generating a trajectory of the electronic eyewear device using the gesture tracker; receiving a trajectory of the remote electronic eyeglass device generated by the remote electronic eyeglass device, and determining a position of the remote electronic eyeglass device from the received trajectory; as well as The received sound is demixed based on the determined position of the remote electronic eyewear device.

10. The method according to claim 9, wherein: The processor uses the trajectory to establish a shared coordinate system between the electronic-glass device and the remote electronic-glass device.

11. The method according to claim 10, wherein: The processor demixes the received sound using the trajectory and the microphone array.

12. The method according to claim 11, wherein: The processor determines a direction of arrival of the received sound relative to the electronic eyewear device.

13. The method of claim 11, further comprising at least one speaker supported by the head mounted frame and presenting the demixed speech to a user of the electronic eyewear device.

14. The method according to claim 9, wherein: The electronic eyeglass device has an orientation calibrated to the remote electronic eyeglass device.

15. The method according to claim 11, wherein: The processor: identifying a point associated with a remote user wearing the remote electronic eyeglass device and using the identified point to align a trajectory of the electronic eyeglass device with a trajectory of the remote electronic eyeglass device; as well as The aligned trajectory is used to transform the pose of the electronic eyewear device into the shared coordinate system.

16. The method according to claim 9, wherein: The electronic eyeglass device further includes a beamformer configured to separate the audio signal based on the determined position of the remote electronic device.

17. A non-transitory computer readable medium storing program code, the program code, when executed, for causing an electronic processor of an electronic eyewear device having an inertial measurement unit (IMU), a gesture tracker, a microphone array configured to receive sound from an environment and generate an audio signal, and a head mounted frame supporting the inertial measurement unit, the gesture tracker, and the microphone array to perform the following steps: generating a trajectory of the electronic eyewear device using the gesture tracker; receiving a trajectory of the remote electronic eyeglass device generated by the remote electronic eyeglass device, and determining a position of the remote electronic eyeglass device from the received trajectory; as well as The received sound is demixed based on the determined position of the remote electronic eyewear device.

18. The non-transitory computer readable medium storing program code of claim 17, further comprising code for the processor to establish a shared coordinate system between the electronic eyeglass device and the remote electronic eyeglass device using the trajectory.

19. The non-transitory computer readable medium storing program code according to claim 18, further comprising code for: The processor demixes the received sound using the trajectory and the microphone array.

20. The non-transitory computer readable medium storing program code according to claim 18, further comprising code for: The processor determines a direction of arrival of the received sound relative to the electronic eyewear device.