Eyewear determining facial expressions using muscle sensors

Abstract

Eyeglasses having a sensor configured to detect electrical signals generated from user muscle movements to determine user facial expressions. The facial expressions are processed to provide user input to the eyeglasses, perform actions such as taking an image using a camera, and determine user biometric authentication such as performing an electrocardiogram (ECG or EKG). In an example, the user may raise their eyebrows to command the eyeglasses to take an image and squint to lighten / darken the shade of an optical element.

Description

Technology Field

[0001] This application claims priority to U.S. application serial number 16 / 822,361, titled EYEWEAR DETERMINING FACIAL EXPRESSIONS USING MUSCLE SENSORS, filed on March 18, 2020, the contents of which are fully incorporated herein by reference.

[0002] This topic concerns eyewear devices, such as smart glasses and see-through displays. Background Technology

[0003] Portable eyewear devices available today, such as smart glasses, headwear, and headgear, integrate cameras and see-through displays. Generally, eyewear comprising frames and temples is designed to be worn and provides eyewear features. Brief explanation of the drawing

[0004] The drawings depict one or more implementations merely as examples, not limitations. In the drawings, the same reference numbers refer to identical or similar elements.

[0005] FIG. 1a is a side view of an exemplary hardware configuration of an eyeglass device illustrating a right optical assembly having an image display, and field of view adjustments are applied to a user interface presented on the image display based on head or eye movements detected by the user.

[0006] FIG. 1b is a cross-sectional view of the temple of FIG. 1a depicting a visible light camera, a head movement tracker for tracking the head movements of the user of the eyeglass device, and a circuit board.

[0007] FIG. 2a is a rear view of an exemplary hardware configuration of an eyewear device including an eye scanner on a frame for use in a system for identifying a user of the eyewear device.

[0008] FIG. 2b is a rear view of an exemplary hardware configuration of another eyewear device including an eye scanner on a temple for use in a system for identifying a user of the eyewear device.

[0009] FIGS. 2C and FIGS. 2D are rear views of an exemplary hardware configuration of an eyeglass device including two different types of image displays.

[0010] FIG. 2e illustrates a rear view of an exemplary hardware configuration having muscle sensors for determining facial muscle movements.

[0011] FIG. 3 shows a rear perspective view of the eyeglass device of FIG. 2a depicting an infrared emitter, an infrared camera, a front frame, a rear frame, and a circuit board.

[0012] Figure 4 is a cross-sectional view taken through the infrared emitter and frame of the eyeglass device of Figure 3.

[0013] Figure 5 illustrates the detection of eye gaze direction.

[0014] Figure 6 illustrates the detection of eye position.

[0015] Figure 7 depicts an example of visible light captured by the left visible light camera as the left raw image and visible light captured by the right visible light camera as the right raw image.

[0016] FIG. 8 illustrates a muscle sensor that processes electrical signals and determines facial expressions based on detected electrical signals.

[0017] FIG. 9 illustrates a block diagram of the electronic components of an eyeglass device.

[0018] FIG. 10 illustrates a method for processing signals from sensors, determining facial expressions, and performing actions or functions. Specific details for implementing the invention

[0019] The present disclosure relates to eyewear having a sensor configured to detect electrical signals generated from user muscle movements to determine user facial expressions. The facial expressions are processed to provide user input to the eyewear, perform actions such as taking an image using a camera, and determine user biometric authentication such as performing an electrocardiogram (ECG or EKG). In an example, the user may raise their eyebrows to command the eyewear to take an image and squint to lighten / darken the shade of an optical element.

[0020] Additional objects, advantages, and novel features of the embodiments will be described in part in the following description and will become apparent to those skilled in the art by reviewing the following and the accompanying drawings, or can be learned by the creation or operation of the examples. Objects and advantages of the subject matter may be realized and achieved by the methodologies, means, and combinations specifically indicated in the appended claims.

[0021] In the following detailed description, numerous specific details are provided as examples to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that these teachings can be practiced without such details. In other instances, well-known methods, procedures, components, and circuits are not described in detail at a relatively high level to avoid unnecessarily obscuring the modes of these teachings.

[0022] As used herein, the term “coupled” refers to any logical, optical, physical, or electrical connection, link, etc., through which signals or light generated or supplied by one system element are transmitted to another coupled element. Unless otherwise described, coupled elements or devices do not need to be directly connected to one another and may be separated by intermediate components, elements, or communication media capable of modifying, manipulating, or transmitting light or signals.

[0023] As illustrated in any drawings, the orientations of any complete device integrating an eyepiece device, associated components, and an eye scanner and camera are provided only as examples for the purposes of illustration and discussion. In operation for a specific variable optical processing application, the eyepiece device may be oriented in any other direction suitable for the specific application of the eyepiece device, e.g., up, down, side, or any other orientation. Furthermore, to the extent used herein, any directional terms such as forward, rear, inward, outward, oriented, left, right, side, longitudinal, up, down, upper, lower, top, bottom, and side are merely illustrative and do not limit the orientation or orientation of any optical device or component of an optical device configured as otherwise described herein.

[0024] Now, the examples illustrated in the attached drawings and discussed below are referenced in detail.

[0025] FIG. 1a is a side view of an exemplary hardware configuration of an eyeglass device (100) comprising a right optical assembly (180B) having an image display (180D) (Fig. 2a). The eyeglass device (100) comprises a plurality of visible light cameras (114A-B) (Fig. 7) forming a stereo camera, wherein the right visible light camera (114B) is located on the right temple (110B).

[0026] The left and right visible light cameras (114A-B) have image sensors sensitive to wavelengths in the visible light range. Each visible light camera (114A-B) has a different forward-facing coverage angle, for example, the visible light camera (114B) has the described coverage angle (111B). The coverage angle is the angular range in which the image sensor of the visible light camera (114A-B) picks up electromagnetic radiation to generate images. Examples of these visible light cameras (114A-B) include high-resolution CMOS (Complementary Metal-Oxide-Semiconductor) image sensors and Video Graphics Array (VGA) cameras such as 640p (e.g., 640 x 480 pixels for a total of 0.3 megapixels), 720p, or 1080p. Image sensor data from the visible light cameras (114A-B) is captured along with geolocation data, digitized by an image processor, and stored in memory.

[0027] To provide stereoscopic vision, visible light cameras (114A-B) may be coupled to an image processor (element (912) in FIG. 9) for digital processing, along with a timestamp at which an image of the scene is captured. The image processor (912) includes circuitry for receiving signals from the visible light cameras (114A-B) and processing these signals from the visible light cameras (114A-B) into a format suitable for storing in memory (element (934) in FIG. 9). The timestamp may be added by the image processor (912) or another processor controlling the operation of the visible light cameras (114A-B). The visible light cameras (114A-B) allow the stereo camera to simulate human binocular vision. The stereo cameras provide the ability to reproduce three-dimensional images (elements (715) in FIG. 7) based on two captured images (elements (758A-B) in FIG. 7) from visible light cameras (114A-B) each having the same timestamp. These three-dimensional images (715) allow for an immersive, realistic experience, for example, for virtual reality or video games. In the case of stereoscopic vision, a pair of images (758A-B) are generated at a given moment, that is, one image is generated for each of the left and right visible light cameras (114A-B). When a pair of images (758A-B) generated from the forward-facing coverage angles (111A-B) of the left and right visible light cameras (114A-B) are stitched together (e.g., by an image processor (912)), depth perception is provided by the optical assembly (180A-B).

[0028] In the example, the user interface field of view adjustment system includes an eyeglass device (100). The eyeglass device (100) includes a frame (105), a right temple (110B) extending from the right side (170B) of the frame (105), and a projection image display (180D) (Fig. 2A-B) comprising an optical assembly (180B) for presenting a graphic user interface to the user. The eyeglass device (100) includes a left visible light camera (114A) connected to the frame (105) or the left temple (110A) to capture a first image of the scene. The eyeglass device (100) further includes a right visible light camera (114B) connected to the frame (105) or the right temple (110B) to capture a second image of the scene that partially overlaps with the first image (e.g., simultaneously with the left visible light camera (114A)). Although not illustrated in FIG. 1a-1b, the user interface field of view adjustment system further includes a processor (932) coupled to the eyeglass device (100) and connected to visible light cameras (114A-B), a memory (934) accessible to the processor (932), and programming within the memory (934), for example, within the eyeglass device (100) itself or other parts of the user interface field of view adjustment system.

[0029] Although not shown in FIG. 1a, the eyewear device (100) also includes a head movement tracker (element (109) in FIG. 1b) or an eye movement tracker (element (213) in FIG. 2a-b). The eyewear device (100) further includes projection image displays (180C-D) of an optical assembly (180A-B) for presenting a sequence of displayed images, and an image display driver (element (942) in FIG. 9) coupled to the projection image displays (180C-D) of the optical assembly (180A-B) to control the image displays (180C-D) of the optical assembly (180A-B) for presenting a sequence of displayed images (715), which is described in more detail below. The eyewear device (100) further includes a memory (934), and a processor (932) that accesses the image display driver (942) and the memory (934). The eyeglass device (100) further includes programming in memory (element (934) of FIG. 9). The execution of programming by the processor (932) configures the eyeglass device (100) to perform functions including presenting an initial displayed image of a sequence of displayed images through the projection image displays (180C-D), and the initial displayed image has an initial field of view (element 230 of FIG. 5) corresponding to an initial head direction or an initial eye gaze direction.

[0030] Execution of programming by the processor (932) further configures the eyewear device (100) to detect the movement of the user of the eyewear device by (i) tracking the user's head movement through a head movement tracker (element (109) of FIG. 1b) or (ii) tracking the eye movement of the user's eyes through an eye movement tracker (elements (113, 213) of FIG. 2a-b and FIG. 5). Execution of programming by the processor (932) further configures the eyewear device (100) to determine a field of view adjustment for an initial field of view of an initially displayed image based on the detected movement of the user. The field of view adjustment includes a continuous field of view corresponding to a continuous head direction or a continuous eye direction. Execution of programming by the processor (932) further configures the eyewear device (100) to generate a continuous displayed image of a sequence of displayed images based on the field of view adjustment. The execution of programming by the processor (932) further configures the eyeglass device (100) to present images continuously displayed through the projection image displays (180C-D) of the optical assembly (180A-B).

[0031] FIG. 1b is a top cross-sectional view of the temple of FIG. 1a depicting a right visible light camera (114B), a head movement tracker (109), and a circuit board. The configuration and arrangement of the left visible light camera (114A) are substantially similar to the right visible light camera (114B), except that the connections and couplings are on the left side (170A). As illustrated, the eyeglass device (100) includes a right visible light camera (114B) and a circuit board which may be a flexible printed circuit board (PCB) (140). A right hinge (126B) connects the right temple (110B) to the right temple (125B) of the eyeglass device (100). In some examples, components of the right visible light camera (114B), flexible PCB (140), or other electrical connectors or contacts may be located on the right temple (125B) or right hinge (126B).

[0032] As described, the eyeglass device (100) has a head movement tracker (109) that includes, for example, an inertial measurement unit (IMU). An inertial measurement unit is an electronic device that measures and reports specific forces, angular velocities, and sometimes magnetic fields surrounding the body using a combination of accelerometers, gyroscopes, and sometimes magnetometers. An inertial measurement unit operates by detecting linear acceleration using one or more accelerometers and detecting rotational velocity using one or more gyroscopes. A typical configuration of inertial measurement units includes one accelerometer, gyroscope, and magnetometer per axis for each of the three axes: a horizontal axis (X) for left-right movement, a vertical axis (Y) for up-down movement, and a depth or distance axis for up-down movement (Z). The accelerometer detects gravity vectors. The magnetometer defines rotations of the magnetic field (e.g., south, north, etc.), such as a compass that generates a directional reference. Three accelerometers detect acceleration along the horizontal, vertical, and depth axes defined above, which may be defined in relation to the ground, the eyeglass device (100), or a user wearing the eyeglass device (100).

[0033] The eyeglass device (100) detects the movement of the user of the eyeglass device (100) by tracking the movement of the user's head through a head movement tracker (109). The head movement includes a variation in the direction of the head relative to the horizontal axis, the vertical axis, or a combination thereof from the initial head direction while presenting an image initially displayed on an image display. In one example, the head movement of the user's head through the head movement tracker (109) includes measuring the initial head direction on the horizontal axis (e.g., X-axis), the vertical axis (e.g., Y-axis), or a combination thereof (e.g., horizontal or vertical movement) through an inertial measurement unit (109). Tracking the head movement of the user's head through the head movement tracker (109) further includes measuring the continuous head direction relative to the horizontal axis, the vertical axis, or a combination thereof during the presentation of the initially displayed image through the inertial measurement unit (109).

[0034] Tracking the movement of the user's head through the head movement tracker (109) further includes determining a variation in head direction based on both the initial head direction and the subsequent head direction. Detecting the movement of the user of the eyewear device (100) further includes determining that the variation in head direction exceeds a deviation angle threshold on the horizontal axis, the vertical axis, or a combination thereof in response to tracking the movement of the user's head through the head movement tracker (109). The deviation angle threshold is approximately 3° to 10°. As used herein, when referring to an angle, the term "approximately" means ± 10% of the specified amount.

[0035] Variations along the horizontal axis slide three-dimensional objects, such as characters, Bitmojis, and application icons, in and out of the field of view, for example, by hiding, unhiding, or otherwise adjusting the visibility of the three-dimensional objects. Variations along the vertical axis display, for example, weather information, time, date, calendar appointments, etc., when the user looks upward. In other examples, when the user looks downward on the vertical axis, the glasses device (100) may be turned off.

[0036] The right temple (110B) includes a temple body (211) and a temple cap, and in the cross-section of FIG. 1b, the temple cap is omitted. Inside the right temple (110B), various interconnected circuit boards are arranged, such as PCBs or flexible PCBs, including controller circuits for a right visible light camera (114B), microphone(s) (130), speaker(s) (132), low-power wireless circuits (e.g., for wireless short-range network communication via Bluetooth™), and high-speed wireless circuits (e.g., for wireless short-range network communication via WiFi).

[0037] The right visible light camera (114B) is covered by a visible light camera cover lens that is coupled to or placed on a flexible PCB (140) and aimed through an opening(s) formed in the right temple (110B). In some examples, a frame (105) connected to the right temple (110B) includes an opening(s) for the visible light camera cover lens. The frame (105) includes a front-facing side configured to face outward away from the user's eyes. An opening for the visible light camera cover lens is formed on and through the front-facing side. In an example, the right visible light camera (114B) has an outward-facing coverage angle (111B) with the line of sight or projection of the user's right eye of the eyeglass device (100). The visible light camera cover lens may also have an outward-facing coverage angle but may be attached to an outward-facing surface of the right temple (110B) where an opening is formed in a different outward direction. The coupling may also be indirect through intermediate components.

[0038] The left (first) visible light camera (114A) is connected to the left perspective image display (180C) of the left optical assembly (180A) to generate the first background scene of the first continuously displayed image. The right (second) visible light camera (114B) is connected to the right perspective image display (180D) of the right optical assembly (180B) to generate the second background scene of the second continuously displayed image. The first background scene and the second background scene partially overlap to present a three-dimensional observable area of ​​the continuously displayed image.

[0039] The flexible PCB (140) is placed inside the right temple (110B) and is coupled to one or more other components housed in the right temple (110B). Although it is shown as being formed on the circuit boards of the right temple (110B), the right visible light camera (114B) may be formed on the circuit boards of the left temple (110A), the temples (125A-B), or the frame (105).

[0040] FIG. 2a is a rear view of an exemplary hardware configuration of an eyewear device (100) including an eye scanner (113) on a frame (105) for use in a system for determining the eye position and gaze direction of a wearer / user of the eyewear device (100). As shown in FIG. 2a, the eyewear device (100) is configured to be worn by a user, which is eyewear in the example of FIG. 2a. The eyewear device (100) may take other forms and may incorporate other types of frameworks, such as headgear, a headset, or a helmet.

[0041] In the example of glasses, the spectacle device (100) comprises a frame (105) including a left rim (107A) connected to a right rim (107B) via a bridge (106) adapted to the user's nose. The left and right rims (107A-B) each include apertures (175A-B) that hold respective optical elements (180A-B), such as lenses and viewing displays (180C-D). As used herein, the term lens means covering transparent or translucent pieces of glass or plastic having curved and flat surfaces that cause light to converge / diverge or cause little to no convergence / divergence.

[0042] Although illustrated as having two optical elements (180A-B), the eyeglass device (100) may include other arrangements, such as a single optical element, depending on the application or intended user of the eyeglass device (100). Additionally, as illustrated, the eyeglass device (100) includes a left temple (110A) adjacent to the left side (170A) of the frame (105) and a right temple (110B) adjacent to the right side (170B) of the frame (105). The temples (110A-B) may be implemented as individual components that are integrated into the frame (105) on each side (170A-B) (as illustrated) or attached to the frame (105) on each side (170A-B). Alternatively, the temples (110A-B) may be integrated into temples (not illustrated) attached to the frame (105).

[0043] In the example of FIG. 2a, the eye scanner (113) includes an infrared emitter (115) and an infrared camera (120). A visible light camera typically includes a blue light filter that blocks infrared light detection, and in the example, the infrared camera (120) is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640 x 480 pixels for a total of 0.3 megapixels), with the blue filter removed. The infrared emitter (115) and the infrared camera (120) are positioned together in the frame (105) and are illustrated, for example, as both connected to the upper part of the left rim (107A). One or more of the frame (105) or the left and right temples (110A-B) include a circuit board (not illustrated) containing the infrared emitter (115) and the infrared camera (120). The infrared emitter (115) and the infrared camera (120) can be connected to the circuit board, for example, by soldering.

[0044] Other arrangements of the infrared emitter (115) and infrared camera (120) may be implemented such that both the infrared emitter (115) and the infrared camera (120) are located on the right rim (107B), or at different locations on the frame (105), for example, the infrared emitter (115) is on the left rim (107A) and the infrared camera (120) is on the right rim (107B). In another example, the infrared emitter (115) is on the frame (105) and the infrared camera (120) is on one of the temples (110A-B), or vice versa. The infrared emitter (115) can essentially be connected anywhere on the frame (105), the left temple (110A), or the right temple (110B) to emit a pattern of infrared light. Similarly, an infrared camera (120) can essentially be connected anywhere on the frame (105), the left temple (110A), or the right temple (110B) to capture at least one reflection variation in the emitted pattern of infrared light.

[0045] The infrared emitter (115) and infrared camera (120) are arranged to face inward toward the user's eyes in a partial or full field of view to identify each eye position and gaze direction. For example, the infrared emitter (115) and infrared camera (120) are placed on the temples (110A-B) at the upper part of the frame (105) or at both ends of the frame (105), directly in front of the eyes.

[0046] FIG. 2b is a rear view of an exemplary hardware configuration of another eyewear device (200). In this exemplary configuration, the eyewear device (200) is depicted as including an eye scanner (213) on the right temple (210B). As illustrated, an infrared emitter (215) and an infrared camera (220) are located together on the right temple (210B). It should be understood that the eye scanner (213) or one or more components of the eye scanner (213) may be located on the left temple (210A) and other locations of the eyewear device (200), for example, on the frame (105). The infrared emitter (215) and the infrared camera (220) are similar to those of FIG. 2a, but the eye scanner (213) may be modified to be sensitive to a different light wavelength than previously described in FIG. 2a.

[0047] Similar to FIG. 2a, the eyeglass device (200) comprises a frame (105) including a left rim (107A) connected to a right rim (107B) via a bridge (106); and the left and right rims (107A-B) each include holes that hold each optical element (180A-B) including a see-through display (180C-D).

[0048] FIGS. 2c-d are rear views of exemplary hardware configurations of an eyewear device (100) comprising two different types of projection image displays (180C-D). In one example, these projection image displays (180C-D) of an optical assembly (180A-B) include an integrated image display. As illustrated in FIG. 2c, the optical assemblies (180A-B) include a suitable display matrix (180C-D) of any suitable type, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a waveguide display, or any such display. The optical assembly (180A-B) also includes an optical layer or layers (176) that may include lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components in any combination. The optical layers (176A-N) may include a prism having a suitable size and configuration, comprising a first surface for receiving light from a display matrix and a second surface for emitting light to the user's eye. When the user's eye is looking through the corresponding left and right rims (107A-B), the prism of the optical layers (176A-N) extends across all or at least part of the respective holes (175A-B) formed in the left and right rims (107A-B) so that the user sees the second surface of the prism. The first surface of the prism of the optical layers (176A-N) is oriented upward from the frame (105), and the display matrix is ​​placed over the prism so that photons and light emitted by the display matrix strike the first surface. The prism is sized and shaped so that light is refracted within the prism and directed toward the user's eye by the second surface of the prism of the optical layers (176A-N). In this regard, the second surface of the prism of the optical layers (176A-N) may be convex to direct light toward the center of the eye.The prism may be sized and shaped to optionally magnify an image projected by the projection image displays (180C-D), and light travels through the prism so that the image seen on the second surface is larger in one or more dimensions than the image emitted from the projection image displays (180C-D).

[0049] In another example, the projection image displays (180C-D) of the optical assembly (180A-B) include a projection image display as shown in FIG. 2d. The optical assembly (180A-B) includes a laser projector (150), which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source such as the laser projector (150) is placed on or above one of the temples (125A-B) of the eyeglass device (100). The optical assembly (180A-B) includes one or more optical strips (155A-N) spaced apart across the lens width of the optical assembly (180A-B) or the lens depth between the front and rear of the lens.

[0050] As photons projected by the laser projector (150) travel across the lenses of the optical assembly (180A-B), the photons encounter optical strips (155A-N). When a specific photon encounters a specific optical strip, the photon is redirected toward the user's eye or passed to the next optical strip. A combination of modulation of the laser projector (150) and modulation of the optical strips can control specific photons or light beams. In the example, a processor controls the optical strips (155A-N) by initiating mechanical, acoustic, or electromagnetic signals. Although illustrated as having two optical assemblies (180A-B), the eyeglass device (100) may include other arrangements such as a single or three optical assemblies, or the optical assemblies (180A-B) may be arranged in different configurations depending on the application or the intended user of the eyeglass device (100).

[0051] As further illustrated in FIG. 2c-d, the eyewear device (100) includes a left temple (110A) adjacent to the left side (170A) of the frame (105) and a right temple (110B) adjacent to the right side (170B) of the frame (105). The temples (110A-B) may be implemented as individual components attached to the frame (105) on each side (170A-B) (as illustrated) or integrated into the frame (105) on each side (170A-B). Alternatively, the temples (110A-B) may be integrated into temples (125A-B) attached to the frame (105).

[0052] In one example, the projection image displays include a first projection image display (180C) and a second projection image display (180D). The eyeglass device (100) includes first and second holes (175A-B) that hold the respective first and second optical assemblies (180A-B). The first optical assembly (180A) includes the first projection image display (180C) (e.g., the display matrix or optical strips and projector (not shown) of FIG. 2c). The second optical assembly (180B) includes the second projection image display (180D), e.g., the display matrix or optical strips (155A-N) and projector (150) of FIG. 2c. The continuous field of view of the continuously displayed images includes an angle of view of about 15° to 30°, more specifically 24°, when measured horizontally, vertically, or diagonally. A continuous displayed image having a continuous field of view represents a combined three-dimensional observable area that can be viewed through stitching with two displayed images presented in the first and second image displays.

[0053] As used herein, “angle of view” describes the angular range of the field of view associated with the displayed images presented on each of the left and right image displays (180C-D) of the optical assembly (180A-B). “Coverage angle” describes the angular range that the lens of the visible light camera (114A-B) or the infrared camera (220) can image. Generally, the image circle generated by the lens is large enough to completely cover the film or sensor and may include some vignetting (e.g., a decrease in brightness or saturation of the image toward the periphery compared to the center of the image). If the lens’s coverage angle does not fill the sensor, the image circle is generally seen with strong vignetting toward the edges, and the effective angle of view is limited by the coverage angle. “Field of view” is intended to describe the field of the observable area that a user of the eyeglass device (100) can see through their eyes through the displayed images presented on the left and right image displays (180C-D) of the optical assembly (180A). The image display (180C) of the optical assembly (180A-B) may have a field of view with a coverage angle of 15° to 30°, for example 24°, and may have a resolution of 480 x 480 pixels.

[0054] FIG. 2e illustrates an eyewear device (100) comprising facial muscle sensors (225) positioned on the directional surface of the right temple (110B) to provide a unique user interface. The muscle sensors (225) may also be provided on the left temple (110A) depending on the desired function of the eyewear device. The muscle sensors (225) have pads / electrodes (227) that reliably contact the user's skin and detect electrical signals generated by muscle movements under the skin near the user's temple. The electrical signals are processed to determine facial expressions and which facial expressions can form inputs for the eyewear (100) illustrated in FIG. 8 and described shortly. In the example, the sensors (225) may be positioned on the frame (105).

[0055] FIG. 3 illustrates a rear perspective view of the eyeglass device of FIG. 2a. The eyeglass device (100) includes an infrared emitter (215), an infrared camera (220), a frame front (330), a frame rear (335), and a circuit board (340). In FIG. 3, it may be shown that the upper left rim of the frame of the eyeglass device (100) includes the frame front (330) and the frame rear (335). An opening for the infrared emitter (215) is formed in the frame rear (335).

[0056] As shown in the circular cross-section (4) of the upper middle portion of the left rim of the frame, a circuit board, which is a flexible PCB (340), is interposed between the front of the frame (330) and the rear of the frame (335). Additionally, the attachment of the left temple (110A) to the left temple (325A) via the left hinge (126A) is illustrated in more detail. In some examples, components of the eye movement tracker (213), including an infrared emitter (215), a flexible PCB (340), or other electrical connectors or contacts, may be located on the left temple (325A) or the left hinge (126A).

[0057] FIG. 4 is a cross-sectional view through a frame and an infrared emitter (215) corresponding to a circular cross-section (4) of the eyeglass device of FIG. 3. The multiple layers of the eyeglass device (100) are illustrated in the cross-sectional view of FIG. 4, and as illustrated, the frame includes a frame front (330) and a frame rear (335). A flexible PCB (340) is placed on the frame front (330) and connected to the frame rear (335). An infrared emitter (215) is placed on the flexible PCB (340) and covered by an infrared emitter cover lens (445). For example, the infrared emitter (215) is reflowed to the rear of the flexible PCB (340). Reflowing attaches an infrared emitter (215) to contact pad(s) formed on the back of a flexible PCB (340) by applying the flexible PCB (340) to a controlled heat that melts solder paste to connect two components. In one example, reflowing is used to surface mount the infrared emitter (215) on the flexible PCB (340) and electrically connect two components. However, it should be understood that through-holes may be used to connect leads from the infrared emitter (215) to the flexible PCB (340), for example, through interconnections.

[0058] The rear frame (335) includes an infrared emitter opening (450) for an infrared emitter cover lens (445). The infrared emitter opening (450) is formed on the rear-oriented side of the rear frame (335) configured to be oriented inward toward the user's eyes. In the example, the flexible PCB (340) can be connected to the front frame (330) via flexible PCB adhesive (460). The infrared emitter cover lens (445) can be connected to the rear frame (335) via infrared emitter cover lens adhesive (455). The connection can also be indirect through intermediate components.

[0059] In the example, the processor (932) utilizes an eye tracker (213) to determine the eye gaze direction (230) of the wearer's eye (234) as shown in FIG. 5, and the eye position (236) of the wearer's eye (234) within the eyebox as shown in FIG. 6. The eye tracker (213) is a scanner that uses infrared light illumination (e.g., near-infrared, short-wavelength infrared, mid-wavelength infrared, long-wavelength infrared, far-infrared) to capture images of reflection fluctuations of infrared light from the eye (234) to determine the gaze direction (230) of the pupil (232) of the eye (234), and also the eye position (236) with respect to the viewing display (180D).

[0060] FIG. 7 illustrates an example of capturing visible light with cameras. Visible light is captured by a left visible light camera (114A) having a left visible light camera field of view (111A) as a left raw image (758A). Visible light is captured by a right visible light camera (114B) having a right visible light camera field of view (111B) as a right raw image (758B). Based on the processing of the left raw image (758A) and the right raw image (758B), a three-dimensional depth map (715) of a three-dimensional scene, hereinafter referred to as an image, is generated by a processor (932).

[0061] FIG. 8 illustrates an electrical schematic of an exemplary muscle sensor (225) illustrated in FIG. 2e that detects electrical signals outside the skin generated by muscles beneath the skin and decodes the signals to determine facial expressions. The determined facial expressions form inputs for the eyeglasses (100). Electrical signals (226) picked up by electrodes (227) in contact with the user's skin first pass through a low-noise instrumentation amplifier (228) (TSZ124). The gain of the amplifier (228) is set during initial calibration using a digital potentiometer (not shown). Subsequently, the signals (226) are transmitted through a filtering (230), such as a band-pass filter, to separate the muscle signals up to 500 Hz. The filtered signals are digitized by an analog-to-digital converter (ADC) (231) and then supplied to a digital signal processor (DSP) (232) that executes an algorithm. The DSP (232) detects the user's muscle activity and determines various facial expressions with sufficient processing power and software. The DSP (232) is coupled to the processor (932) (Fig. 9) to exchange signals.

[0062] Different facial expressions are mapped to different movements or functions. In one example, the DSP (232) can detect muscle signals and detect the user's right raised eyebrow, and is programmed to command the processor (932) to have the glasses cameras (114A-B) take an image in response using the cameras (114A-B). In another example, the DSP (232) can detect muscle signals indicating the user's right eye squinting and, in response, command the processor (932) to adjust the shading of the optical elements (180A-B) by making the shading brighter or darker. In yet another example, the DSP (232) can detect muscle signals indicating the mouth is open and, in response, generate a different movement or function.

[0063] In another example, the user's EKG may be taken using muscle sensors (225). Sensor pads (227) receive the user's muscle signals to determine the user's heart rate. The heart rate signals are processed by an ADC (231) and / or a processor (932) to determine the EKG. The EKG may be initiated automatically or optionally when the presence of a user wearing eyeglasses (100) is determined.

[0064] FIG. 9 depicts a high-level functional block diagram including exemplary electronic components placed in eyeglasses (100 and 200). The exemplary electronic components include a processor (932), memory (934), and a viewing image display (180C and 180D).

[0065] The memory (934) includes instructions for the processor (932) to control the image (715) and instructions for execution by the processor (932) to implement the functions of the glasses (100 / 200). The processor (932) receives power from the battery (950) and executes the instructions stored in the memory (934), or is integrated with the on-chip processor (932) to perform the functions of the glasses (100 / 200) and communicates with external devices via wireless connections.

[0066] The user interface adjustment system (900) includes a wearable device, which is an eyeglass device (100) having an eye movement tracker (213) (e.g., illustrated in FIG. 2b as an infrared emitter (215) and an infrared camera (220)). The user interface adjustment system (900) also includes a mobile device (990) and a server system (998) connected via various networks. The mobile device (990) may be a smartphone, tablet, laptop computer, access point, or any other such device that can be connected to the eyeglass device (100) using both a low-power wireless connection (925) and a high-speed wireless connection (937). The mobile device (990) is connected to the server system (998) and the network (995). The network (995) may include any combination of wired and wireless connections.

[0067] The eyeglass device (100) includes two or more visible light cameras (114A-B) (one associated with the left side (170A) and the other with the right side (170B)). The eyeglass device (100) further includes two viewing image displays (180C-D) of an optical assembly (180A-B) (one associated with the left side (170A) and one associated with the right side (170B)). The eyeglass device (100) also includes an image display driver (942), an image processor (912), a low-power circuit (920), and a high-speed circuit (930). The components of the eyeglass device (100) illustrated in FIG. 9 are located on one or more circuit boards, e.g., a PCB or a flexible PCB, in the temples. Alternatively, or additionally, the depicted components may be located in the temples, frames, hinges, or bridges of the eyeglass device (100). The left and right visible light cameras (114A-B) may include digital camera elements such as CMOS (Complementary Metal-Oxide-Semiconductor) image sensors, charge coupling devices, lenses, or any other individual visual or optical capturing elements that can be used to capture data including images of scenes having unknown objects.

[0068] Eye movement tracking programming implements user interface field of view adjustment commands, which include causing the eyewear device (100) to track the eye movements of the user's eye of the eyewear device (100) through the eye movement tracker (213). Other implemented commands (functions) cause the eyewear device (100) to determine a field of view adjustment for the initial field of view of the initial displayed image based on the user's detected eye movements corresponding to the continuous eye direction. Additional implemented commands generate a series of displayed images based on the field of view adjustment. The series of displayed images is generated as a visible output to the user through the user interface. This visible output appears on the projection image displays (180C-D) of the optical assembly (180A-B), which is driven by an image display driver (942) to present a sequence of displayed images including an initial displayed image with an initial field of view and a series of displayed images with a continuous field of view.

[0069] As illustrated in FIG. 9, the high-speed circuit (930) includes a high-speed processor (932), memory (934), and a high-speed wireless circuit (936). In an example, an image display driver (942) is coupled to the high-speed circuit (930) and operated by the high-speed processor (932) to drive the left and right image displays (180C-D) of the optical assembly (180A-B). The high-speed processor (932) may be any processor capable of managing the operation of any general computing system required for high-speed communication and the eyewear device (100). The high-speed processor (932) includes processing resources required to manage high-speed data transmission in a high-speed wireless connection (937) to a wireless local area network (WLAN) using the high-speed wireless circuit (936). In certain examples, the high-speed processor (932) runs an operating system such as a LINUX operating system or another such operating system of the eyewear device (100), and the operating system is stored in memory (934) for execution. In addition to any other responsibilities, a high-speed processor (932) executing the software architecture of the eyewear device (100) is used to manage data transmission with the high-speed wireless circuit (936). In some examples, the high-speed wireless circuit (936) is also configured to implement IEEE (Institute of Electrical and Electronic Engineers) 802.11 communication standards, referred to herein as Wi-Fi. In other examples, other high-speed communication standards may be implemented by the high-speed wireless circuit (936).

[0070] The processor (932) communicates with the facial muscle sensors (225) to acquire and process electrical signals from the sensors (225) and determine facial expressions. The processor (932) performs actions and / or functions based on the determined facial expressions as discussed in relation to FIG. 8.

[0071] The low-power wireless circuit (924) and high-speed wireless circuit (936) of the eyewear device (100) may include short-range transceivers (Bluetooth™) and wireless wide-area, local, or wide-area network transceivers (e.g., cellular or WiFi). A mobile device (990) including transceivers communicating via a low-power wireless connection (925) and a high-speed wireless connection (937) may be implemented using the details of the architecture of the eyewear device (100), as well as other elements of the network (995).

[0072] The memory (934) comprises any storage device capable of storing various data and applications, including, among other things, color maps, camera data generated by the left and right visible light cameras (114A-B) and the image processor (912), and images generated for display by the image display driver (942) on the perspective image displays (180C-D) of the optical assembly (180A-B). Although the memory (934) is illustrated as being integrated with the high-speed circuit (930), in other examples, the memory (934) may be an independent, standalone element of the eyewear device (100). In these specific examples, electrical routing lines may provide a connection from the image processor (912) or the low-power processor (922) to the memory (934) through a chip including the high-speed processor (932). In other examples, the high-speed processor (932) can manage the addressing of the memory (934) so ​​that the low-power processor (922) boots the high-speed processor (932) whenever a read or write operation involving the memory (934) is required.

[0073] The server system (998) may be one or more computing devices as part of a service or network computing system, including a processor, memory, and a network communication interface for communicating with a mobile device (990) and an eyewear device (100) via a network (995), for example. The eyewear device (100) is connected to a host computer. For example, the eyewear device (100) is paired with the mobile device (990) via a high-speed wireless connection (937) or connected to the server system (998) via the network (995).

[0074] The output components of the eyeglass device (100) include visual components such as the left and right image displays (180C-D) of the optical assembly (180A-B) described in FIG. 2C-D (e.g., displays such as a liquid crystal display (LCD), a plasma display panel (PDP), a light-emitting diode (LED) display, a projector, or a waveguide). The image displays (180C-D) of the optical assembly (180A-B) are driven by an image display driver (942). The output components of the eyeglass device (100) further include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. Input components of the eyewear device (100), mobile device (990), and server system (998) may include alphanumeric input components (e.g., keyboard, touch screen configured to receive alphanumeric input, optical-optical keyboard, or other alphanumeric input components), point-based input components (e.g., mouse, touchpad, trackball, joystick, motion sensor, or other pointing mechanism), haptic input components (e.g., physical button, touch screen providing location and force of touches or touch gestures, or other haptic input components), audio input components (e.g., microphone), etc.

[0075] The eyewear device (100) may optionally include additional peripheral device elements. These peripheral device elements may include biometric authentication sensors, additional sensors, or display elements integrated with the eyewear device (100). For example, the peripheral device elements may include any I / O components including output components, motion components, position components, or any other such elements described herein. The eyewear device (100) may take on different forms and may incorporate different types of frameworks, such as headgear, headsets, or helmets.

[0076] For example, the biometric authentication components (856) of the user interface field of view adjustment unit (900) may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brainwaves), and personal identification (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or brainwave-based identification). Motion components may include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Position components include position sensor components for generating position coordinates (e.g., a Global Positioning System (GPS) receiver component), WiFi or Bluetooth™ transceivers for generating positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be induced), orientation sensor components (e.g., a magnetometer), etc. These positioning system coordinates may also be received from a mobile device (990) via wireless connections (925 and 937) via a low-power wireless circuit (924) or a high-speed wireless circuit (936).

[0077] According to some examples, an “application” or “applications” is a program(s) that execute functions defined in programs. Various programming languages ​​may be used to create one or more applications composed of various methods, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, a third-party application (e.g., an application developed using an ANDROID™ or IOS™ software development kit (SDK) by an entity other than a vendor of a specific platform) may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In this example, the third-party application may call API calls provided by the operating system to facilitate the functions described herein.

[0078] Referring to FIG. 10, a method (1000) for processing facial muscle signals to determine facial expressions and perform functional operations is illustrated as discussed in relation to FIG. 8.

[0079] In block (1002), muscle sensors (225) detect electrical muscle signals through electrodes / pads (227). Facial muscles generate unique electrical signals during different movements. For example, the sensors (225) detect electrical signals such as when raising eyebrows, when squinting eyes, etc. Different electrical signals are generated for different muscle movements that are unique to the movement and processed. The sensors (225) can also determine the presence of a user wearing glasses (100) to initiate or allow certain features to be activated.

[0080] In block (1004), the DSP (232) processes and decodes electrical signals to determine different facial expressions. For example, the DSP (232) can detect and process certain electrical signals from the sensors (225) to determine the user's right raised eyebrow, the user's right eye squinting, and other facial expressions.

[0081] In block (1006), the DSP (232) communicates with the processor (932) to generate functions and / or actions. For example, the processor (932) may command the camera (114A-B) to take an image in response to determining raised eyebrows. In another example, the processor (932) may lighten or darken the shade of the optical assemblies (180A-B) in response to determining squinting eyes. In yet another example, the processor (932) may perform an EKG in response to determining a user wearing eyeglasses (100).

[0082] It will be understood that the terms and expressions used herein have the general meanings given to them in relation to their respective areas of investigation and study, except where a specific meaning is otherwise described herein. Relational terms such as First, Second, etc., may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any such actual relationship or order between such entities or actions. The terms “comprises,” “includes,” “comprising,” or “including,” or any other variations thereof, are intended to include a non-comprehensive inclusion so that a process, method, article, or device comprising a list of elements or steps does not include only these elements or steps, but also includes other elements or steps not explicitly listed or unique to such process, method, article, or device. An element preceding the “singular” does not exclude, without additional restrictions, the presence of additional identical elements in a process, method, article, or device comprising the element.

[0083] Unless otherwise stated, any and all measurements, values, grades, positions, sizes, dimensions, and other specifications set forth herein, including the following claims, are approximate and not precise. Such quantities are intended to have a reasonable range consistent with the functions to which they relate and the art to which they belong. For example, unless otherwise expressly stated, parameter values, etc., may vary by ±10% from the stated quantities.

[0084] Furthermore, in the foregoing detailed description, it can be seen that various features in various examples are grouped together to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than explicitly cited in each claim. Rather, as reflected in the following claim, the subject matter of protection is less than all features of any single disclosed example. Accordingly, the following claims are incorporated into the detailed description, and each claim is a subject matter claimed separately in itself.

[0085] Although examples other than those considered to be the best mode have been described above, it is understood that various modifications may be made thereto, the subject matter disclosed herein may be embodied in various forms and examples and applied to numerous applications, and only some of which are described herein. Any and all modifications and variations falling within the true scope of the concepts are intended by the following claims.

Claims

Claim 1 Eyewear comprising: a frame; an optical member supported by said frame; a temple coupled to said frame; a sensor coupled to said frame or said temple and configured to detect electrical signals indicating facial muscle movements; and a processor configured to process said electrical signals and determine a facial expression. Claim 2 In claim 1, the processor performs an action or function in response to the determined facial expression, eyeglasses. Claim 3 In claim 2, the operation or function is selected from the group of: acquiring an image and adjusting the optical member, eyeglasses. Claim 4 In claim 1, the facial expression is selected from the group of raised eyebrows and squints, eyeglasses. Claim 5 In claim 1, the sensor is located on the temple, eyeglasses. Claim 6 In claim 5, the eyeglasses are configured such that the sensor comes into contact with the user when the user wears the eyeglasses. Claim 7 In claim 1, the processor is configured to map the determined facial expressions to different eyewear movements or functions. Claim 8 A method of using eyeglasses, wherein the eyeglasses comprise: a frame; an optical member supported by the frame; a temple coupled to the frame; a sensor coupled to the frame or the temple and configured to detect electrical signals indicating facial muscle movements; and a processor, wherein the processor processes the electrical signals and determines a facial expression. Claim 9 A method of using eyeglasses according to claim 8, wherein the processor performs an action or function in response to the determined facial expression. Claim 10 In claim 9, the operation or function is selected from the group of acquiring an image and adjusting the optical member, a method of using eyeglasses. Claim 11 In claim 8, the method of using eyeglasses, wherein the facial expression is selected from the group of raised eyebrows and narrowed eyes. Claim 12 In claim 8, the method of using eyeglasses, wherein the sensor is located on the temple. Claim 13 In claim 12, the sensor is a method of using eyeglasses that come into contact with the user. Claim 14 In claim 8, the method of using eyeglasses, wherein the processor maps the determined facial expressions to different eyeglasses movements or functions. Claim 15 A non-transient computer-readable medium for storing program code, wherein the program code, when executed, causes a computing device to perform the steps of: processing electrical signals by a processor of eyewear having a frame, an optical member, a temple, and a sensor coupled to the frame or the temple and configured to detect electrical signals representing facial muscle movements; and determining facial expressions by the processor based on the processed electrical signals. Claim 16 A non-transient computer-readable medium according to claim 15, further comprising code that operates to perform a step of processing to perform an action or function in response to the determined facial expression by the processor. Claim 17 In claim 16, the operation or function is selected from the group of: acquiring an image and adjusting the optical member, a non-transient computer-readable medium. Claim 18 In claim 15, the facial expression is a non-transient computer-readable medium selected from the group of raised eyebrows and narrowed eyes. Claim 19 In claim 15, the sensor is a non-transient computer-readable medium located in the temple. Claim 20 A non-transient computer-readable medium according to claim 15, further comprising code that operates to perform the step of processing the determined facial expressions to map them to different eyewear movements or functions by the processor.

Images