Determining shape of face based on waveform transmitted by head-mounted device
By employing soundwaves that diffract around the face, head-mounted devices can determine facial features and expressions without line-of-sight, addressing the limitations of existing technologies and enhancing applications like videoconferencing and avatar generation.
Patent Information
- Application Number
- PCT/US2023/067858
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-05
AI Technical Summary
Existing head-mounted devices, such as smartglasses, face challenges in determining the shape of a user's face due to the short wavelength of light emitted by cameras and lidar sensors, which requires line-of-sight and is difficult to achieve given the form factor close to the user's face.
The use of soundwaves with a greater wavelength that diffract around the face, allowing a head-mounted device to transmit and process reflections to determine facial features without the need for line-of-sight, using speakers and microphones to generate and capture periodic sonic waveforms.
Enables accurate determination of facial features and expressions, facilitating applications like videoconferencing and avatar generation by creating a facial map based on reflected waveforms, improving usability and accuracy in feature recognition.
Smart Images

Figure US2023067858_05122024_PF_FP_ABST
Abstract
Description
DETERMINING SHAPE OF FACE BASED ON WAVEFORM TRANSMITTED BY HEADMOUNTED DEVICETECHNICAL FIELD
[0001] This description relates to head-mounted devices.BACKGROUND
[0002] Head-mounted devices, such as smartglasses, can present content to a user. The head-mounted devices can also communicate with other computing devices.SUMMARY
[0003] A head-mounted device transmits a waveform via a speaker included in the head-mounted device. The waveform reflects off of a face of a user who is wearing the head-mounted device. A microphone included in the head-mounted device captures the reflected waveform. The head-mounted device, and / or a device in communication with the head-mounted device, determines a feature of the face of the user based on the reflected waveform.
[0004] In one general aspect, a head-mounted device includes a frame configured to mount the head-mounted device onto a head of a user; at least one speaker supported by the frame, the at least one speaker being configured to generate a waveform; at least one microphone supported by the frame, the at least one microphone being configured to process a reflection of the waveform; and at least one processor supported by the frame, the at least one processor being configured to determine a feature of a face of a user wearing the head-mounted device based on the processed reflection of the waveform.
[0005] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS
[0006] FIG. 1 shows a head-mounted device on a user generating a facial mapbased on a reflected waveform.
[0007] FIG. 2 shows a system for generating a facial map based on a reflected waveform.
[0008] FIG. 3 shows a portion of a head-mounted device that includes a speaker and two microphones.
[0009] FIG. 4 shows a head-mounted device transmitting waveforms.
[0010] FIG. 5A shows a graph generated from reflected waveforms based on a moving object shown in FIGs. 5B and 5C.
[0011] FIG. 5B shows the object in a first position with respect to a headmounted device.
[0012] FIG. 5C shows the object in a second position with respect to the headmounted device.
[0013] FIG. 6A shows a model generated based on reflected waveforms when the user has a baseline facial expression.
[0014] FIG. 6B shows a model generated based on reflected waveforms when the user has a smiling facial expression.
[0015] FIG. 6C shows a model generated based on reflected waveforms when the user has a surprised facial expression.
[0016] FIG. 7 shows a pipeline for classifying a facial expression based on the model of FIG. 6A, 6B, or 6C.
[0017] FIG. 8A shows a raw classification matrix.
[0018] FIG. 8B shows a normalized classification matrix.
[0019] FIG. 9 shows a method for determining a shape of a face of a user.
[0020] FIGs. 10A, 10B, and 10C show an example of a head-mounted device.
[0021] FIG. 11 shows the head-mounted device in communication with another device.
[0022] Like reference numbers refer to like elements.DETAILED DESCRIPTION
[0023] Determining features, such as a shape, of a face of a user wearing a headmounted device, can facilitate, for example, videoconferencing. As a specific example,determining features of a face of a user can enable generation of an avatar with features that match current features of the user, such as facial expressions and / or whether the user is speaking.
[0024] Some technologies such as cameras and light detection and ranging (lidar) can be used to detect a shape of a face of a user. However, a technical problem with cameras and lidar sensors is that the wavelength of emitted light is so short that the camera or lidar sensor needs to have a line-of-sight to salient portions of the user’s face to determine the features of the user’s face. The form factor of a head-mounted device, with the head-mounted device being close to the user’s face, makes achieving line of sight between cameras or lidar sensors and the user’s face difficult.
[0025] A technical solution is to transmit soundwaves and process reflections of the soundwaves. The greater wavelength of soundwaves enables the soundwaves to diffract around the user’s face. The diffraction of the soundwaves around the user’s face, and processing of reflections of the soundwaves, enables determination of the features of the user’s face without achieving line of sight between speakers and the user’s face or microphones and the user’s face.
[0026] A head-mounted device can determine a shape (or feature) of a face of a user who is wearing the head-mounted device based on wavefonns, such as sonic waveforms and / or soundwaves. In some examples, the waveforms can be periodic (e.g., from a temporal perspective). In other words, the waveforms can be produced periodically or in pulses spaced apart in time. At least one speaker included in the headmounted device generates and / or transmits a sonic waveform, such as a periodic waveform. In some examples, the waveform includes unique code. The unique code distinguishes the waveform from soundwaves generated by other objects. In some examples, the waveform reflects off of the user’s face. A reflection of the waveform is captured and / or processed by at least one microphone included in the head-mounted device. At least one processor included in the head-mounted device, and / or included in a device in communication with the head-mounted device, determines a shape or feature of the face of the user based on the reflection of the waveform.
[0027] While this description refers to a head-mounted device such as smartglasses (shown in FIGs. 3, 10A, 10B, and 10C), the methods, functions, and / ortechniques described herein can be performed by any head-mounted computing device that generates and / or transmits periodic and / or sonic waveforms and captures and / or processes reflections of the periodic and / or sonic waveforms. In some examples, the methods, functions, and / or techniques described herein can be distributed between multiple computing devices in communication with each other, such as between a local computing device and a remote server in communication with the remote computing device.
[0028] FIG. I shows a head -mounted device 104 on a user 102 generating a facial map 110 based on a reflected waveform. In this example, the user 102 wears the headmounted device 104 on a head of the user 102. The head -mounted device 104 can include smartglasses. The head-mounted device 104 can rest on portions of the head of the user 102 such as on a nose of the user 102 and on ears of the user 102.
[0029] The head-mounted device 104 includes at least one speaker. In some examples, the head-mounted device 104 includes one speaker in each arm (not labeled in FIG. 1). A speaker in each arm, and one or more microphones in each arm, enables the speaker and microphone(s) in each arm to generate and process sonic waveforms that each indicate the shape of one half of the face of the user 102. In some examples, the speakers are in bottom portions of the arms, facing toward shoulders of the user 102. The disposition of the speakers on bottom portions of the arms allows the transmitted sound (which can be considered soundwaves, periodic waveforms, and / or sonic waveforms) to travel toward a face 106 of the user 102, rather than being absorbed by a head of the user 102, which could happen if the speakers were disposed on inside portions of the arms, facing toward the head of the user 102.
[0030] In some examples, at least one speaker generates a waveform, such as a sonic waveform and / or soundwave. In some examples, the waveform includes at least one unique code. The unique code can enable the head-mounted device 104 to correlate times of capturing and / or processing reflections of the periodic waveform with times of transmission of the periodic waveform.
[0031] The waveform can include or can be at least one unique code. The unique code can distinguish the waveform transmitted by the speaker of the head-mounted device 104 from sounds emitted by objects other than the head-mounted device 104. Thehead-mounted device 104 can record times of transmission of the unique code and a time of reception of the unique code within a reflected waveform. The head-mounted device 104 can determine times of reception of the unique codes based, for example, on times at which convolutions of the respective unique sums peak and / or satisfy an impulse response threshold. The head-mounted device 104 can determine round-trip times of each unique code by subtracting the time of transmission for a unique code from the time of reception and / or processing of the unique code.
[0032] In some examples, the waveform has a frequency of between 18 kilohertz and 25 kilohertz. In some examples, a frequency of at least 18 kilohertz ensures that the waveform is inaudible to human ears. In some examples, a frequency of no more than 25 kilohertz ensures that the waveform will diffract around a face 106 of the user 102. Diffraction of the waveform around the face 106 of the user 102 enables the headmounted device 104 to determine features of the face 106 of the user 102 that are not within a line of sight of either the at least one speaker or any microphone included in the head-mounted device 104.
[0033] In some examples, the waveform is periodic and has a periodicity of at least five milliseconds (5ms), such as ten milliseconds (10ms). A human face typically has a height and / or width of no more than thirty centimeters (30cm or 0.3m). With a speed of sound of three hundred forty meters per second (340m / s), a time for sound to travel across the human face is no more than 0.3m / 340m / s = 0.00088s, or nine tenths of a millisecond (0.9ms). Periodicity of at least five milliseconds, or more than five times the time it takes sound to travel across the human face, ensures that periodically-sent waveforms with codes with will not overlap, avoiding ambiguity as to when a processed code was sent.
[0034] The head-mounted device 104 includes at least one microphone. The microphone(s) captures and / or processes the reflection of the periodic waveform. In some examples, the head -mounted device 104 includes at least one microphone on each arm of the head-mounted device 104. In some examples, the head -mounted device 104 includes two microphones on each arm of the head-mounted device 104. Including at least two microphones on each arm enables the head -mounted device 104 to perform beamforming, improving the quality of the received signals. In some examples, the microphones areincluded on bottom portions of the arms. The inclusion of the microphones on bottom portions of the arms enables the microphones to capture the reflections of the periodic waveforms generated and / or transmitted by speakers that are disposed on bottom portions of the arms.
[0035] In some examples, the head-mounted device 104 performs a transformation (108) on the reflection of the waveform. The transformation (108) can include transforming soundwaves and / or sonic waves, such as the waveform, into a facial map 110.
[0036] The transformation (108) can be based on multiple reflections of the waveform. In some examples, the reflections are captured by a single microphone included the head -mounted device 104. In some examples, the reflections are captured by at least one microphone in each of two arms included in the head-mounted device 104. In some examples, the reflections are captured by two microphones in each of two arms included in the head-mounted device 104. The transformation (108) results in the facial map 110. The facial map 110 is a representation of the face 106 of the user 102. In some examples, the facial map 110 includes facial expressions representing facial expressions of the face 106 of the user 102.
[0037] In some examples, the facial map 110 includes a three-dimensional reconstruction of a surface of the face 106 of the user 102. In some examples, the facial map 110 includes a mesh representation of at least a portion of the face 106 of the user 102. In some examples, the facial map 110 includes a triangle mesh that comprises a set of triangles that are connected by common edges or vertices. In some examples, the facial map 110 includes features of portions of the face 106 of the user 102, such as features of lips of the user 102, features of cheeks of the user 102, and / or features of eyes of the user 102. In some examples, the facial map 110 includes a direction that the face 106 of the user 102 is facing and / or looking. In some examples, the head-mounted device 104 can classify an expression and / or emotion of the user 102 based on the facial map 110.
[0038] In some examples, the head-mounted device 104 and / or a computing device in communication with the head-mounted device 104 can generate an avatar of the user 102 based on the facial map 110. In some examples, the avatar can include facial expressions based on the facial map 110. In some examples, the avatar can appear to bespeaking, with the appearance of speaking being based on the facial map 110.
[0039] FIG. 2 shows a system for generating the facial map 110 based on a reflected waveform. The system can include a facial mapper 200. The facial mapper 200 can perform the transformation (108) shown in FIG. 1. The facial mapper 200 can be implemented by a processor, such as the processor 1011 shown in FIG. 10A, included in the head-mounted device 104.
[0040] At transmission time (202), at least one speaker 210 included in the headmounted device 104 (not labeled in FIG. 2) generates and / or transmits waveforms 206. The waveforms 206 can include one or more waveforms. In some examples, the waveforms 206 are sonic waveforms. In some examples, the sonic waveforms are inaudible to human ears. In some examples, the one or more waveforms 206 are periodic waveforms. In some examples, the waveforms 206 have a frequency of between 18 kilohertz and 25 kilohertz, as discussed above with respect to FIG. 1. In some examples, the waveforms 206 include at least one unique code, such an ultrasonic transmission code 212, as discussed above with respect to FIG. 1.
[0041] In some examples, the unique code includes a Barker code or Barker sequence with a finite sequence of digital values with an ideal autocorrelation property for synchronizing the waveforms 206 with reflections 208 (discussed below). In some examples, the code 212 is predesigned and / or predetermined before transmission. In some examples, the waveforms 206 have a period of at least five milliseconds (5ms) as discussed above with respect to FIG. 1.
[0042] In some examples, the waveform 206 is a pulse (also referred to as a chirp). The pulse can have a duration and / or width of less than ten milliseconds. In some examples, the pulse has a duration of between one millisecond and ten milliseconds, such as four milliseconds. The short duration of the pulse enables the at least one speaker 210 to transmit pulses and / or waveforms with a high frequency to capture changes to the face 106 of the user 102. In some examples, the at least one speaker 210 transmits the pulses and / or waveforms with a frequency between twenty and forty Hertz, such as thirty Hertz (30Hz).
[0043] In some examples, the code 212 includes a unique frequency profile embedded onto the waveform 206. In some examples, the waveform 206 is a uniquefrequency profile that is the code 212. The code 212 can, for example, be frequency modulated onto the waveform 206. The code 212 can be stored on the head-mounted device 104. In some examples, the code 212 can be considered a frequency profile. The head-mounted device 104 can transmit the same code 212, and / or modulate the same code 212 onto the waveform 206, repeatedly.
[0044] In some examples, the frequency modulation of the code 212 onto the waveform 206 causes the waveform 206 to include multiple different frequencies. In some examples, the facial mapper 200 transmits the waveform 206 at multiple different times. The multiple different frequencies and multiple different times of transmission cause multiple different responses by the reflections based on the distances traveled and resulting constructive and destructive interference. The different responses based on the variation in the frequency and time profiles of the waveforms 206 enable the facial mapper 200 to determine features of the face 106 at micrometer and / or millimeter level.
[0045] In some examples, the head-mounted device 104 can change the code 212, and / or begin transmitting a different code, based on determining that a different device is transmitting the same code as the head-mounted device 104. The head-mounted device 104 can determine that a different device is transmitting the same code as the headmounted device 104 based, for example, on receiving and / or processing the code at a time that does not correspond to a time that the head -mounted device 104 transmitted the code.
[0046] In some examples, the at least one speaker 210 includes a speaker amplifier. In some examples, the at least one speaker 210 includes a speaker amplifier and a digital -to-analog converter (DAC). The at least one speaker 210 can convert the digital code to an analog signal that is modulated onto the waveform 206, 206A, 206B.
[0047] In some examples, the head-mounted device 104 includes multiple speakers 210 that generate and / or transmit multiple waveforms 206A, 206B. The waveforms 206A, 206B represent the waveforms 206. In some examples, multiple speakers 210 multiplex the waveforms 206A, 206B in the time domain. In some examples, multiple speakers 210 multiplex the waveforms 206A, 206B in the frequency domain.
[0048] At reflection time (204), reflections 208 of the waveforms 206 reflect offof the face (not labeled in FIG. 2) of the user 102. The reflections 208 can be captured and / or processed by a microphone 216. The reflections 208 can be time-delayed back reflections from a surface of the face 106 of the user 102.
[0049] In some examples, the head-mounted device 104 and / or facial mapper 200 includes at least one microphone 216. The at least one microphone 216 can capture multiple reflections 208 A, 208B. The reflections 208 A, 208B can have the properties of the reflections 208.
[0050] In some examples, the at least one microphone 216 includes a microphone amplifier. In some examples, the at least one microphone 216 includes a microphone amplifier and an analog-to-digital converter (ADC). The at least one microphone 216 can demodulate the reflections 208, which can be reflections of the waveform, into the unique code.
[0051] In some examples, the head-mounted device 104 and / or facial mapper 200 includes a compressor 214. The compressor 214 compresses pulses within the reflections of the waveforms. The compressor 214 reduces a number of bits or other symbols that represent the reflections 208 and / or waveforms 206. In some examples, the compressor 214 performs per-channel pulse compression. Per-channel pulse compression can include performing compression on predefined frequency bands within the reflections of the periodic waveforms. In some examples, the compressor 214 associates timings of the reflections of the periodic waveforms captured and / or processed by the at least one microphone 216 with timings of the waveforms generated and / or transmitted by the at least one speaker 210.
[0052] In some examples, the facial mapper 200 includes a beamformer 218. In some examples, the beamformer 218 is an ultrasonic beamformer that performs beamforming on soundwaves and / or sound waveforms. The beamformer 218 receives the compressed waveforms from the compressor 214. The beamformer 218 can determine an angle of reflection of the reflections 208. The beamformer 218 can perform beamforming on the reflections 208 based on the determined angle of reflection. The beamforming performed by the beamformer 218 can strengthen the signal that the facial mapper 200 receives and / or processes based on the reflections 208.
[0053] In some examples, the facial mapper 200 includes a demodulator 220. Insome examples, the demodulator 220 receives the waveform and / or the reflection of the waveform from the compressor 214. In some examples, the demodulator 220 receives the waveform and / or the reflection of the waveform from the beamformer 218. In some examples, the demodulator 220 senses and / or determines an envelope of the reflections 208 of the waveform. Based on the sensing and / or determination of the envelope of the reflections 208, the demodulator 220 processes the signal of the reflections 208 and / or waveforms 206 by, for example, removing noise from the signal.
[0054] In some examples, the facial mapper 200 includes an aligner 222. The aligner 222 can receive the demodulated signal from the demodulator 220. The aligner 222 can align periodic signals that represent a same object (such as the face 106 of the user 102) at different times. The aligner 222 can align, for example, waterfall representations (examples of which are shown in FIG. 5A) of the reflections 208 that occur over the period of the waveform transmitted by the at least one speaker 210. The alignment of the periodic signals and / or representations can show changes of the facial expressions of the user 102.
[0055] In some examples, the facial mapper 200 generates a map 224. The map 224 can be a map around and / or of the face 106 of the user 102. The map 224 can include the facial map 110 shown and described with respect to FIG. 1.
[0056] In some examples, the map 224 is an ultrasonic three-dimensional map around and / or of the face 106 of the user 102. In some examples, the map 224 includes a three-dimensional reconstruction of a surface of the face 106 of the user 102. In some examples, the map 224 includes features of portions of the face 106 of the user 102, such as features of lips of the user 102, features of cheeks of the user 102, and / or features of eyes of the user 102. In some examples, the map 224 includes a direction that the face 106 of the user 102 is facing and / or looking.
[0057] In some examples, the map 224 shows and / or indicates changes to the structure of the face 106 of the user 102 over time. The changes to the structure of the face 106 can be over a predetermined time period, such as over one second, three seconds, five seconds, or ten seconds. The changes to the structure of the face 106 of the user 102 over time can be based on changes to the facial expression of the user 102. The changes to the facial expression of the user 102 can indicate an emotion, such as sad ifthe change is to no longer have a facial expression of smiling, or surprised if the facial expression changes quickly. Determining a facial expression based on multiple, changing images and / or maps of the structure of the face 106 improve the accuracy of determining the emotion of the user 102, but increases computational complexity and computing resources. Determining a facial expression based on a static and / or single image and / or map of the structure of the face 106 is less accurate than determining the facial expression based on multiple images and / or maps, but is less computationally intensive and requires fewer computing resources.
[0058] FIG. 3 shows a portion of a head-mounted device 104 that includes a speaker 210 and two microphones 216A, 216B. In some examples, the speaker 210 is supported by an arm 302. In some examples, the at least one speaker 210 is included on a bottom portion of the arm 302 that will face toward a shoulder of the user 102 when the user 102 is wearing the head-mounted device 104. The arm 302 is included in a frame (not labeled in FIG. 3). The frame is included in the head-mounted device 104. The arm 302 is hingedly attached to a rim 304. The rim 304 is included in the frame. The headmounted device 104 includes a pair of lenses (one lens 306 is shown in FIG. 3). The lens 306 shown in FIG. 3 is supported by the rim 304 shown in FIG. 3.
[0059] The head-mounted device 104 includes at least one microphone 216A, 216B. In some examples, the at least one microphone 216A, 216B is supported by the arm 302. In some examples, the at least one microphone 216A, 216B is included in a lower portion of the arm 302, facing a shoulder of the user 102 when the user 102 is wearing the head-mounted device 104. In some examples, the at least one microphone 216A, 216B is included in an interior portion of the arm 302, facing a head of the user 102 when the user 102 is wearing the head-mounted device 104. In some examples, the at least one microphone 216A, 216B is closer to the rim 304 of the head-mounted device 104 than the at least one speaker 210.
[0060] In some examples, the head-mounted device 104 includes two microphones 216A, 216B on each arm 302, such as a first microphone 216Aand a second microphone 216B. In some examples, both the first microphone 216A and the second microphone 216B are closer to the rim 304 than the at least one speaker 210. The disposition of the at least one speaker 210 and the microphones 216A, 216B on the arm302 creates a linear array that enables the beamformer 218 to perform orthogonal beamforming.
[0061] FIG. 4 shows a head-mounted device 104 transmitting waveforms 402. The user 102 is wearing the head-mounted device 104. The head-mounted device 104 is mounted on a nose and ears of the user 102. The head-mounted device 104 is transmitting the transmitting waveforms 402 via a speaker (not labeled in FIG. 4) included in the head-mounted device 104. The speaker can include features and / or functionalities of the at least one speaker 210 described above. In some examples, the speaker is mounted on a bottom portion of the arms of the head-mounted device 104, facing toward a shoulder of the user 102 when the user 102 is wearing the head-mounted device 104. The disposition of the speaker on the bottom portion of the arm causes the transmitting waveforms 402 to project downward toward the shoulder of the user 102. The transmitting waveforms 402 can have similar properties to the waveforms 206, 206A, 206B.
[0062] The transmitted waveforms 402 and corresponding reflections of the waveforms generated and processed by at least one speaker and at least one microphone on an arm of the head-mounted device 104 can indicate features of half of the face 106 of the user 102. The transmitted waveforms 402 and corresponding reflections of the waveforms generated and processed by at least one speaker and at least one microphone on an arm of the head-mounted device 104 can indicate features of, for example, one eye of the user 102, a left or right half of a nose of the user 102, a left or right half of a cheek of the user 102, a left or right half of both lips of the user 102, and / or a left or right half of a chin of the user 102. The speakers and microphones on both arms of the head-mounted device 104 can transmit waveforms 402 and process reflections of the waveforms to determine features of a left half and right half of the face 106 or the user 102, and / or the entire face 106 of the user 102.
[0063] FIG. 5A shows a graph 500 generated from reflected waveforms based on a moving object 520 shown in FIGs. 5B and 5C. The graph 500 shows object range 504 in centimeters (cm) as a function of time 502 in seconds. The graph 500 is a reconstructed distance-time waterfall feature generated based on reflected waveforms captured and / or processed by microphone 216A, 216B. As shown in FIGs. 5B and 5C, a distance 522 between the object 520 and the head-mounted device 104 changes as the moving object520 moves with respect to the head-mounted device 104.
[0064] FIG. 5B shows the object 520 in a first position with respect to a headmounted device 104. FIG. 5C shows the object 520 in a second position with respect to the head-mounted device 104. The distance 522 in FIG. 5B is greater than the distance 522 in FIG. 5C. FIG. 5A shows an ascent 510 as the object 520 moves to a peak 506 at close range and a descent 512 as the object 520 moves toward a valley 508 at far range.
[0065] FIG. 6A shows a model 602A generated based on reflected waveforms when a user 102A has a baseline facial expression. The user 102A can have similar features to the user 102. The user 102A can wear a head-mounted device 104 that transmits periodic waveforms and processes reflections of the transmitted waveforms, as described above.
[0066] In FIGs. 6A, 6B, and 6C, models 602A, 602B, 602C represent topographical maps of the face of the user 102A. The head-mounted device 104 detects peaks and valleys along acoustical pathways. The acoustical pathways can be paths traversed by the waveform 206, such as from the temple of the user 102A (near the location of the speaker 210) to the chin of the user 102A, from the temple of the user 102Ato the lips of the user 102A, from the temple of the user 102Ato the nose of the user 102A, or from the temple of the user 102Ato the eyes of the user 102A, as nonlimiting examples. In FIGs. 6A, 6B, and 6C, “hot spots,” or areas with multiple changes of color (indicated in FIGs. 6A, 6B, and 6B by vertical lines), indicate muscular activity within the face 106A of the user 102A. The topographical maps represented by the models 602A, 602B, 602C are correlated to reflections along the acoustical paths.
[0067] The head-mounted device 104 generates the model 602A based on the facial expression of the user 102A. The model 602 A includes a depth map for a left half and a right half of a face 106A of the user 102A. Channel 0 (CH0) shows depth values for reflections captured and / or processed by the first microphone 216A on each arm 302. Channel 1 (CHI) shows depth values for reflections captured and / or processed by the second microphone 216B on each arm 302. In the example shown in FIG. 6A, the model 602Ais based on the user 102A having a baseline (non-expressive) facial expression. The head-mounted device 104 can generate a facial map 110 based on the model 602A.
[0068] FIG. 6B shows a model 602B generated based on reflected waveformswhen the user 102Ahas a smiling facial expression. The head-mounted device 104 generates the model 602B based on the facial expression of the user 102 A. The model 602B includes a depth map for the left half and the right half of the face 106A of the user 102A. Channel 0 (CHO) shows depth values for reflections captured and / or processed by the first microphone 216A on each arm 302. Channel 1 (CHI) shows depth values for reflections captured and / or processed by the second microphone 216B on each arm 302. In the example shown in FIG. 6B, the model 602B is based on the user 102A having a smiling facial expression. In this example, the smiling facial expression is shown by the amplified depth map in the model 602B at channel 0 (CHO).
[0069] FIG. 6C shows a model 602C generated based on reflected waveforms when the user 102Ahas a surprised facial expression. The head-mounted device 104 generates the model 602C based on the facial expression of the user 102 A. The model 602C includes a depth map for the left half and the right half of the face 106A of the user 102A Channel 0 (CHO) shows depth values for reflections captured and / or processed by the first microphone 216A on each arm 302. Channel 1 (CHI) shows depth values for reflections captured and / or processed by the second microphone 216B on each arm 302. In the example shown in FIG. 6C, the model 602C is based on the user 102A having a smiling facial expression. In this example, the surprised facial expression is shown by the tail signal, in the center right (indicated by the ovals), of channel 0 (CHO).
[0070] While three different facial expressions (baseline, smiling, and surprised) are shown and described with respect to FIGs. 6A, 6B, and 6C, models can be generated and / or trained based on any number of different facial expressions. In some examples, models can be generated and / or trained based on any combination of facial expressions, such as the seven facial expressions angry, disgust, fear, happy, sad, surprise, and neutral. The head-mounted device 104 can be trained to recognize facial expressions by training a neural network, such as a convolutional neural network, with images of faces of subjects that have been classified into the different facial expressions. In some examples, the head-mounted device 104 can be trained to recognize facial expressions by training a neural network, such as a convolutional neural network, with back propagation.
[0071] FIG. 7 shows a pipeline for classifying a facial expression based on the model of FIG. 6A, 6B, or 6C. The pipeline begins with a model 702. The model 702 is anexample of any of the models 602A, 602B, 602C. The model 702 can include one or more feature vectors, such as ultrasonic feature vectors, based on reflections of periodic waveforms captured by one or microphones. In some examples, the model 702 can include two feature vectors, with a first feature vector being based on reflections of waveforms captured by the first microphone 216A and a second feature vector based on reflections of waveforms captured by the second microphone 216B. The model 702 can be trained based on different reflections from different parts of the face 106A, such as the temples of the user 102A, the eyes of the user 102A, and the chin of the user 102A, as non-limiting examples. A determined topography of the face 106A of the user 102A can be correlated with the reflections. The classification can be trained based on topographic maps of the face 106A of the user 102A. The training can be based on a mesh reconstruction, a regression model, a sentiment classifier, or transcription model based on silent speech (mouthing words but not speaking words) of the user 102A, as non-limiting examples.
[0072] The pipeline can include one or more residual blocks 704, 706, 708. In some examples, the residual blocks 704, 706, 708 are layers of a neural network. The neural network may have been trained based on images of faces of subjects that have been classified into different facial expressions, as discussed above with respect to FIGs. 6A, 6B, and 6C. In some examples, the residual blocks 704, 706, 708 are residual connections that connect the output of an earlier layer (or residual block) to the input of a later layer (or residual block). In some examples, the residual blocks 704, 706, 708 can pass an identity matrix to a subsequent layer. After passing through all of the residual blocks 704, 706, 708, the vectors may be fully connected 720. After the full connection of the vectors, the pipeline can output a classification result 722. The classification result 722 can include a facial expression, such as a smile or surprise.
[0073] FIG. 8A shows a raw classification matrix 800. The raw classification matrix 800 includes vector values that indicate the determined facial feature of the user 102A. In some examples, the classification matrix 800 shows results of a sentiment classifier. In some examples, the sentiment classifier includes a convolutional neural network. The classification result 722 provides either the row value or the column value, and the original classification (such as by human testers) provides the other of the rowvalue or the column value (i.e. either the classification result 722 provides the row value and the original classification provides the column value, or the classification result 722 provides the column value and the original classification provides the row value).
[0074] The raw classification matrix 800 includes a number of rows and columns corresponding to the number of facial expressions for which the pipeline and / or neural network is being trained. In the example of FIG. 8 A, the raw classification matrix 800 includes three rows and columns corresponding to the facial expressions of baseline, smile, and surprise. The integer values in the upper-left cell, middle cell, and lower-right cell show the number of correct classifications for the baseline facial expression, smiling facial expression, and surprise facial expression, respectively. The remaining cells show the number of incorrect classifications.
[0075] FIG. 8B shows a normalized classification matrix 850. The normalized classification matrix 850 is based on the raw classification matrix 800, but includes normalized vector values that indicate the determined facial feature of the user 102A.
[0076] FIG. 9 shows a method 900 for determining a shape of a face 106 of a user 102. The method 900 includes generating a waveform (902). The generated waveform can include a periodic sound waveform, such as the waveforms 206, 402. The generated waveform can be generated and / or transmitted by one or more speakers included in a head-mounted device 104, such as the at least one speaker 210.
[0077] The method 900 includes processing a reflection of the waveform (904). The processed reflection can include a reflection of a periodic sound waveform, such as the reflections 208. The processed reflection can be processed and / or captured by one or more microphones included in the head -mounted device 104, such as the micophones 216A, 216B.
[0078] The method 900 includes determining a shape of a face (906). In some examples, the determined shape is a shape of a face 106 of the user 102. In some examples, the determined shape includes portions of the face 106 of the user 102, such as the eyes, the cheeks, and / or the lips of the user 102. In some examples, the determined shape includes features of the face 106 of the user 102, such as smiling, frowning, or wide eyes. The shape can be determined based on the reflection of the wave form, such as the reflection of the period sound waveform. The reflections can be processed by themicrophones 216A, 216B in each arm 302 to determine round-trip times of unique codes transmitted by the at least one speaker 210 in each arm 302. Based on the round-trip times of the unique codes, the head-mounted device 104 can generate models 602A, 602B, 602C as shown and described with respect to FIG. 6A, 6B, and 6C. Based on the generated models 602A, 602B, 602C, the head-mounted device 104 can determine the shape fo the face (906) and / or determine an expression or emotion of the user 102, 102A.
[0079] In some examples, the determination of the shape of the face (906) includes determining a shape of lips of the user 102 while the user 102 is speaking and / or while audio and / or spoken input is being transcribed. The head-mounted device 104, and / or a computing device in communication with the head-mounted device 104, can transcribe words spoken by the user 102, and the head-mounted device 104, and / or a computing device in communication with the head-mounted device 104, can improve and / or increase the accuracy of the transcription based on a determination of the shape of the lips of the user 102. Tn some examples, the head-mounted device 104, and / or a computing device in communication with the head-mounted device 104 can determine one or more visemes formed by the user 102 while the user 102 is speaking. A viseme is a shape of lips and / or mouth that corresponds to one or multiple phonemes, or sounds within a language that are formed or generated with a same shape of the mouth and / or lips.
[0080] In some examples, the head-mounted device 104 can determine a location of and / or a gesture performed by a hand of the user 102. In some examples, the waveforms 206 reflect off of the hand of the user 102, and the head -mounted device 104 can determine the location of and / or a gesture performed by a hand of the user 102 based on the reflections of the waveforms 206. The determined gestures may be processed as touchless inputs into the head-mounted device 104. In some examples, the head-mounted device 104 can perform actions based on the determined gesture, such as capturing a photograph, presenting an image on a display included in the head-mounted device 104, or turning a computing device other than the head-mounted device 104 on or off.
[0081] In some examples, the head-mounted device 104 can determine whether a user (such as either of the users 102, 102A) is wearing the head -mounted device 104, and control a power level of the head-mounted device 104 based on whether the user iswearing the head-mounted device 104. The head-mounted device 104 can, for example, determine a proximity of the face 106 of the user 102 based on the reflection of the periodic waveforms 206. The head-mounted device 104 can control the power level based on the determined proximity. The head-mounted device 104 can, for example, maintain a low power level if the face 106 is determined not to be within a threshold distance, such as twelve inches or thirty centimeters. The head-mounted device 104 can, for example, maintain and / or initiate a high power level if the face 106 is determined to be within the threshold distance, such as twelve inches or thirty centimeters.
[0082] FIGs. 10A, 10B, and 10C show an example of the head-mounted device 104. As shown in FIGs. 10A, 10B, and 10C, the example head-mounted device 104 includes a frame 1002. The frame 1002 includes a front frame portion defined by rim portions 304A, 304B surrounding respective optical portions in the form of lenses 306A, 306B, with a bridge portion 1006 connecting the rim portions 304A, 304B. Arm portions 302A, 302B included in the frame 1002 are coupled, for example, pivotably or rotatably coupled, to the front frame by hinge portions 1010A, 1010B at the respective rim portion 304 A, 304B. In some examples, the lenses 306A, 306B may be corrective / prescri ption lenses. In some examples, the lenses 306A, 306B may be an optical material including glass and / or plastic portions that do not necessarily incorporate corrective / prescription parameters. Displays 1004A, 1004B may be coupled in a portion of the frame 1002. In the example shown in FIG. 10B, the displays 1004A, 1004B are coupled in the arm portions 302A, 302B and / or rim portions 304A, 304B of the frame 1002. In some examples, the head-mounted device 104 can also include an audio output device 1016 (such as, for example, one or more speakers), an illumination device 1018, at least one processor 1011, at least one memory 1012, an outward-facing image sensor 1014 (or camera), and / or gaze-tracking cameras 1026 A, 1026B that can capture images of eyes of the user 102 to track a gaze of the user 102. The at least one processor 1011 can execute instructions. The at least one memory 1012 can include a non -transitory computer- readable storage medium comprising instructions stored thereon that, when executed by the at least one processor 1011, are configured to cause the head-mounted device 104 to perform any combination of methods, functions, and / or techniques described herein.
[0083] In some examples, the head-mounted device 104 may include a see-through near-eye display. For example, the displays 1004A, 1004B may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e.g., 30-45 degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the world, for example, through the lenses 306A, 306B, next to content (for example, digital images, user interface elements, virtual content, and / or virtual objects) generated by the displays 1004 A, 1004B. In some implementations, waveguide optics may be used to depict content on and / or by the displays 1004A, 1004B via outcoupled light. The images 1020A, 1020B projected by the displays 1004A, 1004B onto the lenses 306A, 306B may be translucent, allowing the user 102 to see the images projected by the displays 1004A, 1004B as well as physical objects beyond the head -mounted device 104.
[0084] In the example shown in FIG. 10C, the head-mounted device 104 includes at least one lens 306 A, 306B supported by the frame 1002. The lenses 306A, 306B can be supported by respective rim portions 304A, 304B that are included in the frame 1002. In some examples, the lenses 306A, 306B, in conjunction with the displays 1004A, 1004B, present, to the user 102, images generated by the processor 1011 A first rim 304A can be coupled to a second rim 304B via the bridge 1006. In the example shown in FIG. 10C, the head-mounted device 104 includes at least one speaker 210, 1008 supported by the frame 1002. In some examples, the head-mounted device 104 includes at least a first speaker 1008 supported by the first arm and / or arm portion 302Athat is included in the frame 1002 and at least a second speaker 210 supported by the second arm and / or arm portion 302B that is included in the frame 1002. In the example shown in FIG. 10C, the head -mounted device 104 includes at least one microphone 216A, 216B, 1009A, 1009B supported by the frame 1002. In some examples, the head-mounted device 104 includes at least a first microphone 1009A supported by the first arm and / or arm portion 302Athat is included in the frame 1002 and at least a second microphone 216A supported by the second arm and / or arm portion 302B that is included in the frame 1002. In some examples, the head-mounted device 104 includes a third microphone 1009B supported by the first arm and / or armportion 302Athat is included in the frame 1002 and a fourth microphone 216B supported by the second arm and / or arm portion 302B that is included in the frame 1002. The speakers 210, 1008 can have similar features and / or functionalities to the speaker 210 described above with respect to FIG. 3. The microphones 216A, 216B, 1009A, 1009B can have similar features and / or functionalities to the microphones 216A, 216B described above with respect to FIG. 3.
[0085] FIG. 11 shows the head-mounted device 104 in communication with another device 1100. The device 1100 can be considered a companion device. The device 1100 can include, for example, a smartphone, a tablet, a smartwatch, a laptop computer, a desktop computer, or a remote server, as non-limiting examples.
[0086] Methods, functions, and / or techniques can be distributed between the head-mounted device 104 and the device 1100. In some examples, the head-mounted device 104 generates and / or transmits the waveform(s), captures and / or processes the reflection(s) of the waveform(s), and performs the transformation (108) to generate the facial map 110.
[0087] In some examples, the head-mounted device 104 generates and / or transmits the waveform(s), captures and / or processes the reflection(s) of the waveform(s), and transmits data (such as timing data with respect to the transmission and capture of the waveform(s)) to the device 1100. In some examples, the device 1100 performs the transformation (108) to generate the facial map 110 based on the data received from the head-mounted device 104, and transmits the generated facial map 110 to the head-mounted device 104.
[0088] The device 1100 can include at least one processor 1102. The processor 1102 can be an example of the processor 1011. The at least one processor 1102 can execute instructions, such as instructions stored in at least one memory device 1104, to cause the head-mounted device 104 to perform any combination of methods, functions, and / or techniques described herein.
[0089] The device 1100 can include at least one memory device 1104. The at least one memory device 1104 can include a non-transitory computer-readable storage medium. The at least one memory device 1104 can store data and instructions thereon that, when executed by at least one processor, such as the processor 1102, are configuredto cause the device 1100 to perform any combination of methods, functions, and / or techniques described herein. Accordingly, in any of the implementations described herein (even if not explicitly noted in connection with a particular implementation), software (e.g., processing modules, stored instructions) and / or hardware (e.g., processor, memory devices, etc.) associated with, or included in, the device 1100 can be configured to perform, alone, or in combination with the head-mounted device 104, any combination of methods, functions, and / or techniques described herein.
[0090] The device 1100 may include at least one input / output node 1106. The at least one input / output node 1106 may receive and / or send data, such as from and / or to, the head-mounted device 104, and / or may receive input and provide output from and to a user. The input and output functions may be combined into a single node, or may be divided into separate input and output nodes. The input / output node 1106 can include, for example, a microphone, a camera, an IMU, a display, a speaker, a microphone, one or more buttons, and / or one or more wired or wireless interfaces for communicating with other computing devices.
[0091] Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
[0092] Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may beimplemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
[0093] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.
[0094] To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
[0095] Implementations may be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such back-end, middleware, or front-end components. Components may be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
[0096] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the embodiments of the invention.
Claims
WHAT IS CLAIMED IS:
1. A head-mounted device comprising: a frame configured to mount the head-mounted device onto a head of a user; at least one speaker supported by the frame, the at least one speaker being configured to generate a waveform; at least one microphone supported by the frame, the at least one microphone being configured to process a reflection of the waveform; and at least one processor supported by the frame, the at least one processor being configured to determine a feature of a face of a user wearing the head-mounted device based on the processed reflection of the waveform.
2. The head-mounted device of claim 1, wherein the waveform includes a unique code.
3. The head-mounted device of either of claims 1 or 2, further comprising at least one lens supported by the frame, the at least one lens being configured to present, to the user, at least one image generated by the at least one processor.
4. The head -mounted device of either of claim 3, wherein the frame includes: at least one rim, the at least one rim supporting the at least one lens; and at least one arm hingedly coupled to the at least one rim, the at least one arm supporting the at least one speaker and the at least one microphone.
5. The head -mounted device of claim 4, wherein: the at least one lens is a first lens and the at least one image is a first image; the at least one rim is a first rim supporting the first lens; the at least one arm is a first arm hingedly coupled to the first rim; the at least one speaker is a first speaker supported by the first arm; the frame further comprises a second rim coupled to the first rim and a second arm hingedly coupled to the second rim; andthe head-mounted device further comprises a second lens supported by the second rim and a second speaker supported by the first arm, the second lens being configured to present a second image to the user and the second speaker being configured to generate the waveform.
6. The head-mounted device of claim 5, wherein: the at least one microphone is a first microphone; the reflection of the waveform is a first reflection of the waveform; the head-mounted device further comprises a second microphone configured to process a second reflection of the waveform; and the at least one processor is configured to determine a shape of the face of the user wearing the head-mounted device based on the processed first reflection of the waveform and the processed second reflection of the waveform.
7. The head -mounted device of claim 6, wherein: the first microphone is supported by the first arm; and the second microphone is supported by the second arm.
8. The head-mounted device of either of claims 6 or 7, further comprising: a third microphone supported by the frame, the third microphone being configured to process a third reflection of the waveform; and a fourth microphone supported by the frame, the fourth microphone being configured to process a fourth reflection of the waveform, the at least one processor being configured to determine the shape of the face of the user wearing the head-mounted device based on the first reflection of the waveform, the second reflection of the waveform, the third reflection of the waveform, and the fourth reflection of the waveform.
9. The head-mounted device of claim 8, wherein: the third microphone is supported by the first arm; and the fourth microphone is supported by the second arm.
10. The head-mounted device of any of the preceding claims, wherein the waveform has a frequency between 18 kilohertz and 25 kilohertz.
11. The head-mounted device of any of the preceding claims, wherein the waveform is periodic and has a periodicity of at least five milliseconds.
12. The head-mounted device of any of the preceding claims, wherein the at least one processor is configured to determine at least one viseme based on the shape of the face of the user.
13. The head-mounted device of any of the preceding claims, wherein the at least one processor is configured to determine a gesture of a hand of the user based on the processed reflection of the waveform.
14. The head-mounted device of any of the preceding claims, wherein the at least one processor is configured to: determine a proximity of the face of the user based on the reflection of the waveform; and control a power level of the head-mounted device based on the determined proximity.
15. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a head-mounted device to: generate a sound waveform; process a reflection of the sound waveform; and based on the reflection of the sound waveform, determine a feature of a face of a user wearing the head-mounted device.
16. The non-transitory computer-readable storage medium of claim 15, wherein the waveform is periodic and has a periodicity of at least five milliseconds.
17. The non-transitory computer-readable storage medium of either of claims 15-16, wherein the at least one processor is configured to determine at least one viseme based on a shape of the face of the user.
18. The non-transitory computer-readable storage medium of any of claims 15-17, wherein the sound waveform has a frequency between 18 kilohertz and 25 kilohertz.
19. A method performed by a head-mounted device, the method comprising: generating a sound waveform; processing a reflection of the sound waveform; and based on the reflection of the sound waveform, determining a feature of a face of a user wearing the head-mounted device.
20. A method performed by a head-mounted device, the method comprising: generating a sound waveform; processing a reflection of the sound waveform; sending, to a companion device, data based on the reflection of the sound waveform; and receiving, from the companion device, a feature of a face of a user wearing the head-mounted device.
21. The method of either of claims 19 or 20, wherein the sound waveform is periodic and has a periodicity of at least five milliseconds.
22. The method of any of claims 19-21, further comprising determining at least one viseme based on a shape of the face of the user.
23. The method of any of claims 19-22, wherein the sound waveform has a frequency between 18 kilohertz and 25 kilohertz.
24. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to perform the method of any of claims 20-23.
Citation Information
Patent Citations
Head-mounted speaker array
US11451922B1
Wearable facial movement tracking devices
US20230077010A1
Capturing audio impulse responses of a person with a smartphone
US9591427B1
Beamforming systems and methods for detecting heart beats
WO2022067077A1
Cited By
Programs, information processing devices, methods, and systems
JP7868818B1
