Gaze-Based Audio Beamforming

By employing stored beam patterns based on user gaze direction, AR devices like smart glasses achieve efficient audio beamforming that enhances audio clarity while conserving power, addressing the resource demands of eye-tracking beamforming.

JP2024542246AActive Publication Date: 2024-11-13GOOGLE LLC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024530011
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-22
Filing Date
2022-11-17
Publication Date
2024-11-13
Estimated Expiration
2042-11-17

AI Technical Summary

Technical Problem

Existing augmented reality (AR) devices face challenges in efficiently processing audio while minimizing power consumption and processing resources, particularly in head-mounted computing devices like smart glasses, where eye-tracking beamforming demands are high.

Method used

The implementation of stored beam patterns based on user gaze direction, enabling selective audio beamforming that reduces complexity and enhances processing/power efficiency by using eye-tracking to determine and apply beam patterns from a database.

Benefits of technology

This approach allows for responsive audio beamforming that enhances user experience by amplifying sounds in the line of sight without significantly impacting battery life or processing resources, improving audio clarity and reducing power consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542246000001_ABST
    Figure 2024542246000001_ABST
Patent Text Reader

Abstract

Beamforming can improve the quality and / or strength of sound in a particular direction, which can aid in a user's hearing, especially in a particular environment. The augmented reality glasses can track the user's eye(s), which can trigger and direct beamforming toward a particular object of interest, such as a person speaking. Beamforming can be made more efficient by storing the information required for beamforming in a database and recalling the information at run time.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. Patent Application No. 17 / 456,007, filed November 22, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to augmented reality, and more particularly, to an augmented reality device configured to process audio based on eye tracking. [Background technology]

[0003] Head-mounted computing devices (e.g., smart glasses) can be configured with various sensors that enable augmented reality (AR), in which virtual elements are presented along with real elements of the environment. The virtual elements can be presented on a head-up display, so that they appear as if they are in the real world. The head-up display can be implemented with a device similar to glasses (i.e., AR glasses).

[0004] The AR glasses may be configured with an eye-tracking sensor(s) that determines the user's gaze direction and / or gaze point as they change over time. The AR glasses may also be configured with multiple microphones operating as a microphone array with a sensitivity pattern having a beam, so that sounds from the beam direction are received with the highest sensitivity of the microphone array. The sound from the microphone array may be processed such that the beams can be steered (i.e., beamformed) in different directions. Summary of the Invention

[0005] In at least one aspect, the present disclosure generally describes a method. The method includes receiving audio channels from a plurality of microphones configured to operate as a microphone array of an augmented reality (AR) device. The method further includes tracking the eyes of a user of the AR device to identify the user's gaze direction. The method further includes selecting a beam pattern for the microphone array from a set of stored beam patterns based on the user's gaze direction. The method further includes generating a beamformed audio signal based on the selected beam pattern, and transmitting the beamformed audio signal to a speaker (e.g., a built-in speaker, a paired speaker) of the AR device to play the beamformed audio signal to the user.

[0006] In another aspect, the present disclosure generally describes an AR device such as smart glasses. The AR device may include components, specifically a microphone array, an eye tracker, a speaker, and a processor configured to implement the proposed method. For example, the proposed smart glasses include a microphone array having a microphone configured to generate an audio channel based on sounds from the environment. The smart glasses further include an eye tracker configured to identify a user's gaze direction. The smart glasses further include a speaker and a processor. The processor is configured with software to receive the audio channel from the microphone array and the gaze direction from the eye tracker. The processor is further configured to obtain weights for the audio channel from a lookup table (or another type of data memory, specifically another type of database, array, or table) based on the gaze direction. The processor is further configured to apply weights to the channels and sum the channels to generate a beamformed audio signal that amplifies sounds in the environment from the gaze direction. The processor is further configured to transmit the beamformed audio signal to a speaker for playback to the user.

[0007] The foregoing summary, as well as other illustrative objects and / or advantages of the present disclosure, and the manner in which they are accomplished, are further described in the following detailed description and accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] 13 is a sensitivity plot of a microphone array after beamforming, according to a possible embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates a view through AR glasses including a measured viewpoint, according to a possible embodiment of the present disclosure. [Diagram 3] 3A and 3B are diagrams illustrating beamforming based on gaze direction according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a perspective view of AR glasses according to a possible embodiment of the present disclosure. [Diagram 5] 1 illustrates a beamforming process according to a possible embodiment of the present disclosure. [Figure 6] 6A and 6B are diagrams illustrating possible processing of beamforming audio according to an embodiment of the present disclosure. [Figure 7] 1 is a flowchart of a method for eye-tracking audio beamforming according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates retrieving a beam pattern from a database based on a line of sight according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart of a method for selecting a beam pattern from a database based on a line of sight, according to an embodiment of the present disclosure. [Figure 10] FIG. 2 illustrates a flowchart of a method for generating a beam pattern according to an embodiment of the present disclosure. [Figure 11] 1 is a flowchart for detecting line of sight for beamforming according to an embodiment of the present disclosure. [Figure 12] FIG. 1 illustrates beamforming zoom-in and zoom-out according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The components in the drawings are not necessarily drawn to scale relative to each other. Like reference numerals indicate corresponding parts throughout the several views.

[0010] The present disclosure describes audio beamforming (i.e., beamforming) of a microphone array in an augmented reality (AR) device, such as smart glasses (e.g., AR glasses), based at least in part on the position(s) of a user's eye(s) (i.e., eye-tracking beamforming). The technical problem of eye-tracking beamforming relates to the demands placed on the power / processing resources of the AR glasses. To be effective, eye-tracking and beamforming need to be responsive (i.e., avoiding noticeable delays) without consuming too much power (i.e., extending battery life). The present disclosure provides a system and method of eye-tracking beamforming based on techniques that reduce complexity and increase processing / power efficiency. The disclosed techniques may have the technical effect of automatically enhancing signals in the direction of the user's gaze without significantly impacting the battery life or processing resources of the AR glasses.

[0011] The power / processing efficiency of the disclosed eye-tracking beamforming approach may result from several different aspects. First, the disclosed eye-tracking beamforming may rely at least in part on stored beam patterns that may be acquired and applied based on the user's line of sight. Second, the eye-tracking beamforming may be configured to be enabled / disabled in certain situations so that it is not always in operation.

[0012] The technical effects of the disclosed eye-tracking beamforming techniques may enable new audio applications. For example, the present disclosure further describes implementations in which audio beamforming can zoom in or out in the gaze direction to enhance the user's audio experience.

[0013] Beamforming (i.e., beam steering) is a signal processing technique in which multiple audio channels may be processed (e.g., filtered, delayed, phase shifted) to generate a beamformed audio signal in which audio from various directions may be enhanced (i.e., amplified) or weakened (i.e., attenuated). For example, the first and second microphones may be spatially separated by a certain distance along the array direction. The spatial separation distance and the direction of the sound (relative to the array direction) may result in an interaural delay between the first audio stream at the first microphone and the second audio stream at the second microphone. Beamforming may include further delaying one of the audio streams by a beamforming delay, such that after beamforming, the first and second audio streams are phase shifted by the interaural delay and the beamforming delay. The phase-shifted audio streams are then combined (e.g., summed) to produce the beamformed audio. By adjusting the beamforming delay relative to the interaural delay, sounds from a particular direction can be adjusted (e.g., canceled, attenuated, enhanced) by the summation process. For example, a pure sine wave received by the first and second microphones can be completely canceled for a particular direction if, after the interaural delay and beamforming delay, the sine wave version at the combiner is 180 degrees out of phase. Alternatively, if, after the interaural delay and beamforming delay, the sine wave version at the combiner is in phase (i.e., phase difference is 0 degrees), the sine wave version at the combiner can be enhanced.

[0014] Multiple audio channels may be captured (i.e., collected) by an array of microphones (i.e., a microphone array). Each microphone in a microphone array may be of the same type or different types. For example, all microphones in a microphone array may be omnidirectional. The microphones may be spaced (e.g., equally spaced) in one, two, or three dimensions. A one-dimensional microphone array may be capable of beam steering in one dimension, while a two-dimensional microphone array may be capable of beam steering in either or both of the two dimensions. The number and spacing of microphones in a microphone array may correspond to the beamwidth (i.e., directivity, focus, angular range) of the beam.

[0015] FIG. 1 shows a polar plot of the sensitivity of a microphone array 101 after beamforming. The pattern of sensitivity is known as the "beam pattern" of the microphone array. The microphone array beam pattern has a beam 110 with maximum sensitivity in a beam direction 120, and a beam width 130 with a particular sensitivity (e.g., -3 decibels (dB)) on either side of the beam direction 120. The beamformed sound generated from the beam pattern reinforces (e.g., amplifies) sounds from sound sources in the beam direction 120 and suppresses (e.g., attenuates) sounds from sound sources not in the beam direction. In other words, a first sound 104 from a direction that is aligned with the beam direction 120 sounds louder to a listener than a second sound 105 from a direction that is not aligned with the beam direction 120.

[0016] The spatially selective enhancement / suppression caused by beamforming may help users distinguish between sounds (e.g., in noisy environments). Additionally (or alternatively), beamforming may improve the accuracy of other computer-assisted speech applications (e.g., speech recognition, speech-to-text (VTT), language translation, etc.). Furthermore, beamforming may improve privacy because other sounds received from directions other than the speech direction (e.g., nearby conversations) may be amplified much less than the speech sounds. Controlling beamforming based on the listener's intent, which may be determined by tracking the listener's eyes, may improve the versatility of these applications.

[0017] Eye-tracking beamforming involves adjusting the processing (e.g., filtering, delaying, phase shifting) of multiple audio channels from a microphone array according to a user's eye(s) to generate a beam with a beam direction that approximately matches (e.g., exactly matches) the user's line of sight. The user's line of sight may include the direction in which the user is looking (i.e., gaze direction). Determining the gaze direction (e.g., gaze(θ), gaze(φ,θ)) may include determining a viewpoint (e.g., gaze(x,y)) within the field of view at which the user is looking.

[0018] Gaze may be determined by tracking the user's eye(s). One possible method of eye tracking involves using a camera to measure eye metrics and determine eye position. In one possible implementation, pupil position relative to a light pattern (near infrared light) projected onto the eye may be measured by analyzing high-resolution images of the eye and the pattern. The eye position may then be applied to a machine learning model to determine the gaze. Variations of this method that do not use projected patterns are also possible. For example, there are standard glint-based tracking or convolutional neural net methods that can convert a two-dimensional (2D) infrared image captured by a camera pointed at the eye (or an image of the eye reflected from a mirror) into coordinates (x, y) within the field of view of the AR glasses, as shown in FIG. 2.

[0019] 2 shows a field of view 205 through the AR glasses 201, including a measured viewpoint 210. The viewpoint may correspond to a gaze direction relative to the AR glasses. A gaze direction remaining stationary for a certain amount of time may indicate a user's interest in the corresponding viewing area. This interest may trigger beamforming of a gaze direction corresponding to the viewpoint. For example, the gaze direction may be based on a projection, with respect to the coordinate system of the AR glasses, intersecting the viewpoint on the viewing surface.

[0020] 3A and 3B illustrate beamforming based on gaze direction. As shown by the dotted arrows, the gaze direction may be determined based on the user's eye position. FIG. 3A illustrates a first beam pattern 321 along a first gaze direction 331 of a user 301, and FIG. 3B illustrates a second beam pattern 322 along a second gaze direction 332 of a user 302. The user's gaze may change over time. Thus, the first beam pattern 321 may be used at a first time, and the second beam pattern 322 may be used at a second time. The user's gaze may or may not be associated with the user's head position. In some implementations, the user's head position and gaze direction may be combined when determining the beam direction.

[0021] FIG. 4 is a perspective view of AR glasses according to a possible embodiment of the present disclosure. The AR glasses 400 are configured to be worn on the head and face of a user. The AR glasses 400 include a right ear temple 401 and a left ear temple 402 that are supported by the user's ears. The AR glasses further include a bridge portion 403 that is supported by the user's nose so that a left lens 404 and a right lens 405 can be placed in front of the user's left eye and the user's right eye, respectively. The portions of the AR glasses may be collectively referred to as a frame of the AR glasses. The frame of the AR glasses may contain electronics that enable functionality. For example, the frame may include a battery, a processor, a memory (e.g., a non-transitory computer-readable medium), and electronics that support sensors (e.g., a camera, a depth sensor, etc.), and an interface device (e.g., a speaker, a display, a network adapter, etc.).

[0022] The AR glasses 400 may include a FOV camera 410 (e.g., an RGB camera) oriented with a camera field of view that overlaps with the natural field of view of the user's eyes when the glasses are worn. In a possible implementation, the AR glasses may further include a depth sensor 411 (e.g., a LIDAR camera, a structured light camera, a time-of-flight camera, a depth camera) oriented with a depth sensor field of view that overlaps with the natural field of view of the user's eyes when the glasses are worn. Using data from the depth sensor 411 and / or the FOV camera 410, the depth within the field of view (i.e., the area of ​​interest) of the user (i.e., the wearer) may be measured. In a possible implementation, the camera field of view and the depth sensor field of view may be calibrated so that the depth (i.e., range) of an object in an image from the FOV camera 410 can be determined, and the depth is measured between the object and the AR glasses.

[0023] The AR glasses 400 may further include a display 415. The display may present AR data (e.g., images, graphics, text, icons, etc.) on a portion of the lens(es) of the AR glasses such that the user can view the AR data when looking through the lenses of the AR glasses. In this manner, the AR data may be overlaid with the user's view of the environment.

[0024] The AR glasses 400 may further include an eye tracking sensor. The eye tracking sensor may include a right eye camera 420 and a left eye camera 421. The right eye camera 420 and the left eye camera 421 may be disposed in the lens portion of the frame such that the right FOV 422 of the right eye camera includes the right eye of the user and the left FOV 423 of the left eye camera includes the left eye of the user when the AR glasses are worn. The viewpoint (x, y) may be measured at the frequency of the video feed of the camera (e.g., the right eye camera 420, the left eye camera 421). For example, the gaze coordinate (x, y) may be measured at the frame rate of the camera (e.g., 15 frames / second) or less.

[0025] The AR glasses 400 may further include multiple microphones (i.e., two or more microphones). The multiple microphones may be spaced apart on the frame of the AR glasses. As shown in FIG. 4, the multiple microphones may include a first microphone 431, a second microphone 432, a third microphone 433, a fourth microphone 434, and a fifth microphone 435. The multiple microphones may be configured to operate together as a microphone array that directs beams in a particular direction with respect to the coordinate system 430 of the AR glasses 400. Alternatively, the microphones may be configured to operate in groups (i.e., subarrays), with each group configured to operate as a microphone array. In one example, the third microphone 433 and the fourth microphone 434 may be configured to operate as a (horizontal) microphone array along the X direction of the coordinate system 430 of the AR glasses 400. In other words, the third microphone 433 and the fourth microphone 434 may be used for horizontal beamforming. Furthermore, the third microphone 433 and the fifth microphone 435 may be configured to operate as a (vertical) microphone array along the Y direction of the coordinate system 430 of the AR glasses 400. In other words, the third microphone 433 and the fifth microphone 435 may be used for vertical beamforming. Furthermore, the first microphone 431 and the second microphone 432 may be configured to operate as a (horizontal) microphone array along the Z direction of the coordinate system 430 of the AR glasses 400. The number of microphones used for each subarray may be two or more. Increasing the number of microphones in a subarray can narrow the beam width of the beam pattern resulting from the subarray. Since the beamforming process of the subarray can be parallelized, two or more subarrays may share one or more microphones.

[0026] The AR glasses 400 may further include a left speaker 441 and a right speaker 442 configured to transmit audio (e.g., beamforming audio) to the user. Additionally or alternatively, transmitting audio to the user may include transmitting audio to a listening device (e.g., a hearing aid, earphones, etc.) via a wireless communication link 445. For example, the AR glasses may transmit audio (e.g., beamforming audio) to a left wireless earphone 446 and a right earphone 447. In the case of beamforming audio tracking the user's viewpoint (x, y), sounds in the audio from a field of view that includes the viewpoint may be amplified, while sounds from other fields of view may not be amplified or may be attenuated. In other words, the speakers of the AR device may include speakers (e.g., earphones) communicatively connected (i.e., paired) with the AR glasses 400, or speakers integrated (i.e., built-in) in the AR glasses 400.

[0027]

number

[0028]

number

[0029] Beamforming may be parallelized by orientation, such that arrays arranged horizontally with respect to the coordinate system (i.e., horizontal arrays) have a first set of weights, while arrays arranged vertically with respect to the coordinate system (i.e., vertical arrays) have a second set of weights. The audio from the arrays in each direction may be processed separately to generate horizontal and vertical beamforming signals. The horizontal and vertical beamforming signals may be averaged to form beamforming audio for a direction that includes a horizontal component (e.g., x) and a vertical component (e.g., y). This parallel processing approach may have the advantage of simplicity, but other approaches may be possible. For example, it may be possible to determine beamforming weights in both the horizontal and vertical directions so that an additional averaging step is not required. Furthermore, adding a third dimension (e.g., z) to the aforementioned steps may enable three-dimensional (3D) beamforming.

[0030]

number

[0031]

number

[0032] 7 is a flow chart of a method of eye-tracking audio beamforming according to an embodiment of the present disclosure. Method 700 may be implemented as a computer program product tangibly embodied on a non-transitory computer-readable medium. In other words, the steps of method 700 may be included as part of a computer program (i.e., a module, software, application, code) that may be implemented in a programming language or machine language. When executed, the computer program may configure at least one processor to perform the method of eye-tracking audio beamforming.

[0033] The method 700 of eye-tracking audio beamforming includes capturing 705 audio from a plurality of microphones (i.e., a microphone array). In a possible implementation, each microphone in the microphone array has an omnidirectional sensitivity pattern. In another possible implementation, one or more of the microphones in the microphone array have a directional sensitivity pattern.

[0034] The microphones may be integrated with the AR glasses. In a possible implementation, the AR glasses may be configured in a beamforming mode (i.e., with beamforming) or a normal mode (i.e., without beamforming). In the beamforming mode, audio from the microphone array may be processed to steer the sensitivity of the microphone array in a direction corresponding to the user's gaze. The selection of the mode may depend on various factors. For example, whether to perform beamforming may be determined based on processing and power resources available in the AR glasses. In particular, when the AR glasses are in a low power mode (e.g., power level is less than 25%), eye tracking may be avoided. Thus, the method 700 may optionally include determining (710) whether the device is in a beamforming mode. When the AR device is not in a beamforming mode, audio from one or more of the microphones may be provided to the user (745). However, when the AR glasses are in a beamforming mode, steps may be performed to perform eye tracking audio beamforming. In some implementations, the AR glasses are configured to automatically beamform audio when the user's line of sight meets the criterion(s). In these implementations, determining whether the device is in beamforming mode (710) may be omitted.

[0035] The method 700 includes tracking 715 the user's eye(s). Results of the eye tracking may be used to detect gaze 720. If gaze is detected, audio may be beamformed and provided to the user as beamformed audio, otherwise audio may be provided to the user without beamforming 745. Details regarding determining when to beamform and when not to beamform based on gaze are discussed further below (see, e.g., FIG. 11).

[0036] After detecting the gaze, the gaze direction may be determined (725). As previously described (e.g., FIG. 2), determining the user's gaze may include determining a viewpoint from a measured position of one or both eyes. The position may be measured from an image captured of the eye(s). In some implementations, the method may further include collecting additional information to facilitate confirmation and / or improvement (730) (i.e., adjustment) of the gaze direction. For example, an image 727 of the user's field of view at the determined gaze direction may be captured and analyzed. The analysis may include searching for known sound sources within the image 727. For example, if there is a person speaking in the gaze direction, it may be confirmed that the gaze direction is toward the person speaking. If the person speaking is in a direction close to the determined gaze direction (e.g., within ±10 degrees) but does not exactly match the gaze direction, an adjustment may be made to align the gaze direction with the person speaking.

[0037] After the gaze direction is determined, the method 700 includes selecting a beam pattern as a function of the gaze direction (735). The beam pattern may be selected from a plurality of beam patterns stored in a memory. The memory may be local to the AR glasses or may be available over a network communicatively connected to the AR glasses. For example, the beam patterns may be stored in a lookup table or database 737 that can be queried using (at least) the gaze direction.

[0038] 8 illustrates selecting and retrieving a beam pattern from a database based on a line of sight, according to an embodiment of the present disclosure. A database or lookup table may be queried using a line of sight direction to return a set of weights (w1, w2, ... wn) to provide a beam in the line of sight direction (or in a direction close to the line of sight direction). As shown, the database includes multiple beam patterns (BP1, BP2, BP3, BPn) in multiple directions (d1, d2, d3, ... dn), each having a beam width (bw1, bw2, bw3, ... bwn).

[0039] The returned set of weights (w1, w2, ... wn) may have each weight corresponding to a microphone. The beam width of the beam pattern may correspond to the number of microphones in the array. Thus, the stored beam patterns may include different numbers of weights to provide different beam widths. For example, two beam patterns with the same direction but different beam widths may have different numbers of weights. Alternatively, two beam patterns with the same direction but different beam widths may have the same number of weights, but one of the beam patterns may have some of the weights with zero values. If the weight value is zero, the microphone corresponding to the weight may actually be turned off.

[0040] Optionally, the selection (query) of the database or lookup table of beam patterns may further include the mode / metric of the device. For example, a device may be capable of various microphone configurations, and the selection may be made based on the particular microphone configuration of the device. Specifically, based on the mode / metric of the device, only horizontal beam patterns may be selected. Alternatively, some microphones may be disabled (e.g., based on power state). This disabling may result in only beam patterns being selected that have a beam width (i.e., number of weights) that corresponds to the number of enabled microphones based on the mode / metric of the device. Further details regarding the selection of beam patterns based on line of sight are discussed further below (e.g., see FIG. 9).

[0041] Returning to FIG. 7, after the beam pattern is selected (735), the method 700 further includes generating a beamformed audio based on the selected beam pattern (740) and providing the beamformed audio to a user (745). Generating the beamformed audio may be performed as described above (see, e.g., FIG. 5). For example, generating a beamformed audio signal based on the selected beam pattern may include obtaining a weight set corresponding to the selected beam pattern. Each audio channel is then applied to a corresponding weight in the weight set to generate weighted audio channels that are summed to generate the beamformed audio signal.

[0042] FIG. 9 is a flow chart of a method for selecting (i.e., obtaining) a beam pattern from the database of FIG. 8 based on the line of sight, according to an embodiment of the present disclosure. The method 900 includes comparing the determined line of sight direction (905) with the beam directions of a stored set of beam patterns. If a beam pattern for the line of sight direction is found (910), the weights of that beam pattern may be obtained (915) and used to generate a beamforming sound (see, e.g., FIG. 5). If a beam pattern for the line of sight direction is not found, the search of the database (or lookup table) may be expanded to include beam patterns for directions within an angular range (e.g., ±10 degrees) around the line of sight direction. If multiple beam patterns are found within the range (920), the beam pattern for the direction closest to the line of sight direction may be selected. The weights of the closest beam pattern may be obtained (925) and used for beamforming. Although the closest weights may cause a mismatch between the beam and the object of interest, the resulting beamforming may still provide enhanced (i.e., amplified) audio from the object of interest since the object of interest may still be within the beamwidth of the beam (see, e.g., FIG. 1). If no beam pattern is found in the line of sight direction and in the range of directions around the line of sight, a default beam pattern may be obtained (930) and used for beamforming. The default beam pattern may have weights that steer the beam in a direction (e.g., z direction) in front of the user (e.g., azimuth 0 degrees, elevation 0 degrees with respect to the coordinate system 430 of the AR glasses). Alternatively, the default pattern may have weight values ​​of zero for all microphones except one or two of the microphones in the array (e.g., stereo L / R). If the weight values ​​are zero, their corresponding microphones may actually be disabled (see, e.g., FIG. 5), while if the weight values ​​are non-zero, the gain is one and no phase shift may occur. In one possible implementation, the default pattern may allow the left and right microphones to operate as a stereo pair of microphones with no phase shift other than the interaural delay caused by the spacing of the microphones on the AR glasses.In another possible implementation, the default pattern may average the audio from the microphones by applying equal weights (eg, 1) to each channel.

[0043] The stored beam patterns may be generated based on training that is performed before (i.e., offline) beamforming is used in operation (i.e., online, run-time). Figure 10 shows a flowchart of a method for generating beam patterns according to an embodiment of the present disclosure. The method includes determining a line of sight direction (1005).

[0044] The gaze direction may be determined based on the popularity of the gaze direction over time. The popular gaze direction may be determined based on gazes monitored over time for one or more users. For example, the user's eyes may be tracked over time to determine the probability of various gaze directions or gaze points (see, for example, FIG. 2). For example, a gaze point may be determined if the gaze stays on the gaze point for longer than a certain time. A probability map of the gaze points may be generated based on the gaze points collected during the training period. For example, each possible gaze point (x, y) may have a number of gaze points collected during the training period. The probability (i.e., likelihood) of the gaze point may be the number of gaze points of the gaze point collected during the training period divided by the total number of gaze points detected during the training period. The probability map may be implemented as a heat map image in which the intensity of the more popular gaze points is higher than the intensity of the less popular gaze points. The probability map may be analyzed to determine a set of gaze directions. For example, one or more gaze points (i.e., pixels) having an intensity higher than a threshold in an area of ​​the heat map may be highlighted (i.e., selected) as popular gaze points, and gaze directions toward the popular gaze points may be determined.

[0045] The method further includes selecting (1010) a first gaze direction from the (set of) gaze directions. A target beam pattern for the selected gaze direction may be determined (1015). Determining the target beam pattern may include determining a beam width suitable for the particular gaze direction. For example, a wide beam width may be selected so that a single beam pattern can cover a range of popular gaze directions (i.e., a region of popular viewpoints). Once the target beam pattern is selected, weights for that beam pattern may be calculated. Calculating the weights may include collecting sounds from multiple directions (1020) and optimizing the following equation according to a least squares optimization process:

[0046]

number

[0047] In the above equation, y is the target beam pattern (e.g., a two-dimensional matrix with sensitivity values ​​corresponding to the beam pattern), X is the audio from multiple angles (e.g., a full rank pseudo-inverse data matrix), and w is the weights to solve for (e.g., a vector of weights corresponding to the number of audio channels). Inversion to learn weights for a particular gaze direction may be possible when the forward matrix (X) is full rank.

[0048] A practical (offline) setup for collecting audio may include moving the sound source around the AR glasses while recording the audio from each channel so that the same audio data may be collected from the sound source at various angles. An optimization may then be performed that tries different weights for each channel until the audio has a spatial sensitivity pattern that corresponds to that of the target beam pattern. As a result, a set of weights that approximates the target beam pattern is generated. The quality of the approximation may be based on the number of weights (i.e., microphones). For example, increasing the number of weights may result in a better match with the target beam pattern, such that the least squares optimization process is minimized closer to zero.

[0049] Returning to FIG. 10, after the optimal weights for the target beam patterns are calculated (1025), they may be stored in a database (or look-up table) (1030). The weights may be indexed in the database by their corresponding beam direction and / or beam width. The method may be repeated for other gaze directions by selecting the next gaze direction from the gaze directions (1035), determining the target beam pattern for the next gaze direction, and repeating the optimization process to calculate / store the weights for the next beam pattern in the database. After all beam patterns for all determined gaze directions are stored in the database, they may be downloaded to the local memory of the AR glasses or stored in a cloud memory accessible online by the AR glasses.

[0050] The stored beam patterns and lookup approach to beamforming is computationally efficient, power efficient, and fast because no optimization needs to be done on the AR glasses while the user is using them (i.e., at run time). At run time, beamforming can be done by simply recalling the weights from the database. While the weights do not provide a beam pattern that perfectly matches the user's line of sight, in many cases the audio can be enhanced enough to allow the user to better hear what they are looking at.

[0051] To determine when beamforming should occur (e.g., item 720 of FIG. 7), determining that the user's gaze does not move from the gaze point can be utilized offline (e.g., training as described above) and / or online. FIG. 11 shows a flowchart for detecting gaze for beamforming according to an embodiment of the present disclosure. The method includes determining a gaze point (i.e., gaze coordinates) (x, y) (1105). Determining gaze can be difficult due to rapid eye movements (i.e., saccades). Therefore, the method further includes temporally filtering the gaze coordinates (1110). For example, gaze coordinates obtained from real-time eye tracking can be low-pass filtered to generate a time-varying signal corresponding to gaze with little change over time. In a possible embodiment, the eye tracking coordinates can be measured and averaged over time to obtain an average eye tracking coordinate. When the average eye tracking coordinate meets a dwell time criterion, the user's gaze direction can be determined. For example, an average eye tracking coordinate that stays within a range (e.g., area) for longer than a threshold time can indicate a stable gaze. Beamforming may be determined based on the stability of the gaze. For example, if the gaze is stable (1115), beamforming may be performed in a direction corresponding to the gaze determined from the filtered gaze coordinates. In another example, beamforming may be performed in a direction corresponding to the average eye tracking coordinates determined while the gaze is stable. A stable gaze (1115) may trigger the AR glasses to enter beamforming mode (1120), while an unstable gaze (i.e., an unstable gaze) may not trigger beamforming (i.e., no beamforming (1125)). For example, in a possible implementation, the AR glasses are not set to beamforming mode unless a stable gaze is detected.

[0052] The computational efficiency, power efficiency, and speed of the stored beam patterns and lookup techniques may enable new beamforming applications. For example, beamforming may be gradually focused (zoomed in) or defocused (zoomed out) in a direction. In other words, beamforming may be changed over time. Rather than enabling beamforming on / off like a switch, zooming beamforming corresponds to increasing / decreasing beamforming over time, creating an "audio zoom" experience of the object on which the user is visually focused. Combined with the dwell time gaze detection described above, gradual beamforming (i.e., zooming beamforming) may enable smoother audio transitions than beamforming in / out.

[0053] FIG. 12 illustrates zooming in and out of beamforming according to an embodiment of the present disclosure. The figure shows three beam patterns. A first beam pattern (BP1) has a first beam width, a second beam pattern (BP2) has a second beam width smaller than the first beam width, and a third beam pattern (BP3) has a beam width smaller than the second beam width. The first beam pattern (BP1), the second beam pattern (BP2), and the third beam pattern (BP3) are oriented in the same direction (d1). The beam patterns can be stored in a database and applied in sequence (i.e., one at a time) to gradually change the beamforming (i.e., zoom in or out).

[0054] FIG. 12 includes a time graph 1210 of possible zoom-in and zoom-out sequences, displayed below the beam patterns. At a first time point 1215, audio is provided to the user without beamforming (i.e., no BF). Beamforming is triggered at a second time point 1220. Once beamforming is triggered (e.g., by a sustained line of sight), the beamforming can be zoomed in by applying a sequence that transitions from no beamforming to full beamforming, including applying a sequence of at least two beam patterns with decreasing beam widths for example for zooming in. FIG. 12 shows one possible sequence. In the sequence, at the second time point 1220, a first beam pattern (BP1) is obtained (e.g., from a database) and applied to the audio channel. Then, at a third time point 1230, a second beam pattern (BP2) is obtained and applied to the audio channel. Then, at a fourth time point 1240, a third beam pattern (BP3) is obtained and applied to the audio channel. At the fourth time point 1240, beamforming may be fully on and remain in this state based on the user's line of sight. At the fifth time point 1250, beamforming may end (e.g., the user's line of sight changes). At the fifth time point 1250, beamforming may zoom out by transitioning from the third beam pattern (BP3) to the second beam pattern (BP2). Then, at the sixth time point 1260, beamforming transitions from the second beam pattern (BP2) to the first beam pattern (BP1). Finally, at the seventh time point 1270, beamforming transitions back to no beamforming (no BF). The duration and beam width of the sequence may be adjustable (e.g., by the user). Additionally, some sequences may include more (or less) beam patterns.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Methods and materials similar or equivalent to those described herein may be used in the practice or testing of the present disclosure. As used herein and in the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. As used herein, the term "comprising" and variations thereof are used synonymously with the term "including" and variations thereof and are open-ended terms. As used herein, the term "optional" or "optionally" means that the subsequently described feature, event, or circumstance may or may not occur, and the description includes instances when the feature, event, or circumstance occurs and instances when the feature, event, or circumstance does not occur. Ranges may be expressed herein, such as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, the embodiment includes from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed as approximations, by use of the antecedent "about," it will be understood that the particular value forms another aspect. It will be further understood that the endpoints of each of the ranges are meant both in relation to the other endpoint, and independently of the other endpoint.

[0056] As described herein, certain features of the described embodiments have been illustrated, but numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It is to be understood that these are presented by way of example only, and not by way of limitation, and that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except in mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

[0057] When the foregoing description refers to an element being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, it will be understood that the element may be directly on, directly connected to, or directly coupled to another element, or there may be one or more intervening elements. In contrast, when an element is referred to as being directly on, directly connected to, or directly coupled to another element, there are no intervening elements. Throughout the detailed description, elements that are shown to be directly on, directly connected to, or directly coupled may be so referenced, even if the terms directly on, directly connected, or directly coupled are not used. The claims of this application may be amended to recite the exemplary relationships, if any, described in the specification or shown in the drawings.

[0058] As used herein, the singular may include the plural unless the context clearly indicates otherwise in a particular instance. Spatially relative terms (e.g., on, above, upper of, below, below, below, and below) are intended to encompass various orientations of the device during use or operation in addition to the orientation depicted in the drawings. In some embodiments, the relative terms "on" and "below" may include vertically above and vertically below, respectively. In some embodiments, the term "adjacent" may include laterally adjacent or horizontally adjacent.

Claims

1. receiving audio channels from a plurality of microphones of an augmented reality (AR) device, the plurality of microphones being configured to operate as a microphone array; Tracking the eyes of a user of the AR device to determine a gaze direction of the user; selecting a beam pattern for the microphone array, the beam pattern being selected from a set of stored beam patterns based on the gaze direction of the user; generating a beamformed audio signal based on the selected beam pattern; transmitting the beamformed audio signal to a speaker of the AR device for playing the beamformed audio signal to the user; A method comprising:

2. generating the beamformed audio signal based on the selected beam pattern, obtaining a set of weights corresponding to the selected beam pattern; applying a corresponding weight from the set of weights to each audio channel to generate weighted audio channels; summing the weighted audio channels to generate the beamformed audio signal, the beamformed audio signal being amplified according to the beam pattern; The method of claim 1 , comprising:

3. The method of claim 1 or 2, wherein each beam pattern in the set of stored beam patterns has a beam direction and a beam width.

4. Selecting the beam pattern from the set of stored beam patterns based on the line of sight direction of the user includes: comparing the line of sight direction to a beam direction of each beam pattern in the set of stored beam patterns; obtaining a beam pattern from the set of stored beam patterns having a beam direction closest to the line of sight direction; The method of claim 3 , comprising:

5. selecting a beam pattern from the set of stored beam patterns based on the beam width if a plurality of beam patterns has the beam direction closest to the line of sight direction; The method of claim 4 further comprising:

6. determining a power state of the AR device; selecting a beam pattern having a beamwidth based on the power conditions; The method of claim 5 further comprising:

7. Zooming in the beam patterns in the line of sight direction by generating a beamforming audio signal based on a sequence of selected beam patterns having the beam direction, each successive beam pattern in the sequence having a smaller beamwidth; The method of claim 5 or 6, further comprising:

8. Zooming out the beam patterns in the line of sight direction by generating a beamforming audio signal based on a sequence of selected beam patterns having the beam direction, each successive beam pattern in the sequence having a larger beamwidth; The method of any one of claims 5 to 7, further comprising:

9. transmitting the beamformed audio signal to a speaker of the AR device for playback to the user; splitting the beamformed audio signal into a left channel and a right channel; adjusting a phase and amplitude difference between the left and right channels based on the gaze direction of the user; 13. A method according to any one of the preceding claims, comprising:

10. determining a target beam pattern based on a training experiment; calculating weights for a beam pattern that approximates the target beam pattern; storing the weights of the beam pattern in a memory as a first beam pattern having a first line of sight and a first beamwidth; repeating the determining, calculating, and storing for other target beam patterns to generate the set of stored beam patterns; and 2. The method of any one of the preceding claims, further comprising:

11. The method of claim 10 , wherein the training experiments include gaze direction likelihoods.

12. The calculating weights of a beam pattern that approximates the target beam pattern includes: performing a least squares optimization; The method of claim 10 or 11, comprising:

13. Tracking the eyes of a user of the AR device to identify a gaze direction of the user includes: Measuring eye tracking coordinates over time; and averaging the eye tracking coordinates over time to obtain average eye tracking coordinates; determining the gaze direction of the user based on the average eye tracking coordinates if the average eye tracking coordinates are within range of each other during a dwell time; 13. A method according to any one of the preceding claims, comprising:

14. acquiring an image of the user's field of view; analyzing the image within a region based on the gaze direction; ascertaining the gaze direction based on the analysis; and triggering beamforming when the gaze direction is determined, the beamforming including selecting a beam pattern from a set of stored beam patterns based on the gaze direction of the user; The method of claim 13 further comprising:

15. 1. An augmented reality (AR) device, comprising: a microphone array including a microphone configured to generate audio channels based on sounds from an environment; an eye tracker configured to determine a gaze direction of a user of the AR device; A speaker and A processor configured by software; The processor, by the software, receiving the audio channel from the microphone array; receiving the gaze direction from the eye tracker; selecting a beam pattern for the microphone array from a set of stored beam patterns based on the gaze direction of the user; generating a beamformed audio signal based on the selected beam pattern; transmitting the beamformed audio signal to the speaker for playing the beamformed audio signal to the user; The AR device is configured as follows.

16. A smart glass, a microphone array including a microphone configured to generate audio channels based on sounds from an environment; An eye tracker configured to determine a gaze direction of a user of the smart glasses; A speaker and A processor configured by software; The processor, by the software, receiving the audio channel from the microphone array; receiving the gaze direction from the eye tracker; obtaining weights for the audio channels from a look-up table based on the gaze direction; applying the weights to the channels and summing the channels to generate a beamformed audio signal that amplifies sounds in the environment from the line of sight; transmitting the beamformed audio signal to the speaker for playback to the user; The smart glasses are configured as follows.

17. To obtain the weights for the audio channels from the lookup table, the processor further comprises: accessing the lookup table containing a set of weights for a plurality of predetermined gaze directions; selecting a weight for one predetermined gaze direction that is closest to said gaze direction; The smart glasses of claim 16, configured to:

18. The smart glasses of claim 17 , wherein the plurality of predetermined gaze directions is based on likelihoods of gaze directions determined by a training process.

19. 19. The smart glasses of claim 17 or 18, wherein the lookup table is stored on the smart glasses.

20. 19. The smart glass of claim 17 or 18, wherein the lookup table is stored on a network connected to the smart glass.

21. The smart glass of any one of claims 16 to 21, wherein the microphone array includes a vertical array generating a vertical audio channel and a horizontal array generating a horizontal audio channel.

22. 22. The smart glass of claim 21, wherein the gaze direction includes a horizontal gaze direction and a vertical gaze direction, and the processor is configured to determine a horizontal weight for the horizontal audio channel based on the horizontal gaze direction and to determine a vertical weight for the vertical audio channel based on the vertical gaze direction.

Citation Information

Patent Citations

  • Sound input device

    JP1995177595A

  • Design and implementation of super-directional beamforming

    JP2003514481A

  • Sound collecting device, hearing aid, and sound collecting device set

    JP2019054385A

  • Gaze-based audio direction

    US20160080874A1

  • Hearing device or system adapted for navigation

    US20190174237A1