Device, method and medium for audio spatialization and enhancement between multiple head mounted devices
By using the audio system of a head-mounted device to spatialize and enhance audio signals based on the user's position and gaze direction, the problem of audio signals being difficult to identify and switch in multi-sound-source environments is solved, achieving clearer audio transmission and focusing effects.
Patent Information
- Application Number
- CN202080034504.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-07
- Filing Date
- 2020-05-05
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-05-05
AI Technical Summary
In environments with multiple sound sources, listeners may find it difficult to distinguish and switch between audio signals from different sources, especially in environments where multiple people are speaking at the same time. This can lead to cocktail party problems, where it becomes difficult to identify and focus on specific speakers.
The audio system of the head-mounted device spatializes and enhances the audio signal based on the user's position and gaze direction, making the audio data appear to originate from a specific location, and adjusts the amplitude of the audio signal according to the deviation so that the user can hear the audio data they are interested in more clearly.
It improves the ability to focus on specific audio signals in noisy environments, allowing users to hear and distinguish different sound sources more clearly and reducing distractions.
Smart Images

Figure CN113796097B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Application No. 16 / 405,950, filed May 7, 2019, the contents of which are incorporated herein by reference in their entirety for all purposes.
[0003] background
[0004] The present disclosure relates generally to audio communications between users on a shared communication channel, and more particularly to spatialization and enhancement of audio signals transmitted between multiple different users of a shared communication channel.
[0005] In an environment with multiple sound sources, a listener may have trouble tuning out sounds from other sound sources while tuning into a particular sound source. For example, in a busy room with multiple people speaking at the same time, it is difficult for a listener to distinguish the words of a particular speaker from the voices of other speakers in the room. This phenomenon is known as the cocktail party problem. In some cases, different sound sources, such as speakers, may have microphones that record their voices, which are transmitted to the listener to hear. However, it may be difficult for the listener to identify the sound source, especially when there are a large number of sound sources, or it may be difficult to switch their attention between different sound sources.
[0006] Overview
[0007] Embodiments relate to establishing a shared communication channel for transmitting and receiving audio content between multiple users. Each user is associated with a head-mounted device that is configured to transmit audio data to and receive audio data from the head-mounted devices of other users. In response to receiving audio data corresponding to a second user, the head-mounted device of a first user spatializes the audio data based on the relative positions of the first user and the second user, so that the audio data presented to the first user appears to originate from the position corresponding to the second user. The head-mounted device can also enhance the audio data based on the deviation between the position of the second user and the enhancement direction (e.g., the gaze direction of the first user), thereby allowing the first user to more clearly hear the audio data from other users that they are paying attention to.
[0008] In some embodiments, a head mounted device is described. The head mounted device includes a gaze determination system configured to determine a gaze direction of a first user wearing the head mounted device. The head mounted device also includes a receiver configured to receive audio data associated with a second user, the audio data including an audio output signal. The head mounted device also includes a processing circuit configured to identify a relative position associated with the second user relative to the first user, and to determine a deviation of the identified relative position of the second user relative to the gaze direction of the first user. The processing circuit spatializes the audio output signal associated with the second user based on the relative position associated with the second user. In response to the deviation of the identified position of the second user relative to the gaze direction of the first user being within a threshold amount, the processing circuit may also enhance the amplitude of the audio output signal based on the deviation. The head mounted device also includes an audio output interface configured to send the spatialized and enhanced audio output signal to one or more speakers to produce an output sound so as to present the output sound to the first user so that the output sound is perceived as originating from the position of the second user.
[0009] The method may be performed by an audio system. For example, an audio system as part of a head-mounted device (e.g., a near-eye display, a head-mounted display). The audio system includes a microphone assembly, a transceiver, a controller, and a speaker assembly (e.g., a speaker array).
[0010] In particular, embodiments according to the invention are disclosed in the attached claims relating to head mounted devices, methods and storage media, wherein any feature mentioned in one claim category (e.g. head mounted devices) may also be claimed in another claim category (e.g. methods, storage media, systems and computer program products). Dependencies or back references in the attached claims are selected for formal reasons only. However, any subject matter resulting from an intentional back reference to any previous claim (in particular multiple dependencies) may also be claimed, such that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features set out in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in a claim may be combined with any other feature or combination of other features in the claims. Furthermore, any embodiment and feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or with any feature of the attached claims.
[0011] In one embodiment, a head mounted device may include:
[0012] a gaze determination circuit configured to determine a gaze direction of a first user of the head mounted device;
[0013] a transceiver configured to receive an audio signal associated with a head mounted device of a second user;
[0014] A processing circuit configured to:
[0015] determining a relative position associated with the second user relative to the first user;
[0016] determining a deviation of a second user's position relative to an augmented direction of the first user, wherein the augmented direction is based at least in part on a gaze direction of the first user;
[0017] spatializing an audio signal associated with the second user based at least in part on a relative position associated with the second user; and
[0018] enhancing the amplitude of the audio output signal based at least in part on the deviation of the identified position of the second user relative to the enhancement direction of the first user; and
[0019] A speaker assembly is configured to project sound based on the spatialized and enhanced audio output signal such that the projected sound is perceived to originate from a position of the second user.
[0020] In one embodiment, the head mounted device may include a microphone array including a plurality of microphones arranged in a plurality of different positions, and the microphone array may be configured to capture sound in a local area of the first user and generate an audio input signal.
[0021] The processing circuit can be configured to:
[0022] analyzing the audio input signal to identify sounds originating from a specific area in the local area of the first user; and
[0023] A user audio signal is generated from the audio input signal by enhancing a portion of the audio input signal corresponding to the sound originating from the specific area.
[0024] The specific area may correspond to the first user's mouth.
[0025] The transceiver may be configured to receive location information of the second user.
[0026] In one embodiment, the head mounted device may include an antenna array configured to determine a relative position associated with a second user relative to the first user.
[0027] The processing circuitry may be configured to spatialize the audio output signal based on whether there is a line of sight between the first user and the second user.
[0028] The gaze determination circuit may be configured to:
[0029] receiving a location of a first user, the location comprising at least a head orientation of the first user; and
[0030] determining a relative orientation of the first user's eyes relative to the first user's head; and
[0031] Wherein spatializing the audio output signal associated with the second user is based on a relative direction of a position of the second user with respect to a head orientation of the first user.
[0032] The receiver may be configured to receive a second audio signal from a third user, and the processing circuit may be configured to:
[0033] identifying a relative location associated with a third user relative to the first user;
[0034] determining a deviation of the relative position of the identified third user relative to the augmented direction of the first user;
[0035] comparing the identified deviation of the relative position of the third user to the identified deviation of the relative position of the second user; and
[0036] Based on a result of the comparison, the amplitude of the second audio signal associated with the third user is enhanced.
[0037] In an embodiment, a method may include:
[0038] At a head mounted device of the first user, determining an augmented direction of the first user;
[0039] receiving, at the first user's head mounted device, an audio signal associated with the second user's head mounted device;
[0040] identifying a relative position associated with the second user relative to the first user;
[0041] determining a deviation of the identified relative position of the second user relative to the enhanced direction of the first user;
[0042] spatializing an audio signal associated with the second user based at least in part on a relative position associated with the second user; and
[0043] enhancing the amplitude of the audio output signal based at least in part on the deviation of the identified position of the second user relative to the enhancement direction of the first user; and
[0044] Sound is projected based on the spatialized and enhanced audio output signal such that the projected sound is perceived to originate from the location of the second user.
[0045] In one embodiment, a method may include using a microphone array including a plurality of microphones arranged in a plurality of different locations to capture sound in a local area of a first user and generate an audio input signal.
[0046] In one embodiment, a method may include:
[0047] analyzing the audio input signal to identify sounds originating from a specific area in the local area of the first user;
[0048] A user audio signal is generated from the audio input signal by enhancing a portion of the audio input signal corresponding to the sound originating from the specific area.
[0049] The specific area may correspond to the first user's mouth.
[0050] In one embodiment, a method may include receiving location information of a second user.
[0051] In one embodiment, a method may include receiving, at an antenna array, a signal from a head mounted device of a second user, and determining a relative position associated with the second user with respect to a first user based on the received signal.
[0052] The spatialized audio output signal may be based on whether there is a line of sight between the first user and the second user.
[0053] Determining the enhanced direction of the first user may include determining the gaze direction of the first user by the following steps:
[0054] receiving a location of a first user, the location comprising at least a head orientation of the first user;
[0055] determining a relative orientation of the first user's eyes relative to the first user's head; and
[0056] determining a gaze direction based on the head orientation and a relative orientation of the first user's eyes with respect to the first user's head; and
[0057] Wherein spatializing the audio output signal associated with the second user is based on a relative direction of the second user's position relative to the orientation of the first user.
[0058] In one embodiment, a method may include:
[0059] receiving a second audio signal from a third user;
[0060] identifying a relative location associated with a third user relative to the first user;
[0061] determining a deviation of the relative position of the identified third user relative to the augmented direction of the first user;
[0062] comparing the identified deviation of the relative position of the third user to the identified deviation of the relative position of the second user; and
[0063] Based on a result of the comparison, the amplitude of the second audio signal associated with the third user is enhanced.
[0064] In one embodiment, a non-transitory computer-readable medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
[0065] At a head mounted device of the first user, determining an augmented direction of the first user;
[0066] receiving, at the first user's head mounted device, an audio signal associated with the second user's head mounted device;
[0067] identifying a relative position associated with the second user relative to the first user;
[0068] determining a deviation of the identified relative position of the second user relative to the enhanced direction of the first user;
[0069] spatializing an audio signal associated with the second user based at least in part on a relative position associated with the second user; and
[0070] enhancing the amplitude of the audio output signal based at least in part on a deviation of the identified position of the second user relative to the enhancement direction of the first user; and
[0071] Sound is projected based on the spatialized and enhanced audio output signal such that the projected sound is perceived to originate from the location of the second user.
[0072] Determining the enhanced direction of the first user may include determining the gaze direction of the first user by the following steps:
[0073] receiving a location of a first user, the location comprising at least a head orientation of the first user;
[0074] determining a relative orientation of the first user's eyes relative to the first user's head; and
[0075] determining a gaze direction based on the head orientation and a relative orientation of the first user's eyes with respect to the first user's head; and
[0076] Wherein spatializing the audio output signal associated with the second user is based on a relative direction of the second user's position relative to the orientation of the first user.
[0077] In an embodiment, one or more computer-readable non-transitory storage media may contain software that, when executed, is operable to perform a method according to or in any of the embodiments described above.
[0078] In an embodiment, the system may include: one or more processors; and at least one memory coupled to the processor and including instructions executable by the processor, and the processor may be operable to perform a method according to any of the above embodiments or a method in any of the above embodiments when executing the instructions.
[0079] In an embodiment, a computer program product, preferably comprising a computer readable non-transitory storage medium, is operable to perform a method according to or in any of the above embodiments when executed on a data processing system. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] Figure 1 A high-level diagram of an environment in which a system for audio spatialization and enhancement may be used in accordance with one or more embodiments is shown.
[0082] Figure 2 is an example of a head mounted device including an audio system that may be worn by a user sharing a communication channel in accordance with one or more embodiments.
[0083] Figure 3 A block diagram of an audio system is shown in accordance with one or more embodiments.
[0084] Figure 4 An example of an environment with multiple users utilizing a shared communication channel is shown in accordance with one or more embodiments.
[0085] Figure 5 A diagram illustrating filtering a user audio signal in accordance with one or more embodiments is shown.
[0086] Figure 6 is a flow chart of a process for spatializing and enhancing audio data received from other users in a shared communication channel in accordance with one or more embodiments.
[0087] Figure 7 is a flow chart of a process for processing an audio signal corresponding to a user's speaking voice according to one or more embodiments.
[0088] Figure 8 A system environment of a head mounted device including the audio system as described above according to one or more embodiments.
[0089] The accompanying drawings depict several embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the present invention described herein.
[0090] Detailed Description
[0091] Embodiments relate to a head mounted device having an audio system configured to receive audio signals from multiple audio sources and play back the received audio signals to a user (e.g., a wearer of the head mounted device). The audio system spatializes the audio signals received from a particular audio source based on the relative position of the audio source so that the audio signals played back to the user appear to originate from the position of the audio source. In some embodiments, the audio system enhances the audio signals received from the audio sources based on the position of the audio sources and the enhancement direction of the user (e.g., the gaze direction) so as to emphasize the audio data received from the particular audio source and allow the user to switch their attention between different audio sources.
[0092] In some embodiments, a shared communication channel is established between multiple users within a local area network. Each user wears a head mounted device including a transceiver for communicating with other users in the shared communication channel (e.g., transmitting and receiving audio signals). Each head mounted device also includes a sensor configured to track the position and gaze direction of its user, which can be used to determine the relative positions of other users sharing the communication channel and how the positions of other users relate to the gaze direction of the user.
[0093] The head mounted device processes audio signals received from other users of a shared communication network based on the relative positions of the other users of the shared communication channel, so that when the audio signals are played back to the user, they will appear to originate from positions corresponding to the other users. The audio signals are also enhanced based on the user's enhancement direction (the user's enhancement direction can be based on the user's gaze direction and can be used to infer which other users the user is focusing on), where audio signals from other users at positions aligned with the user's enhancement direction can be more strongly enhanced. For example, a first user receives an audio signal from each other user of a shared communication channel, the audio signal is spatialized to indicate the relative position of each other user relative to the first user, and is enhanced based on which other user the first user is currently looking at (e.g., the other user determined by the gaze direction).
[0094] In addition, the head mounted device also includes a microphone for recording the user's own voice. The user's own voice can then be transmitted to the head mounted devices of other users in the shared communication channel. In addition, in some embodiments, the user's own voice can be played back to the user to help the user adjust the volume of their own voice.
[0095] Various embodiments may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof for, for example, creating content in artificial reality and / or otherwise being used in artificial reality (e.g., performing activities therein). Artificial reality systems that provide artificial reality content can be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
[0096] Figure 1 A high-level diagram of an environment including an audio system 115 is shown in accordance with one or more embodiments. The audio system 115 may be integrated as part of a head mounted device 110 that may be worn by a user 105A.
[0097] User 105A wearing a head mounted device 110 including an audio system 115 is in an environment close to multiple other users (users 105B, 105C, 105D, and 105E). Users 105A-E may be collectively referred to as users 105. Users may talk to each other, and thus each user may be considered to correspond to an audio source. In addition, there may be additional audio sources in the environment. In an environment where a large number of audio sources are close to each other, user 105A may have difficulty focusing on any particular audio source (e.g., the voice of a particular other user in the environment).
[0098] To facilitate conversations between users in the environment, each user can wear a corresponding head-mounted device with a corresponding audio system. The audio system 115 communicates with the audio systems of other head-mounted devices to receive audio signals corresponding to the voices of other users and plays the audio signals back to user 105A. This can allow user 105A to hear the voices of other users more clearly. In addition, the audio system 115 processes the received audio signals so that the audio signals played back to user 105 are spatialized so that the played sounds are perceived to originate from the locations of other users. The played back audio signals can also be enhanced based on which other user the user 105 is currently looking at.
[0099] In some embodiments, multiple users can establish a shared communication channel. For example, Figure 1 Shown is a first shared communication channel 120A with three users, and a second shared communication channel 120B with two users.Shared communication channel 120 may correspond to a specific user group wishing to talk to each other.For example, shared communication channel 120 may include multiple users (e.g., users sitting on the same table) within a specific proximity to each other.As used herein, shared communication channel may refer to a grouping of multiple users, each user being associated with a corresponding audio system, wherein the audio system of each user may communicate with the audio system of each other user in the grouping.For example, each user in three users 105A, 105B and 105C of shared communication channel 120A has a corresponding audio system communicating with each other, and each user in two users 105D and 105E of shared communication channel 120B has a corresponding audio system communicating with each other.
[0100] In some embodiments, a shared communication channel may include one or more remote users. A shared communication channel may include multiple users within a specific geographic area (e.g., corresponding to a specific room, building, etc.). In some embodiments, the geographic area may be defined based on one or more structures (e.g., walls). As used herein, a remote user may correspond to a user participating in a shared communication channel that is located outside the geographic area corresponding to the channel. For example, a shared communication channel may include a group of users sitting at a common table, and one or more additional remote users located in different buildings.
[0101] although Figure 1 Each shared communication channel 120A and 120B is shown corresponding to a different area, but in some embodiments, different shared communication channels cover overlapping areas. For example, users of shared communication channel 120B can be mixed with users of shared communication channel 120A in a common area. In some embodiments, a particular user can be part of more than one shared communication channel (e.g., both shared communication channels 120A and 120B).
[0102] In some embodiments, a shared communication channel 120 can be established by a group of one or more users through information exchange. For example, a first user can join a common shared communication channel with respect to a second user by scanning (e.g., using their respective head mounted device 110 or other scanning device) information corresponding to the second user (e.g., based on a head mounted device worn by the second user or a scannable object such as a badge). In some embodiments, the shared communication channel is implemented as part of a peer-to-peer network established between at least the head mounted devices of the first user and the second user.
[0103] In some embodiments, one or more users 105 access application server 130 via network 125. The network may include the Internet, a local area network (LAN), a wide area network (WAN), a mobile wired or wireless network, a private network, a virtual private network, or a combination thereof.
[0104] The application server 130 includes one or more applications that facilitate communication between head mounted devices of different users, and may correspond to an online system, a local console, or some combination thereof. For example, the application server 130 may include an application that establishes a shared communication channel between two or more users and maintains metadata corresponding to the established shared communication channel. The application server may include an online system. Each user may log into the online system on the application server 130 and indicate one or more other users with whom they wish to communicate. In some embodiments, if two users 105 both indicate that they wish to communicate with another user, a connection between the two users may be established. A shared communication channel may be formed for each user group, wherein each user in the group is connected to each other user in the group.
[0105] In other embodiments, a first user may establish a shared communication channel 120, and then additional users may join the shared communication channel 120. For example, the first user may provide a password or other type of authentication to each additional user to allow the additional user to join the shared communication channel via the application server 130 (e.g., providing the password to the additional user verbally or in writing, or transmitting the password to the head mounted device of the additional user indicated by the first user through a user interface). In some embodiments, the application server 130 maintains the shared communication channel 120 and transmits updates about the current state of the channel to the head mounted device of each user of the channel (e.g., when a new user joins the channel, or when an existing user exits the channel). In some embodiments, the application server 130 is used to maintain information corresponding to the shared communication channel 120 and transmit the current state information about the shared communication channel to the head mounted device of each user, and the communication of audio data between the head mounted devices can be performed peer to peer.
[0106] In some embodiments, the application server 130 includes a social networking system. The social networking system may maintain a social graph or other data structure indicating relationships (e.g., friend relationships) between different users. In some embodiments, only users with a specific type of relationship on the social networking system can establish a connection with each other to form a shared communication channel. In some embodiments, the social graph maintained by the application server 130 can be used to automatically establish a shared communication channel 120 between multiple users. For example, a group of users who are all located in a specific geographic area and have a specific type of social network relationship with each other can be automatically included in the shared communication channel.
[0107] In some embodiments, some or all of the functions of the application server 130 may be performed by a local console. For example, the local console may be connected to multiple head mounted devices 110 corresponding to different users 105 in the local environment, and establish and maintain one or more shared communication channels between the user groups in the environment. In some embodiments, one or more head mounted devices 110 may be connected to the application server 130 via the local console.
[0108] Figure 2 is an example of a head mounted device 110 including an audio system that can be worn by a user in a shared communication channel according to one or more embodiments. The head mounted device 110 presents media to the user. In one embodiment, the head mounted device 110 can be a near eye display (NED). In another embodiment, the head mounted device 110 can be a head mounted display (HMD). Typically, the head mounted device can be worn on the face of a user (e.g., user 105) so that content (e.g., media content) is presented using one or two lenses 210 of the head mounted device. However, the head mounted device 110 can also be used so that the media content is presented to the user in a different manner. Examples of media content presented by the head mounted device 110 include one or more images, videos, audio, or some combination thereof. The head mounted device 110 includes an audio system and can include a frame 205, a lens 210, a camera assembly 235, a position sensor 240, an eye tracking sensor 245, and a controller 215 for controlling the audio system and other various sensors of the head mounted device 110. Although Figure 2 Components of the head mounted device 110 are shown in example locations on the head mounted device 110, but the components may be located elsewhere on the head mounted device 110, on a peripheral device that is paired with the head mounted device 110, or some combination of both.
[0109] The head mounted device 110 may correct or enhance the user's vision, protect the user's eyes, or provide images to the user. The head mounted device 110 may be glasses that correct the user's imperfect vision. The head mounted device 110 may be sunglasses that protect the user's eyes from sunlight. The head mounted device 110 may be safety glasses that protect the user's eyes from impact. The head mounted device 110 may be a night vision device or infrared goggles to enhance the user's vision at night. The head mounted device 110 may be a near-eye display that generates artificial reality content for the user. Alternatively, the head mounted device 110 may not include the lens 210 and may be a frame 205 with an audio system that provides audio content (e.g., music, radio, podcasts) to the user.
[0110] The lens 210 provides or transmits light to a user wearing the head mounted device 110. The lens 210 may be a prescription lens (e.g., a single vision lens, a bifocal and trifocal or a progressive lens) to help correct the user's imperfect vision. The prescription lens transmits ambient light to the user wearing the head mounted device 110. The transmitted ambient light may be altered by the prescription lens to correct the user's imperfect vision. The lens 210 may be a polarized lens or a tinted lens to protect the user's eyes from sunlight. The lens 210 may have one or more waveguides as part of a waveguide display, wherein the image light is coupled to the user's eyes through the ends or edges of the waveguides. The lens 210 may include an electronic display for providing image light and may also include an optical block for amplifying the image light from the electronic display.
[0111] In some embodiments, the head mounted device 110 includes a camera assembly 235 that captures visual information about a local area around the head mounted device 110. In some embodiments, the camera assembly 235 corresponds to a depth camera assembly (DCA) that captures data describing depth information of the local area. In some embodiments, the DCA may include a light projector (e.g., structured light and / or flash lighting for time of flight), an imaging device, and a controller. The captured data may be an image of light projected by the light projector to the local area captured by the imaging device. In one embodiment, the DCA may include two or more cameras and a controller that are oriented to capture portions of the local area in a stereoscopic manner. The captured data may be an image of the local area captured in a stereoscopic manner by two or more cameras. The controller uses the captured data and depth determination technology (e.g., structured light, time of flight, stereo imaging, etc.) to calculate the depth information of the local area. Based on the depth information, the controller 215 may be able to determine the absolute position information of the head mounted device 110 within the local area. The DCA may be integrated with the head mounted device 110 or may be located within the local area outside the head mounted device 110. In the latter embodiment, the controller of the DCA may transmit the depth information to the controller 215 of the head mounted device 110 .
[0112] The position sensor 240 is configured to generate one or more measurement signals and estimate the current position of the head mounted device 110 based on the generated signals. In some embodiments, the current position of the head mounted device 110 is determined relative to the initial position of the head mounted device 110. The estimated position may include the position of the head mounted device 110 and / or the orientation of the head mounted device 110 or the head of the user wearing the head mounted device 110, or some combination thereof. For example, the orientation may correspond to the position of each ear relative to a reference point. In some embodiments, when the camera assembly 235 includes a DCA, the position sensor 240 uses depth information and / or absolute position information from the DCA to estimate the current position of the head mounted device 110. The position sensor 240 may include one or more accelerometers for measuring translational motion (forward / backward, up / down, left / right) and one or more gyroscopes for measuring rotational motion (e.g., pitch, yaw, roll). In some embodiments, the position sensor 240 includes other types of sensors that can be used to detect motion, such as one or more magnetometers.
[0113] In some embodiments, the position sensor 240 includes an inertial measurement unit (IMU) that rapidly samples received measurement signals and calculates an estimated position of the head mounted device 110 based on the sampled data. For example, the IMU integrates the measurement signals received from the accelerometer over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on the head mounted device 110. A reference point is a point that can be used to describe the position of the head mounted device 110. Although a reference point can generally be defined as a point in space, in practice a reference point is defined as a point within the head mounted device 110. In some embodiments, the IMU can be implemented as part of the local controller 215 instead of the position sensor 240.
[0114] Eye tracking sensor 245 is configured to provide sensor readings (e.g., captured images of the user's eyes) that can be used to determine the user's gaze direction. When wearing head mounted device 110, the user's eyes can move relative to their head, allowing the user to look in different directions without having to move their head. In this way, the user may be looking in a direction other than straight ahead relative to the position and orientation of head mounted device 110 (e.g., determined by position sensor 240).
[0115] In some embodiments, the eye tracking sensor 245 includes an eye tracking sensor configured to determine the orientation of the user's eyes. The eye tracking sensor captures and analyzes images of the user's eyes to determine the orientation of the user's eyes relative to the head-mounted device 110. In some embodiments, the eye tracking sensor includes one or more light sources and one or more cameras. One or more light sources illuminate the eyes with IR light (e.g., infrared flashes (e.g., for time-of-flight depth determination), structured light patterns (e.g., dot patterns, bar patterns, etc.), flashing patterns, etc. For example, the source can be a vertical cavity emitting laser, a light emitting diode, a micro-LED, some other infrared light IR source, or some combination thereof. One or more cameras are configured to capture images of one or both eyes illuminated with IR light from one or more sources. The camera includes an image sensor (e.g., complementary metal oxide semiconductor, charge coupled device, etc.) configured to detect light emitted from one or more sources. In some embodiments, the camera may also be able to detect light in other bands (e.g., visible light bands). The eye tracking sensor uses the captured images and depth determination technology to determine the eye orientation of one or both eyes of the user. Depth determination techniques may include, for example, structured light, time of flight, stereoscopic imaging, some other depth determination method familiar to those skilled in the art, etc. In some embodiments, the eye tracking sensor determines eye orientation based on captured images and a model of the user's eyes.
[0116] The eye orientation determined by the eye tracking sensor can be combined with the determined position of the head mounted device 110 (e.g., the position determined using the position sensor 240) to determine the user's gaze direction. For example, a vector corresponding to the user's eye orientation (which indicates the orientation of the user's eyes relative to their head) can be added to a vector corresponding to the position of the head mounted device (which indicates the position and orientation of the head mounted device within the local environment) to determine a vector corresponding to the user's gaze direction (which indicates the direction the user is looking at in the local environment). By determining the user's gaze direction, the direction the user is looking at in the environment is identified, which can be combined with knowledge of the positions of other users sharing the communication channel, allowing determination of which other user the user is looking at.
[0117] In some embodiments, eye tracking sensor 245 also receives one or more images depicting a local area within the camera's FOV from a camera of camera assembly 235 and maps the determined eye orientation to a location within the received image. In some embodiments, the module uses object recognition to identify one or more objects (e.g., other users) in the one or more images and maps the determined eye orientation to the one or more identified objects.
[0118] The audio system of the head mounted device 110 is configured to allow the user to communicate with other users within the shared communication channel 110. In some embodiments, the audio system includes a microphone assembly 225, a transceiver 230, a speaker assembly having one or more speakers, and a controller 215. The controller 215 is configured to coordinate the operation between the various components of the head mounted device 110. For example, the controller 215 can control the microphone assembly 225 to capture audio data corresponding to the user's voice for transmission to other users within the shared communication channel via the transceiver 230. In addition, the controller 215 can receive audio data corresponding to other users of the shared communication channel via the transceiver 230, and process the received audio data (e.g., spatialize and / or enhance the audio data) based on the relative positions of the other users relative to the current position of the user (e.g., the current position determined by the position sensor 240). The processed audio data can be played back to the user using the speaker assembly. For additional details about the audio system, refer to Figure 3 Have a discussion.
[0119] The microphone assembly 225 records sounds within a local area of the head mounted device 110. The local area is the environment surrounding the head mounted device 110. For example, the local area may be a room where the user wearing the head mounted device 110 is located, or the user wearing the head mounted device 110 may be outside and the local area is an external area where the microphone assembly can detect sounds. The microphone assembly 225 includes one or more acoustic sensors. In some embodiments, the acoustic sensor is configured to record the voice of the user of the head mounted device 110. To this end, the acoustic sensor may be located near the user's mouth and may have a short capture range to avoid capturing other sounds that do not originate from the user. In some embodiments, the acoustic sensor may be located on a separate mouthpiece or other structure to be positioned closer to the user's mouth.
[0120] In some embodiments, the acoustic sensor includes a port corresponding to a hole in the frame 205 of the head mounted device 110. The port provides an input coupling point for sound from the local area to an acoustic waveguide, which guides the sound to the acoustic sensor. The acoustic sensor captures the sound emitted from one or more sound sources in the local area and is configured to detect the sound and convert the detected sound into an electronic format (analog or digital). The acoustic sensor can be a sound wave sensor, a microphone, an acoustic transducer, or a similar sensor suitable for detecting sound.
[0121] although Figure 2 The microphone assembly 225 is shown as being located at a single location on the head mounted device 110, but in some embodiments, the microphone assembly 225 includes a microphone array having multiple acoustic detection locations located on the head mounted device 110. Each acoustic detection location can include an acoustic sensor or port. The acoustic detection location can be placed on an outer surface of the head mounted device 110, placed on an inner surface of the head mounted device 110, separate from the head mounted device 110 (e.g., part of some other device), or some combination thereof.
[0122] The transceiver 230 is configured to communicate with the transceivers of other head mounted devices of other users. For example, the transceiver 230 can transmit data (e.g., audio corresponding to the user's voice) to the head mounted devices of other users within the shared communication channel, and receive data (e.g., audio signals corresponding to the other user's voice) from the head mounted devices of other users. In some embodiments, the transceiver 230 can access a network (e.g., network 125) to communicate with an application server or console (e.g., an application server configured to maintain a shared communication channel). The transceiver 230 may include a transmitter, a receiver, or both.
[0123] The head mounted device 110 also includes a speaker assembly configured to play back one or more audio signals as sound projected to the user of the head mounted device 110. In some embodiments, the speaker assembly includes two or more speakers, which allows the sound projected to the user to be spatialized (e.g., by adjusting the amplitude of the sound projected through each speaker 220) so that the sound can be heard by the user as originating from a specific location or direction in the local area. For example, Figure 2 As shown, the speaker assembly may include a left speaker 220a and a right speaker 220b corresponding to the left and right ears of the user. In some embodiments, the speaker may cover the user's ears (e.g., a headset) or be inserted into the user's ears (e.g., an earplug).
[0124] although Figure 2 Two speakers are shown (e.g., a left speaker 220a and a right speaker 220b), but in some embodiments, the speakers may include a speaker array that includes multiple sound emission locations on the head mounted device 110. The sound emission location is the location of a speaker or port in the frame 205 of the head mounted device 110. In the case of a sound emission location, the port provides an output coupling point for sound from an acoustic waveguide that separates the speakers of the speaker array from the port. The sound emitted from the speaker travels through the acoustic waveguide and is then emitted into a local area by the port. In some embodiments, the sound emission location is placed on the outer surface of the frame 205 (i.e., the surface that does not face the user), on the inner surface of the frame 205 (the surface that faces the user), or some combination thereof.
[0125] although Figure 2 Various components of the head mounted device 110 are shown in a particular arrangement, but it should be understood that in other embodiments, the head mounted device 110 may include components different from those described here, and the components of the head mounted device 110 may have a different structure or be arranged differently. In some embodiments, some of the functions discussed above may be performed by different components or combinations of components.
[0126] In the configuration shown, the audio system is embedded in a NED worn by a user. In an alternative embodiment, the audio system may be embedded in a head mounted display (HMD) worn by a user. While the above description discusses audio components embedded in a head mounted device worn by a user, it will be apparent to one skilled in the art that the audio components may be embedded in different head mounted devices that may be worn elsewhere by the user or operated by the user without being worn.
[0127] Audio System
[0128] Figure 3A block diagram of an audio system 300 according to one or more embodiments is shown. The audio system 300 can be implemented as part of a head mounted device (e.g., head mounted device 110) and can include a microphone assembly 225, a transceiver 230, a speaker assembly 330, and a controller 215. Some embodiments of the audio system 300 have different components than those described herein. Similarly, functionality can be distributed between components in a manner different from that described herein. In some embodiments, some functions of the audio system can be part of different components (e.g., some functions can be part of the head mounted device and some functions can be part of the console and / or server).
[0129] Microphone assembly 225 is configured to capture the sound in the local area of the user and generate an audio signal corresponding to the captured sound. In some embodiments, microphone assembly 225 is configured to capture the user's voice and includes multiple microphones configured to perform beamforming (beam form) toward a specific part of the local area (e.g., near the user's mouth) to increase the detection of the sound spoken by the user of the head-mounted device. For example, each microphone generates an audio input signal corresponding to the sound detected by the microphone. By analyzing the audio input signal of each microphone, the sound originating from a specific area in the local area of the user (e.g., near the user's mouth) can be identified. Controller 215 generates a user audio signal according to the audio input signal by enhancing the part of the audio input signal corresponding to the sound originating from the specific area. In this way, a user audio signal can be generated so that it reflects the sound originating from the user's mouth or near the mouth (e.g., the sound corresponding to the user's voice). This may be useful because it allows a clear audio signal of the user's voice to be captured even in an environment with a large amount of sound from other sources (e.g., in a crowded room).
[0130] The transceiver 230 is configured to transmit data to other users in a shared communication channel of which the user is a part and to receive data from other users. For example, the transceiver 230 can receive audio data captured by the microphone assembly 225 (e.g., audio data corresponding to the user's own voice) and transmit the received audio data to the transceiver on the head-mounted device of other users in the shared communication channel. In addition, the transceiver 230 receives audio data (referred to as an audio output signal or an audio signal) output by other users of the shared communication channel, which can be processed (e.g., processed by the local controller 215) and played to the first user (e.g., via the speaker 220). The transceiver 230 transmits and receives information via electromagnetic waves. The electromagnetic waves can be, for example, radio frequency (RF), IR, or some combination thereof. In some embodiments, the transceiver 230 uses RF and / or infrared communication to communicate with the transceivers of other users in the local area. In some embodiments, multiple transceivers corresponding to the head-mounted devices of multiple users can communicate with each other (e.g., via Bluetooth or other types of protocols) to establish a local network. In some embodiments, the transceiver 230 may also communicate wirelessly with an application server (e.g., via Wi-Fi) over a network (e.g., the Internet), or with a local console configured to maintain a shared communication channel. In addition, in embodiments where the shared communication channel may include remote users, the transceiver 230 may communicate with the remote user via the application server or the local console.
[0131] In some embodiments, the data transmitted and received by the transceiver 230 includes metadata corresponding to the transmitted / received audio data. The metadata may indicate a user identity (e.g., a user ID) associated with the audio data and information from which the user's location can be derived. For example, the metadata may include current location information of the user (e.g., determined by a location sensor on the user's head-mounted device). In some embodiments, the transceiver 230 of the first head-mounted device 110 includes an antenna array, each antenna being located at a different location on the first head-mounted device 110, so that the relative timing (timing) or phase of the signal received by each antenna from the transceiver of the second head-mounted device can be used to determine the relative position of the second head-mounted device.
[0132] The speaker assembly 330 is configured to play back one or more audio signals as sound projected to the user of the head mounted device. As described above, in some embodiments, the speaker assembly 330 includes two or more speakers, which allows the sound projected to the user (e.g., by adjusting the amplitude of the sound projected through each speaker) to be spatialized so that the sound can be heard by the user as originating from a specific location or direction in a local area.
[0133] The speaker can be, for example, a dynamic transducer, a piezoelectric transducer, some other device that uses an electrical signal to generate sound pressure waves, or some combination thereof. In some embodiments, the speaker assembly 330 also includes a speaker that covers each ear (e.g., a headset, earplugs, etc.). In other embodiments, the speaker assembly 330 does not include any speakers that block the user's ears (e.g., the speakers are on the frame of the head-mounted device).
[0134] Controller 215 includes circuit modules for operating microphone assembly 225, transceiver 230, and speaker assembly 330. These circuit modules may include data storage 335, channel configuration circuit 305, position tracking circuit 310, gaze determination circuit 315, signal manipulation circuit 320, and audio filtering circuit 325. Although Figure 3 The components of the controller 215 are shown to correspond to different circuits, but it should be understood that in other embodiments, the channel configuration circuit 305, the position tracking circuit 310, the gaze determination circuit 315, the signal manipulation circuit 320 and the audio filtering circuit 325 may be embodied in software (e.g., a software module), firmware, hardware, or any combination thereof.
[0135] The data storage 335 stores data used by various other modules of the controller 215. The stored data may include one or more parameters of a shared communication channel (e.g., the identities of other users in the shared communication channel, authentication information for accessing the shared communication channel, etc.). The stored data may include location information associated with the user (e.g., the user's position and posture determined by the location sensor 240) and / or location information associated with the audio system of other users (e.g., received from the head-mounted device of the other user). In some embodiments, the data storage 335 may store one or more models of a local area. For example, the controller 215 may generate a model of a local area that indicates the location of the user and other users in the local environment, one or more objects in the local environment (e.g., detected using the camera assembly 235), etc. The data storage 335 may also store one or more eye tracking parameters (e.g., light patterns for eye tracking, a model of the user's eyes, etc.), audio content (e.g., recorded audio data, received audio data, etc.), one or more parameters for spatializing the audio content (e.g., a head-related transfer function), one or more parameters for enhancing the audio content (e.g., an algorithm for determining an attention score), one or more parameters for filtering the audio content, some other information used by the audio system 300, or some combination thereof.
[0136] The channel configuration circuit 305 is configured to maintain membership of a user in a shared communication channel. As used herein, for example, maintaining membership of a user in a shared communication channel may include: establishing a shared communication channel, adding a user as a member to an existing shared communication channel and / or removing a user, updating one or more parameters of the shared communication channel (e.g., via communication with an application server or with the audio systems of other users in the shared communication channel), performing other actions associated with the shared communication channel, or some combination of the above.
[0137] In some embodiments, the user may establish a shared communication channel by providing information corresponding to one or more additional users to the channel configuration circuit 305 (e.g., via a user interface, via a scanning device, etc.). In response, the channel configuration circuit 305 may establish the shared communication channel to include the user and the one or more additional users. In some embodiments, the channel configuration circuit 305 transmits data (e.g., via the transceiver 230) to the channel configuration circuit associated with each additional user to establish the shared communication channel.
[0138] In some embodiments, the channel configuration circuit associated with each user in the shared communication channel stores information corresponding to the channel in a corresponding data storage (e.g., data storage 335). The information may include the identities of other users in the shared communication channel, authentication information required for communication on the shared communication channel, etc. In some embodiments, the channel configuration circuit 305 may detect changes in one or more channel parameters, such as changes in user membership of the channel (e.g., new users join the channel, users exit the channel, changes in user priority), changes in authentication information associated with the channel, changes in other parameters of the channel, or some combination thereof. In response to detecting changes in channel parameters, the channel configuration circuit 305 may transmit the changes to the channel configuration circuits of other users in the channel. In this way, the channel communication circuits of the users may coordinate with each other so that each channel communication circuit can access the latest parameters of the shared communication channel, so that each audio system 300 can communicate with the audio systems of other users of the channel.
[0139] In other embodiments, the channel configuration circuit 305 communicates (via the transceiver 230) with an application server, which coordinates the establishment of the shared communication channel (e.g., by communicating with the channel configuration circuit of the audio system of each user to be included in the shared communication channel). For example, the channel configuration circuit 305 communicates with the application server to indicate participation in the shared communication channel and to receive parameters associated with the shared communication channel (e.g., the identities of other users within the shared communication channel, any authentication information required to communicate on the shared communication channel, etc.). In addition, the channel configuration circuit 305 can communicate with the application server to indicate any changes related to the user's participation in the channel. The application server can be responsible for maintaining the parameters of the shared communication channel and transmitting these parameters to the channel communication circuits corresponding to the users participating in the channel to ensure that the channel communication circuits have access to the latest parameters of the channel.
[0140] The position tracking circuit 310 is configured to determine the current position of the user. The position tracking circuit 310 receives position information corresponding to the user's head-mounted device from a position sensor (e.g., position sensor 240) and determines the current position of the head-mounted device based on the received position information. The position of the user's head-mounted device can indicate the user's position within the local environment and the user's orientation (e.g., the orientation of the head-mounted device on the user's head, also referred to as the user's "head orientation" below). In some embodiments, the user's position is calculated relative to a reference point. In some embodiments, one or more functions of the position tracking circuit 310 are performed by an IMU.
[0141] The location tracking circuit 310 may also be configured to determine location information corresponding to other users of the shared communication channel. In some embodiments, the location information corresponding to other users may be received directly from the head mounted devices of other users (e.g., via the transceiver 230). For example, the location information may be received as metadata accompanying audio data received from one or more other users of the shared communication channel, the location information indicating the current location of the user from which the audio data is received (e.g., determined by the location tracking module of the head mounted device of the other user). In some embodiments, the location tracking circuit 310 uses the obtained location information of other users to determine the relative position of each other user relative to the user's current position. In some embodiments, the location tracking circuit 310 may use the determined locations of other users to generate or update a model of a local area.
[0142] In other embodiments, the position tracking circuit 310 determines the location of the other user based on analyzing signals received from multiple antennas in an antenna array on the head mounted device of the other user. For example, in some embodiments, the transceiver 230 of the audio system 300 of the first head mounted device includes an antenna array, each antenna being located at a different location on the first head mounted device. The position tracking circuit 310 of the first head mounted device analyzes the signals received from the transceiver of the second head mounted device at each antenna of the array, and determines the relative position of the second head mounted device based on the relative timing or phase of the received signals. In other embodiments, the transceiver 230 receives multiple different signals transmitted by the transceiver of the second head mounted device, wherein the transceiver of the second head mounted device is an antenna array including multiple antennas located at different locations on the second head mounted device. The position tracking circuit 310 analyzes the received signals (e.g., the timing or phase of the received signals), thereby determining the relative position of the second head mounted device relative to the first head mounted device.
[0143] The gaze determination circuit 315 is configured to determine the gaze direction of the user wearing the head mounted device (e.g., based on the eye tracking measurement results of the eye tracking sensor 245, such as the captured image of the user's eyes). As used herein, the gaze direction of the user corresponds to the direction that the user is looking in the local environment. In some embodiments, the gaze direction of the user is determined based on a combination of the user's head orientation and the user's eye position. For example, the gaze determination circuit 315 can receive one or more eye tracking measurement results (e.g., one or more images of the user's eyes captured by the eye tracking camera) from the eye tracking sensor 245 to determine the user's current eye orientation, and receive the user's head orientation (e.g., determined by the position tracking circuit 310), and modify the user's head orientation using the determined eye orientation to determine the user's gaze direction within the local environment. For example, the user's head can be facing a first direction. However, if the user's eyes are oriented to look away from the first direction (e.g., not looking straight ahead), the user's gaze direction will be different from the user's head orientation.
[0144] In some embodiments, the gaze determination circuit 315 can also receive one or more images of the local area within the FOV of the camera from the camera assembly 235, and map the determined eye orientation to the position within the received image. The gaze determination circuit can use object recognition to identify one or more objects (e.g., other users) corresponding to the mapped position in one or more images to determine whether the user's gaze direction is aligned with one or more identified objects. In some embodiments, the identified objects can be used to generate or update a model of the local area. For example, the position of the identified object (e.g., other users) in one or more images can be used to determine whether the user is looking at any identified object, the position of the identified object relative to the user, whether the user has a line of sight to the identified object, etc.
[0145] although Figure 3 The position tracking circuit 310 and the gaze determination circuit 315 are shown as separate modules, but in some embodiments, the position tracking circuit 310 and the gaze determination circuit 315 can be implemented as a single module. For example, the single gaze determination circuit can receive sensor measurements (e.g., position data from the position sensor 240 and eye tracking data from the eye tracking sensor 245) to determine the position of the user, the orientation of the user's head, and the orientation of the user's eyes relative to their head, thereby determining the user's gaze direction.
[0146] The signal manipulation circuit 320 is configured to receive one or more audio signals received via the transceiver 230, each audio signal corresponding to an audio system of another user sharing the communication channel (referred to as a "transmitting audio system"), and based on the relative position of the other audio system relative to the user, process the signal to generate audio data to be presented to the user.
[0147] The signal manipulation circuit 320 identifies the relative position of the sending user relative to the user's current position. The position information can be received from the position determination module 310. In some embodiments, the signal manipulation circuit 320 accesses a model of the local area containing the position information of each user in the local area to determine the relative position of the sending user. In addition, the signal manipulation circuit 320 can receive an indication of the user's current gaze direction from the gaze determination circuit 315. Based on the relative position of the user, the signal manipulation circuit 320 can spatialize the audio signal from the sending user so that when the sound (e.g., via the speaker 220) is played to the user, it will appear to originate from the position of the sending user.
[0148] In some embodiments, the signal manipulation circuit 320 spatializes the audio signal based on one or more generated acoustic transfer functions associated with the audio system. The acoustic transfer function may be a head-related transfer function (HRTF) or other types of acoustic transfer functions. HRTF characterizes how the ear receives sound from a point in space. The HRTF relative to a specific source position of a person is unique to each ear of the person (and unique to the person), because the person's anatomical structure (e.g., ear shape, shoulders, etc.) affects the sound when the sound travels to the person's ears. For example, in some embodiments, the signal manipulation circuit 320 can generate two sets of HRTFs for the user, one set of HRTFs for each ear, which correspond to various frequencies and relative positions. One HRTF or a pair of HRTFs can be used to create audio content that includes sounds that appear to originate from a specific point in space (e.g., from the location of the sending audio system). Several HRTFs can be used to create surround sound audio content (e.g., for home entertainment systems, theater speaker systems, immersive environments, etc.), where each HRTF or each pair of HRTFs corresponds to a different point in space, so that the audio content appears to come from several different points in space. Additional examples of generating HRTFs are described in U.S. Patent Application No. 16 / 015,879, entitled “Audio System for Dynamic Determination of Personalized Acoustic Transfer Functions,” which is hereby incorporated by reference in its entirety.
[0149] In some embodiments, the signal manipulation circuit 320 can enhance the audio signal based on the position of the sending user relative to the enhancement direction. As used herein, the enhancement direction of the user can refer to the direction in which the user is inferred to be paying attention. In some embodiments, the enhancement direction of the user can correspond to the gaze direction of the user. In other embodiments, the enhancement direction can be based on the orientation of the user's head, the orientation of the user's head relative to their torso, etc. For ease of discussion, the enhancement direction will be mainly discussed as corresponding to the gaze direction, but it should be understood that in other embodiments, the enhancement direction can correspond to other directions relative to the user.
[0150] As used herein, enhancing an audio signal may refer to positively enhancing an audio signal (e.g., increasing the amplitude of the audio signal relative to other sounds or audio signals) or negatively enhancing an audio signal (e.g., decreasing the amplitude of the audio signal relative to other sounds or audio signals). For example, in some embodiments, as determined based on the user's gaze direction, an audio signal from a sending user that the user is looking at (e.g., suggesting that the user is paying attention to the sending user) is positively enhanced, while audio signals from other sending users that the user is not looking at are negatively enhanced. This may allow the user to more easily focus on speech from a particular user (e.g., the sending user they are paying attention to), while speech from other users is less distracting, especially when a large number of users are speaking at the same time. In some embodiments, the signal manipulation circuit 320 enhances each received audio signal based on an "attention score" calculated for each sending user, which will be referenced below. Figure 4 Describe in more detail.
[0151] Since the user's ears are located at fixed positions on the user's head, the signal manipulation circuit 320 can spatialize the received audio data based on the user's head orientation. On the other hand, the signal manipulation circuit 320 enhances the audio data based on the user's gaze direction to better emphasize the audio data originating from other users that the user is actually looking at or paying attention to.
[0152] Although the above discussion primarily relates to enhancing audio data based on the direction of a user's gaze, in other embodiments, the enhancement of audio data may be based on other directions, such as the user's head direction, the user's head direction modified based on the angle of the user's head relative to their torso, or some combination thereof.
[0153] The signal manipulation circuit 320 also outputs the spatialized and enhanced audio signal to the speakers of the speaker assembly 330. For example, the signal manipulation circuit 320 may output an audio signal of a different amplitude to each speaker of the speaker assembly 330 based on the spatialization and / or enhancement performed.
[0154] The audio filter circuit 325 is configured to receive a user audio signal corresponding to the user's voice (e.g., captured by the microphone assembly 225) and perform filtering on the user audio signal. The user audio signal can be transmitted to other users in the shared communication channel. In addition, in some embodiments, the user audio signal can also be played back to the user through the speaker assembly 330.
[0155] In some embodiments, because the users of the shared communication channel may be close to each other, the user may be able to hear the actual sound of the voice of the sending user, and can receive the audio data corresponding to the voice of the sending user through their head mounted device. Since it takes time to process the received audio signal, after the voice of the sending user can be heard at the user's position, the audio data can be presented to the user (for example, through the speaker assembly 330). The delay between the moment when the actual voice of the sending user can be heard at the user's position and the moment when the audio data of the sending user is played to the user through the speaker assembly 330 is called processing delay. If the processing delay exceeds a certain amount of time, the audio data presented to the first user may sound like an echo to the first user. This produces an undesirable audio effect that may distract the user's attention. For example, in some embodiments, when the processing delay is greater than 10 to 15 milliseconds, an echo effect is produced.
[0156] In some embodiments, the audio filtering circuit 325 includes an all-pass filter that manipulates the phase of the user audio signal to produce a temporally dispersed user audio signal (hereinafter referred to as a "diffused user audio signal"). The diffuse user audio signal may include multiple diffuse reflections of the user audio signal, which have the same total energy as the original unfiltered signal. For sounds corresponding to speech, the diffuse user audio signal allows it to be less likely to be detected by the human ear as a separate auditory event compared to the unfiltered signal. This allows the user audio signal to experience a longer processing delay before it will be detected as a separate echo by other users receiving the user audio signal. Refer to the following Figure 5 An example of a diffuse user audio signal is described in more detail. Although the present discussion relates to the audio filter circuit 325 performing time dispersion on the user audio signal in preparation for transmitting the user audio signal to other users in the shared communication channel, in some embodiments, the audio filter circuit 325 performs time dispersion on the audio signal received from the audio system of other users before playing it back to the user, rather than performing time dispersion on the user audio signal. In some embodiments, other filtering techniques may be used. For example, in some embodiments, the audio filter circuit 325 may modify the frequency amplitude spectrum of the user audio signal to replace or supplement the time dispersion of the user audio signal.
[0157] In some embodiments, the audio filter circuit 325 also filters the user audio signal to generate a modified user audio signal to be played back to the user of the audio system 300. When a user speaks in a noisy environment and / or speaks into a microphone, the user may often be unaware of the volume of their own voice because the sound of the user's own voice is drowned out by the noisy environment. As a result, the user may inadvertently increase their voice more than necessary.
[0158] In order to prevent the user from trying to speak loudly to cover the noise of the environment, a version of the user audio signal can be played back to the user so that the user can more accurately assess the volume of their speaking voice. Because people hear their own voices sound different from their voices captured by the microphone (for example, due to the vibrations in their skulls caused by the user's vocal cords in addition to the sound waves through the air reaching their ears), the user audio signal can be modified so that the user recognizes the sound of the user audio signal as their own voice. In some embodiments, the user audio signal passes through one or more filters, which approximate the effect of the skull vibration on the user's voice as perceived by the user. In some embodiments, one or more filters are configured to be generally suitable for most people (for example, based on the average skull shape and size). In other embodiments, one or more filters can be customized based on one or more user settings. For example, the user of the head mounted device 110 can configure one or more settings of the filter during setup to more closely approximate how they hear their voices. In some embodiments, the filter may include a low pass filter, wherein the user can adjust the slope and cutoff frequency of the filter. In some embodiments, the filter may include a series of one or more tunable biquad filters, FIR (finite impulse response) filters, or some combination thereof.
[0159] Thus, by feeding back a filtered version of the user audio signal to the user, the user can more accurately assess the volume of their voice even in a noisy environment, and can avoid raising their voice unnecessarily. In some embodiments, the filtered version of the user audio signal is played back to the user only when the amount of noise in the environment (e.g., measured using a microphone assembly) exceeds a threshold, so that if the amount of noise in the environment is such that the user can be expected to be able to hear their own voice, the filtered user audio signal is not played back to the user.
[0160] Location-based audio spatialization and enhancement
[0161] Figure 4An example of an environment with multiple users utilizing a shared communication channel is shown in accordance with one or more embodiments. The environment includes multiple users, including a first user 405A and additional users 405B, 405C, and 405D (collectively, users 405), who are part of the shared communication channel. Each user 405 wears a corresponding head mounted device 410 that includes an audio system that the user uses to communicate with other users of the shared communication channel. For ease of explanation, in Figure 4 Only the head mounted device 410 worn by the first user 405A is marked.
[0162] The head mounted device 410 of the first user 405A includes a position sensor and an eye tracking sensor configured to determine the position and gaze direction of the first user 405A, which can be used to determine the augmented direction 415 of the first user 405A. Figure 4 The augmented direction 415 of the first user 405A is shown aligned with the head mounted device 410 and the orientation of the head of the user 405A, but the augmented direction 415 may not be aligned with the orientation of the head of the user 405A. For example, in some embodiments, the augmented direction 415 may correspond to the gaze direction of the user 405A. Thus, as the user 405A moves their eyes, the augmented direction 415 may change even if the position of the user 405A and the orientation of the head of the user 405A remain stationary. In other embodiments, the augmented direction of the user 405A may correspond to the user's head direction (e.g., based on the orientation of the user's head), the user's head direction modified according to the angle between the user's head and torso orientation (e.g., as the angle between the user's head and torso increases, the augmented direction 415 deviates from the user's head direction), or some combination thereof.
[0163] Each of the other users 405B, 405C, and 405D within the environment may be a sending user. In response to the speech of each of the users 405B, 405C, or 405D, audio data (e.g., through their respective head mounted devices) is recorded and transmitted to the head mounted device 410 of the first user 405A (and other users participating in the channel). The signal manipulation circuit 320 of the head mounted device 410 analyzes the relative position of each other user to determine how each user's audio signal should be manipulated.
[0164] In some embodiments, the audio system of the head-mounted device 410 of the first user 405A determines position information corresponding to each sending user that transmits an audio signal to user 405A, and determines, for each sending user, a relative position of the sending user relative to the head orientation of the first user 405A, and a deviation between the position of the sending user relative to the enhanced direction 415 of the first user 405A.
[0165] The audio system uses the relative position of the sending user relative to the head orientation of the first user 405A to spatialize the audio signal received from the sending user. Using the determined relative position and the determined current head orientation of the user 405A, the audio system spatializes the audio signal so that when projected to the user 405A via the speaker assembly of the head mounted device 410, the sound of the audio signal appears to originate from the position of the corresponding sending user. In some embodiments, the audio system spatializes the audio signal by setting one or more weights corresponding to each speaker of the speaker assembly. In some embodiments, the audio system uses HRTF to spatialize the audio signal. By adjusting the amplitude of the audio signal projected to the user 405A through each speaker of the speaker assembly, the generated sound can be made to appear to originate from different positions (e.g., corresponding to the position of the sending user).
[0166] For example, Figure 4 As shown, user 405B is located directly in front of the user. Thus, the audio signal from user 405B is spatialized so that the generated sound is perceived by user 405A as originating from the front of user 405A. On the other hand, user 405C and user 405D are located to the left and right of user 405A, respectively. Thus, the audio system spatializes the corresponding audio signals so that the audio corresponding to users 405C and 405D appears to originate from the corresponding positions of users 405C and 405D.
[0167] In some embodiments, spatialization is not performed on audio signals received by user 405A from a sending user that is a remote user. In other embodiments, spatialization may be performed on audio signals received from a specific type of remote user (e.g., a remote user associated with a location within a threshold distance from user 405A).
[0168] In addition, in some embodiments, if there is no line of sight between user 405A and the sending user, spatialization is not performed. For example, in some embodiments, the audio system may know (e.g., using camera assembly 235 or other types of sensors to determine) a specific type of object in the local area, such as a wall. If the vector 425 between user 405A and the sending user intersects with such an object, indicating a lack of line of sight between user 405A and the sending user, the audio signal from the sending user may not be spatialized. In some embodiments, if the distance between user 405A and the sending user is less than a threshold amount, the audio signal from the sending user can be spatialized in the absence of line of sight, but if the distance is greater than a threshold amount, it cannot be spatialized. The threshold amount may be a predetermined amount, or may be dynamically determined based on one or more user inputs, one or more determined attributes of the local area (e.g., the size of the room), or some combination thereof.
[0169] In addition, the audio system enhances each received audio signal based on the deviation of each corresponding transmitting user's position from the enhancement direction 415 of user 405A. As used herein, the deviation of the position of a transmitting user (e.g., user 405C) from the enhancement direction of user 405A can be determined based on the angle measured between the enhancement direction 415 of user 405A and the vector 425 connecting users 405A and 405C. In some embodiments, the audio system can also enhance each received audio signal based on the distance of each corresponding transmitting user's position relative to user 405A (e.g., audio signals from transmitting users closer to user 405A are enhanced more than audio signals from transmitting users farther away).
[0170] In the case where multiple audio signals from multiple other users are received and projected to user 405A, user 405A may have difficulty focusing on the voice of any one user even if the audio signals are spatialized. By selectively enhancing the received audio signals, user 405A may more easily focus on the voices from other users they are focusing on, while being less distracted by the voices from users they are not focusing on.
[0171] In some embodiments, it may be inferred which sending user user 405A is focusing on based on user 405A's augmentation direction 415. For example, if user 405A's augmentation direction 415 is aligned with another user's position, user 405A may be inferred to be focusing on that user. Figure 4 As shown, user 405A can be inferred to be paying attention to user 405B. In some embodiments, if the position of the other user is within a threshold deviation 420 relative to the enhancement direction 415, user 405A can be inferred to be paying attention to another user. Figure 4 As shown, since users 405C and 405D are more than the threshold deviation away from the enhancement direction 415, user 405A can be inferred to be not paying attention to users 405C and 405D. In some embodiments, if there are multiple sending users within the threshold deviation 420 relative to the enhancement direction 415, user 405A can be considered to be paying attention to the sending user at the position closest to user 405A, the sending user at the position with the smallest deviation relative to the enhancement direction 415, or some combination thereof.
[0172] In some embodiments, an "attention score" may be calculated for each other user sharing a communication channel. The attention score may be used as a measure of the degree to which a user may be inferred to be paying attention to another user in order to determine how much to enhance the audio signal received from the other user. The attention score for a particular user may be based on the deviation of the user's position relative to the enhancement direction 415 of the first user 405A, the distance of the user's position to the position of the first user 405A, or some combination thereof. In some embodiments, the attention score is determined as a collection (e.g., a weighted sum) of one or more factors.
[0173] The audio system enhances the audio signal received from the sending user based on whether the user 405A is paying attention to the sending user (e.g., based on the attention score of the sending user). For example, if the user 405A is inferred to be paying attention to the sending user, the audio system is positively enhancing the audio signal, and if the user 405A is inferred to be not paying attention to the sending user, the audio system is not positively enhancing the audio signal. In addition, in some embodiments, if the user 405A is inferred to be not paying attention to the sending user, the audio signal can be negatively enhanced to minimize the interference of the audio signal of the sending user that originates from the user 405A is paying attention to. In some embodiments, the audio signal of the sending user can be enhanced based on whether there is another sending user that the user 405A is determined to be paying attention to (e.g., the audio signal from the user 405C is negatively enhanced when the user 405A is inferred to be paying attention to the user 405B, but if there is no user within the threshold deviation 420 of the enhancement direction 415, it is not negatively enhanced).
[0174] In some embodiments, the audio system enhances the received audio signal based on the attention score of the corresponding sending user. In some embodiments, the amount of enhancement can also be based on the attention scores of other users (e.g., the ranking of the sending user's score relative to other sending users). Figure 4 In the example shown, the audio system of the head mounted device 410 can determine the degree to enhance the audio signals from the transmitting users 405B and 405C by comparing the deviation of each transmitting user's position relative to the enhancement direction 415 of the user 405A, and enhance each audio signal based on the result of the comparison. For example, in some embodiments, if there is a second transmitting user with a higher attention score (e.g., due to a lower deviation from the user's gaze direction), the audio signal from the first transmitting user may be enhanced less than if there is no second transmitting user (e.g., not currently transmitting an audio signal) or the second transmitting user has a lower attention score than the first transmitting user.
[0175] Because the sending user's attention score is based on the enhancement direction 415 of user 405A, when the enhancement direction 415 of user 405A changes (e.g., due to movement of their head or eyes), the attention score of each sending user can be adjusted accordingly, resulting in different amounts of enhancement of their respective audio signals. In some embodiments, the attention score of each sending user is updated periodically. In some embodiments, if the audio system detects that the change in the enhancement direction 415 of user 405A exceeds a threshold amount, the attention score of the sending user is updated.
[0176] In embodiments where augmented direction 415 corresponds to the user's gaze direction, augmented direction 415 may change very quickly because the user's 405A eyes may move very quickly. In some embodiments, augmented direction 415 is not updated unless the user's 405A gaze has not changed by more than a threshold amount for at least a threshold period of time, in order to reduce the impact of random eye movements of user 405A.
[0177] In some embodiments, the sending user's attention score may also be based on the sending user's enhancement direction. For example, if the sending user's enhancement direction is toward user 405A, the audio signal corresponding to the sending user may be modified by signal manipulation circuit 320 to be stronger than if the sending user's gaze direction is not toward user 405A. Figure 4 As shown, even though both users 405C and 405D have similar deviation magnitudes from the enhancement direction 415 of user 405A, the audio signal from user 405C may be enhanced more than the audio signal from user 405D. In some embodiments, the weight of the sending user's orientation or gaze direction on the sending user's attention score may vary based on the deviation of the sending user's position relative to the enhancement direction 415.
[0178] In some embodiments, in the case of a shared communication channel with one or more remote users, the signal manipulation circuit 320 can enhance the audio signal from the remote user based on whether the user 405A is currently paying attention to another user in the local area. In some embodiments, the user 405A can indicate via a user interface one or more modifications for how to enhance the audio signal from a specific sending user.
[0179] By processing (e.g., spatializing and / or enhancing) received audio signals based on the relative positions of corresponding transmitting users, the signal manipulation circuitry 320 thus makes it easier for user 405A to hear and focus on audio from other users that the user is paying attention to (e.g., by positively enhancing audio signals from those users), as well as allowing user 405A to better perceive where other users from whom audio signals are being received are located.
[0180] Audio filtering for echo reduction
[0181] Figure 5 A diagram illustrating filtering a user audio signal in accordance with one or more embodiments. Figure 5 A first graph is shown, which shows an audio signal measured at the opening of the ear canal of a first user. The audio system of the first user communicates with the audio system of the second user via a shared communication channel. The audio signal includes a real audio signal 510 and a transmitted audio signal 515. The real audio signal 510 corresponds to a sound pressure wave originating from the second user and measured at the user's ear canal (i.e., the user hears the second user's voice). The transmitted audio signal 515 corresponds to an unfiltered audio signal, which corresponds to the second user's voice recorded (e.g., as the user audio signal of the second user), transmitted to the audio system of the first user and played back to the first user through one or more speakers. Due to the processing delay associated with recording, transmitting, processing and playing back the transmitted audio signal, the transmitted audio signal 515 can be detected at the ear canal (i.e., audible to the user) later than the real audio 510 by the amount of time corresponding to the processing delay ΔT. If the processing delay ΔT is a certain amount of time (eg, 10-15 milliseconds), the first user may hear the transmitted audio signal 515 as a separate auditory event as the real audio signal 510, which may produce an echo effect that distracts the first user.
[0182] The second graph shows the audio measured at the first user's location when the transmitted audio is filtered using an all-pass filter to diffuse the audio signal. As shown in the second graph, the same real audio signal 510 is heard at the first user's location. However, the transmission audio signal has been filtered to produce a filtered transmission audio signal 525 including multiple diffuse reflections. Even if the filtered transmission audio signal 525 is not heard until ΔT after the real audio 510, the diffusion of the transmission audio signal 525 can cause the first user to interpret the real audio signal 510 and the filtered transmission audio signal 525 as part of the same auditory event, thereby reducing or eliminating undesirable echo effects. In this way, by filtering the audio signal, longer processing delays can be accommodated without causing undesirable echo effects to the user. In some embodiments, the audio signal is filtered at the head-mounted device of the sending user before being transmitted to other users of the shared communication channel. In other embodiments, the audio signal is filtered at the head-mounted device of the user receiving the audio signal. In some embodiments where filtering is performed on the receiver side, the audio system of the receiving headset can determine the delay between the real audio and the transmitted audio and adjust one or more filtering parameters (e.g., the amount of dispersion) based on the determined delay.
[0183] In some cases, the first user and the second user may be a distance away from each other such that the transmitted audio 525 is heard at the first user's location before the real audio 510. In some embodiments, the audio system does not perform diffusion filtering on the transmitted audio if the sending user is determined to be at least a threshold distance away from the user.
[0184] In embodiments where the shared communication channel includes at least one remote user, audio signals transmitted between the remote and non-remote users do not need to undergo filtering because the remote user cannot hear the real audio of the non-remote user (and vice versa), so there is no echo effect caused by processing delays. In addition, in some embodiments, if it is determined that the distance between the second user and the first user is at least a threshold amount or there is a specific structure (e.g., a wall) between the first and second users, so that the first user can be inferred to be unable to hear the real audio of the second user, the audio from the second user may not be filtered.
[0185] Channel Priority
[0186] In some embodiments, different users on a shared communication channel may be given different priorities. As used herein, the priority of a user of a shared communication channel may indicate the level at which an audio signal corresponding to the user's voice is enhanced relative to audio signals corresponding to other users, wherein audio signals from users with higher priority are positively enhanced relative to users with lower priority. In some embodiments, a shared communication channel may include a first group of users corresponding to a base priority and at least one user (e.g., a designated speaker or leader) associated with a high priority that is considered to take precedence over the base priority.
[0187] For example, when a user associated with a high priority (hereinafter referred to as a "priority user") is not speaking, as described above, the audio signal corresponding to the basic priority user of the shared communication channel received by the first user can be processed normally (e.g., spatialized and enhanced based on the relative positions of the users). However, after the priority user speaks, the audio signal corresponding to the priority user received by the first user is enhanced, regardless of the relative positions of the first user and the priority user. In addition, during the time when the audio signal from the priority user is played to the first user, the audio signal from the basic priority user can be attenuated to ensure that the first user can clearly hear the voice of the priority user.
[0188] In some embodiments, users of a shared communication channel may be organized into two or more different priorities. The audio signal from a user with a higher priority is enhanced relative to the audio signal from a user with a lower priority, allowing the user to hear the higher priority user's voice more clearly when the higher priority user is speaking. In some embodiments, each user of a shared communication channel may assign personalized priorities to other users of the channel based on which other user they are most interested in following.
[0189] Process Flow
[0190] Figure 6 1 is a flow chart of a process for spatializing and enhancing audio data received from other users in a shared communication channel according to one or more embodiments. The process may be performed by a head mounted device including an audio system (e.g., audio system 300). A head mounted device of a first user participates in a shared communication channel (e.g., Figure 1 10A). In other embodiments, other entities may perform some or all of the steps of the process (e.g., a console). Likewise, embodiments may include different steps and / or additional steps, or perform the steps in a different order.
[0191] The first user's head mounted device determines 605 an augmented direction of the first user. In some embodiments where the augmented direction corresponds to a gaze direction of the user, the head mounted device includes an eye tracking sensor and a position sensor for determining the gaze direction of the user. For example, the position sensor can determine the position and orientation of the head mounted device, from which the position and orientation of the first user's head can be inferred. In addition, the eye tracking sensor can be used to determine the orientation of the first user's eyes relative to their head. In this way, the combination of the position sensor and the eye tracking sensor can be used to determine the direction of the first user's gaze.
[0192] The head mounted device receives 610 an audio signal from one or more sending users sharing a communication channel (e.g., via a transceiver). The audio signal may correspond to the voice of the sending user and may include additional metadata, such as the identity of the sending user and data from which the location of the sending user can be determined.
[0193] The head mounted device determines 615 a location associated with each transmitting user from which an audio signal is received. In some embodiments, the head mounted device receives metadata associated with the audio signal indicating the location of the transmitting user (e.g., determined by a location sensor on the transmitting user's head mounted device). In other embodiments, the head mounted device receives multiple signals transmitted by multiple antennas (e.g., an antenna array) located at different locations on the transmitting user's head mounted device. Based on the phase or timing of the received signals, the head mounted device can determine the relative position of the transmitting user with respect to the first user.
[0194] The head mounted device determines 620 the relative position of each sending user relative to the first user. The relative position of the sending user can indicate the position of the sending user relative to the first user based on the head orientation of the first user (e.g., in front of the first user, to the left of the first user, etc.).
[0195] The head mounted device determines 625 a deviation between the position of each sending user and the augmented direction of the first user. The deviation indicates the position of the sending user relative to the augmented direction of the first user. In some embodiments, additionally, the head mounted device controller can determine a distance between the sending user and the first user.
[0196] The head mounted device spatializes 630 the audio signal of each transmitting user based on the position of the corresponding transmitting user relative to the first user so that the audio signal played to the first user through the two or more speakers sounds as if it originates from a specific location (e.g., the position of the transmitting user). In some embodiments, spatializing the audio signal includes configuring the amplitude of the audio signal played through each speaker so that the user can interpret the different amplitudes of the sound through the different speakers as corresponding to the sound originating from the specific location.
[0197] The head mounted device enhances 635 the audio signal of each sending user based on the deviation of the corresponding sending user's position relative to the enhancement direction of the first user. In some embodiments, if the deviation of the sending user's position does not exceed a threshold amount, the audio signal of the sending user is enhanced. In some embodiments, the magnitude of the enhancement may be inversely proportional to the amount of deviation between the sending user's position and the enhancement direction of the first user. In this way, the audio signal from the sending user will be more strongly enhanced when the sending user's position is closer to the enhancement direction of the first user than when the sending user's position is further away from the enhancement direction. In some embodiments, the amount of enhancement may also be based on the number of audio signals currently received from other sending users, the distance between the sending user and the first user, and the like. In some embodiments, the enhancement of the audio signal may include attenuation of the audio signal (e.g., negative enhancement).
[0198] Thus, by spatializing and enhancing the audio signals received from other users in a shared communication channel, a user of the head mounted device can more easily focus on the speech of the other user they are paying attention to, as well as allowing the user to perceive the location of each other user from whom the audio signal is being received. This allows the user to more clearly hear the speech they wish to focus on, even in a noisy environment, while remaining aware of other users in the environment.
[0199] Figure 7 1 is a flowchart of a process for processing an audio signal corresponding to a user's speech sound according to one or more embodiments. The process may be performed by a head mounted device including an audio system (e.g., audio system 300). A head mounted device of a first user participates in a shared communication channel (e.g., Figure 1 10A). In other embodiments, other entities may perform some or all of the steps of the process (e.g., a console). Likewise, embodiments may include different steps and / or additional steps, or perform the steps in a different order.
[0200] The head mounted device receives 705 a user audio signal corresponding to the speech of the user of the head mounted device. In some embodiments, the user audio signal is recorded by an acoustic sensor (e.g., microphone assembly 225) located near the user's mouth. In some embodiments, the user audio signal is generated by a microphone array that uses beamforming to isolate and capture sounds from a specific area (e.g., near the user's mouth) in a local area.
[0201] The head mounted device applies 710 one or more filters (eg, all-pass filters) to the user audio signal, which disperse the user audio signal in time to produce a diffuse user audio signal.
[0202] The headset transmits 715 the diffused user audio signal to the headsets of other users in the shared communication channel. By temporally dispersing the user audio signal, the amount of processing delay between the time when another user hears the user's actual voice and the time when they hear the transmitted user audio signal played to them through one or more speakers can be increased without causing the other user to hear the user audio signal as a separate auditory event, creating an undesirable echo effect.
[0203] In some embodiments, rather than temporally dispersing the user audio signal and transmitting the diffused user audio signal to the head mounted devices of other users, the temporal dispersal of the audio signal is performed by the head mounted device receiving the audio signal. In some embodiments, the user audio signal is temporally dispersed based on one or more filter parameters that can be adjusted based on the relative position or distance between the head mounted devices of the sending user and the receiving user.
[0204] The headset applies 720 a speech filter to the user audio signal to produce an altered version of the user audio signal. The speech filter is configured to simulate the effect that when a person speaks, the vibrations of a person's vocal cords propagated through the skull affect how they hear their own speech. In some embodiments, the user can manually configure one or more parameters of the speech filter so that the altered user audio signal more closely matches how they hear their own speech.
[0205] The head mounted device plays back 720 the altered user audio signal to the user (e.g., through one or more speakers), allowing the user to better perceive the current volume of their speaking voice so that they can better adjust their speaking volume.
[0206] Examples of Artificial Reality Systems
[0207] Figure 8 The system environment of the head mounted device including the audio system described above according to one or more embodiments is shown. The system 800 can be operated in an artificial reality environment (eg, virtual reality, augmented reality, mixed reality environment, or some combination thereof). Figure 8 The illustrated system 800 includes a head mounted device 805 coupled to a console 810 and an input / output (I / O) interface 815. The head mounted device 805 may be an embodiment of the head mounted device 110. Figure 8 An example system 800 is shown that includes one head mounted device 805 and one I / O interface 815, but in other embodiments, any number of these components may be included in the system 800. For example, there may be multiple head mounted devices 805, each head mounted device 805 having an associated I / O interface 815, each head mounted device 805 and I / O interface 815 communicating with the console 810. In alternative configurations, different and / or additional components may be included in the system 800. Furthermore, in some embodiments, in combination with Figure 8 The functionality described by one or more components shown in the figure may be combined with Figure 8 The described methods are distributed among the components in different ways. For example, some or all of the functions of the console 810 are provided by the head mounted device 805.
[0208] The head mounted device 805 presents content to the user, which includes a view of a physical, real-world environment augmented with computer-generated elements (e.g., two-dimensional (2D) or three-dimensional (3D) images, 2D or 3D video, sound, etc.). The head mounted device 805 may be an eyewear device or a head mounted display. In some embodiments, the presented content includes audio content (audio signals received from other users of a shared communication channel).
[0209] Head mounted device 805 includes an audio system 820, a sensor system 825, an electronic display 830, and an optical block 835. Audio system 820 may correspond to Figure 3 805, and may include a microphone assembly 225, a transceiver 230, a speaker assembly 330, and a controller 215. The audio system 820 is configured to communicate with the audio systems of other HMDs, capture audio signals corresponding to the voice of the user of the HMD 805, process received audio signals (e.g., audio signals from other HMDs), and play back the processed audio signals to the user.
[0210] The sensor system 825 includes one or more sensor modules, which may include a camera assembly 235, a position sensor 240, and an eye tracking sensor 245. The sensor modules may be used to generate information of a local area around the HMD 805, and to track the position of the HMD 805 and the gaze direction of the user of the HMD 805. In some embodiments, the sensors of the sensor system 825 may be used with a tracking module 855 to track the position of the HMD 805.
[0211] The electronic display 830 and the optical block 835 are one embodiment of the lens 210. Some embodiments of the head mounted device 805 have a Figure 8 In addition, in other embodiments, the components described herein are different from the components described herein. Figure 8 The functionality provided by the various components described may be distributed variously among components of the head mounted device 805 , or captured in separate components remote from the head mounted device 805 .
[0212] The electronic display 830 displays 2D or 3D images to the user based on the data received from the console 810. In various embodiments, the electronic display 830 includes a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of electronic display 830 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof.
[0213] In some embodiments, the optical block 835 amplifies the image light received from the electronic display 830, corrects the optical errors associated with the image light, and presents the corrected image light to the user of the head mounted device 805. In various embodiments, the optical block 835 includes one or more optical elements. Example optical elements included in the optical block 835 include: a waveguide, an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 835 may include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 835 may have one or more coatings, such as a partially reflective or anti-reflective coating.
[0214] The magnification and focusing of the image light by the optical block 835 allows the electronic display 830 to be physically smaller, lighter, and consume less power than a larger display. Additionally, the magnification can increase the field of view of the content presented by the electronic display 830. For example, the field of view of the displayed content is such that the displayed content uses nearly all (e.g., about 110 degrees diagonally), and in some cases, all of the user's field of view to be presented. Additionally, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.
[0215] In some embodiments, the optical block 835 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may further include spherical aberration, chromatic aberration, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display 830 for display is pre-distorted, and the optical block 835 corrects the distortion when it receives image light generated based on the content from the electronic display 830.
[0216] The I / O interface 815 is a device that allows a user to send an action request and receive a response from the console 810. An action request is a request to perform a specific action. For example, an action request can be an instruction to start or end the capture of image or video data, or an instruction to perform a specific action within an application. The I / O interface 815 may include one or more input devices. Example input devices include: a keyboard, a mouse, a hand controller, or any other suitable device for receiving an action request and transmitting the action request to the console 810. The action request received by the I / O interface 815 is transmitted to the console 810, and the console 810 performs an action corresponding to the action request. In some embodiments, the I / O interface 815 includes one or more position sensors that capture calibration data indicating an estimated position of the I / O interface 815 relative to the initial position of the I / O interface 815. In some embodiments, the I / O interface 815 can provide tactile feedback to the user according to the instruction received from the console 810. For example, tactile feedback is provided when an action request is received, or the console 810 transmits instructions to the I / O interface 815, so that the I / O interface 815 generates tactile feedback when the console 810 performs an action. The I / O interface 815 can monitor one or more input responses from the user for determining the perceived source direction and / or perceived source position of the audio content.
[0217] The console 810 provides content to the head mounted device 805 for processing based on information received from one or more of the head mounted device 805 and the I / O interface 815. Figure 8 In the example shown, the console 810 includes an application store 850, a tracking module 855, and an engine 845. Some embodiments of the console 810 have a Figure 8 Similarly, the functions described further below may be implemented in different ways in combination with Figure 8 The described approach is distributed among the components of console 810 .
[0218] The application storage 850 stores one or more applications executed by the console 810. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by the application may be in response to input received from the user via movement of the head mounted device 805 or the I / O interface 815. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications. In some embodiments, the console 810 may be used as an application server (e.g., application server 130), and the application may include an application for maintaining a shared communication channel between a group of users (e.g., users of different HMDs 805).
[0219] The tracking module 855 calibrates the system environment 800 using one or more calibration parameters and may adjust the one or more calibration parameters to reduce errors in the determination of the position of the head mounted device 805 or the I / O interface 815. The calibration performed by the tracking module 855 also takes into account information received from one or more sensor modules (e.g., position sensors) of the sensor system 825 in the head mounted device 805 or one or more sensors included in the I / O interface 815. In addition, if tracking of the head mounted device 805 is lost, the tracking module 855 may recalibrate some or all of the system environment 800.
[0220] The tracking module 855 uses information from a sensor (e.g., the position sensor 240, the camera assembly 235, or some combination thereof) to track the movement of the head mounted device 805 or the I / O interface 815. For example, the tracking module 855 determines the position of a reference point of the head mounted device 805 in a map of the local area based on the information from the head mounted device 805. The tracking module 855 can also use data from the position of the head mounted device 805 or use data from one or more sensors included in the I / O interface 815 indicating the position of the I / O interface 815 to determine the position of the reference point of the head mounted device 805 or the reference point of the I / O interface 815, respectively. In addition, in some embodiments, the tracking module 855 can use a portion of the data indicating the position of the head mounted device 805 to predict a future position of the head mounted device 805. The tracking module 855 provides the estimated or predicted future position of the head mounted device 805 or the I / O interface 815 to the engine 845. In some embodiments, the tracking module 855 may provide tracking information to the audio system 820 for use in determining how to spatialize and / or enhance received audio signals.
[0221] The engine 845 also executes applications within the system environment 800 and receives position information, acceleration information, velocity information, predicted future position, or some combination thereof of the head mounted device 805 from the tracking module 855. Based on the received information, the engine 845 determines the content to be provided to the head mounted device 805 for presentation to the user. For example, if the received information indicates that the user is looking to the left, the engine 845 generates content for the head mounted device 805 that reflects the user's movement in a virtual environment or in an environment that enhances a local area with additional content. In addition, the engine 845 performs an action within an application executed on the console 810 in response to an action request received from the I / O interface 815 and provides feedback to the user that the action has been performed. The feedback provided may be visual or auditory feedback via the head mounted device 805 or tactile feedback via the I / O interface 815.
[0222] Additional configuration information
[0223] The above description of the embodiments of the present disclosure is presented for the purpose of illustration; it is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. It can be appreciated by those skilled in the relevant art that many modifications and variations are possible based on the above disclosure.
[0224] Some parts of this specification sheet describe embodiments of the present disclosure based on algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are often used by technicians in the field of data processing to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally or logically, they should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without loss of generality, it is sometimes proven to be convenient to refer to these operational arrangements as modules. The described operations and their associated modules can be implemented in software, firmware, hardware, or any combination thereof.
[0225] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software module is implemented by a computer program product that includes a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations or processes described.
[0226] Embodiments of the present disclosure may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in this specification may include a single processor or may be an architecture that employs a multi-processor design to increase computing power.
[0227] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information generated from the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.
[0228] Finally, the language used in the specification is selected primarily for readability and teaching purposes, and may not be selected to describe or limit the subject matter of the present invention. Therefore, the scope of the present disclosure is not intended to be limited by this detailed description, but by any claims issued based on the application herein. Therefore, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the present disclosure set forth in the appended claims.
Claims
1. A head mounted device, comprising: a gaze determination circuit configured to determine a gaze direction of the first user of the head mounted device based on a head orientation of the first user and a relative orientation of the first user's eyes with respect to the first user's head; a transceiver configured to receive an audio signal associated with a head mounted device of a second user; A processing circuit, the processing circuit being configured to: determining a relative position associated with the second user relative to the first user; determining a deviation of the second user's position relative to an augmented direction of the first user, wherein the augmented direction is based at least in part on a gaze direction of the first user; spatializing an audio signal associated with the second user based at least in part on the relative position of the head orientation associated with the second user relative to the first user; and enhancing the amplitude of the audio output signal based at least in part on the identified deviation of the second user's position relative to the enhancement direction of the first user; and A speaker assembly is configured to project sound based on the spatialized and enhanced audio output signal such that the projected sound is perceived to originate from a location of the second user.
2. The head-mounted device according to claim 1 further includes a microphone array, the microphone array comprising a plurality of microphones arranged in a plurality of different positions, the microphone array being configured to capture sound in a local area of the first user and generate an audio input signal.
3. The head mounted device according to claim 2, wherein: The processing circuit is further configured to: analyzing the audio input signal to identify sounds originating from a specific area in a local area of the first user; as well as A user audio signal is generated according to the audio input signal by enhancing a portion of the audio input signal corresponding to the sound originating from the specific area.
4. The head mounted device according to claim 3, wherein: The specific area corresponds to the first user's mouth.
5. The head mounted device according to claim 1, wherein: The transceiver is further configured to receive location information of the second user.
6. The head mounted device of claim 1, further comprising an antenna array configured to determine the relative position associated with the second user relative to the first user.
7. The head mounted device according to claim 1, wherein: The processing circuit is further configured to spatialize the audio output signal based on whether a line of sight exists between the first user and the second user.
8. The head mounted device according to claim 1, wherein: The transceiver is further configured to receive a second audio signal from a third user, and the processing circuit is further configured to: identifying a relative position associated with the third user relative to the first user; determining a deviation of the identified relative position of the third user relative to the enhanced direction of the first user; comparing the identified deviation of the relative position of the third user with the identified deviation of the relative position of the second user; as well as Based on a result of the comparing, the amplitude of the second audio signal associated with the third user is enhanced.
9. A method for audio spatialization and enhancement between multiple head mounted devices, comprising: At a head mounted device of a first user, determining an augmentation direction of the first user; receiving, at the first user's head mounted device, an audio signal associated with the second user's head mounted device; Wherein, determining the enhancement direction includes determining the gaze direction of the first user through the following steps: receiving a location of the first user, the location comprising at least a head orientation of the first user; determining a relative orientation of the first user's eyes relative to the first user's head; and determining the gaze direction based on the head orientation of the first user and a relative orientation of the first user's eyes with respect to the first user's head; identifying a relative position associated with the second user relative to the head orientation of the first user; determining a deviation of the identified relative position of the second user relative to the enhanced direction of the first user; spatializing an audio signal associated with the second user based at least in part on the relative position of the head orientation associated with the second user relative to the first user; and enhancing the amplitude of the audio output signal based at least in part on the identified deviation of the second user's position relative to the enhancement direction of the first user; and Sound is projected based on the spatialized and enhanced audio output signal such that the projected sound is perceived to originate from the location of the second user.
10. The method of claim 9, further comprising using a microphone array including a plurality of microphones arranged in a plurality of different locations to capture sound in a local area of the first user and generate an audio input signal.
11. The method according to claim 10, further comprising: analyzing the audio input signal to identify sounds originating from a specific area in the local area of the first user; A user audio signal is generated according to the audio input signal by enhancing a portion of the audio input signal corresponding to the sound originating from the specific area.
12. The method according to claim 11, wherein: The specific area corresponds to the first user's mouth. The method of claim 9 , further comprising receiving location information of the second user.
14. The method of claim 9, further comprising receiving a signal from a head mounted device of the second user at the antenna array and determining a relative position associated with the second user relative to the first user based on the received signal.
15. The method according to claim 9, wherein: Spatializing the audio output signal is based on whether there is a line of sight between the first user and the second user.
16. The method according to claim 9, further comprising: receiving a second audio signal from a third user; identifying a relative position associated with the third user relative to the first user; determining a deviation of the identified relative position of the third user relative to the enhanced direction of the first user; comparing the identified deviation of the relative position of the third user with the identified deviation of the relative position of the second user; as well as Based on a result of the comparing, the amplitude of the second audio signal associated with the third user is enhanced.
17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: At a head mounted device of a first user, determining an augmentation direction of the first user; receiving, at the first user's head mounted device, an audio signal associated with the second user's head mounted device; in, Determining the enhanced direction includes determining the gaze direction of the first user by the following steps: receiving a location of the first user, the location comprising at least a head orientation of the first user; determining a relative orientation of the first user's eyes relative to the first user's head; as well as determining the gaze direction based on the head orientation of the first user and a relative orientation of the first user's eyes with respect to the first user's head; identifying a relative position associated with the second user relative to the head orientation of the first user; determining a deviation of the identified relative position of the second user relative to the enhanced direction of the first user; spatializing an audio signal associated with the second user based at least in part on the relative position of the head orientation associated with the second user relative to the first user; as well as enhancing an amplitude of an audio output signal based at least in part on an identified deviation of the second user's position relative to an enhancement direction of the first user; as well as Sound is projected based on the spatialized and enhanced audio output signal such that the projected sound is perceived to originate from the location of the second user.
