Spatial audio for interactive audio environments

Abstract

To provide a system and method for presenting spatial audio for suitable interactive audio environments.SOLUTION: Provided is an audio mixing system 800 for presenting an output audio signal to a listener located at a first location in a virtual environment. For each sound source of a plurality of sound sources 810 in the virtual environment, a respective first intermediate audio signal corresponding to an input audio signal is determined based on a location of the respective sound source in the virtual environment and the respective first intermediate audio signal is associated with a first bus 840. For each sound source of the plurality of sound sources in the virtual environment, a respective second intermediate audio signal is determined. The respective second intermediate audio signal corresponds to reverberation of the input audio signal in the virtual environment, is determined based on the location of the respective sound source and further based on an acoustic property of the virtual environment, and is associated with a second bus 830. The output audio signal is presented to the listener via the first bus and the second bus.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 62 / 686,655, filed June 18, 2018, the contents of which are incorporated herein by reference in their entirety. This application additionally claims priority to U.S. Provisional Application No. 62 / 686,665, filed June 18, 2018, the contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates generally to spatial audio rendering, and more particularly to spatial audio rendering for virtual sound sources within a virtual acoustic environment. [Background technology]

[0003] Virtual environments are ubiquitous in computing environments, finding use in video games (where a virtual environment may represent a game world), maps (where a virtual environment may represent a terrain to be navigated), simulations (where a virtual environment may simulate a real environment), digital storytelling (where virtual characters may interact with one another within a virtual environment), and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, a user's experience with a virtual environment may be limited by the technology for presenting the virtual environment. For example, traditional displays (e.g., 2D display screens) and audio systems (e.g., fixed speakers) may be unable to realize a virtual environment in a way that creates a compelling, realistic, and immersive experience.

[0004] Virtual reality (“VR”), augmented reality (“AR”), mixed reality (“MR”), and related technologies (collectively, “XR”) share the ability to present users of XR systems with sensory information corresponding to a virtual environment represented by data in a computer system. Such systems can provide a uniquely enhanced sense of immersion and presence by combining virtual visual and audio cues with real sights and sounds. Therefore, it may be desirable to present digital sounds to users of XR systems so that the sounds appear to occur naturally in the user's real environment and consistent with the sounds the user expects. Generally speaking, users expect virtual sounds to take on the acoustic properties of the real environment in which they are heard. For example, a user of an XR system in a large concert hall would expect the virtual sounds of the XR system to have a tonal quality similar to a large cavern; conversely, a user in a small apartment would expect the sounds to be more attenuated, close, and immediate.

[0005] Digital or artificial reverberators can be used in audio and music signal processing to simulate the perceived effect of diffuse acoustic reverberation in a room. In XR environments, it is desirable to use digital reverberators to realistically simulate the acoustic properties of the room within the XR environment. Convincing simulation of such acoustic properties can add a sense of authenticity and immersion to the XR environment. Summary of the Invention [Means for solving the problem]

[0006] A system and method are disclosed for presenting an output audio signal to a listener located at a first location within a virtual environment. According to one method embodiment, an input audio signal is received. For each of a plurality of sound sources within the virtual environment, a respective first intermediate audio signal corresponding to the input audio signal is determined based on the location of the respective sound source within the virtual environment, and the respective first intermediate audio signal is associated with a first bass. For each of the plurality of sound sources within the virtual environment, a respective second intermediate audio signal is determined. The respective second intermediate audio signal corresponds to reverberations of the input audio signal within the virtual environment. The respective second intermediate audio signal is determined based on the location of the respective sound source and further based on acoustic properties of the virtual environment. The respective second intermediate audio signal is associated with a second bass. An output audio signal is presented to the listener via the first bass and the second bass. The present specification also provides, for example, the following items: (Item 1) 1. A method for presenting an output audio signal to a listener located at a first location within a virtual environment, the method comprising: receiving an input audio signal; For each of the plurality of sound sources in the virtual environment, determining a respective first intermediate audio signal corresponding to the input audio signal based on a location of a respective sound source within the virtual environment; associating the respective first intermediate audio signals with a first bus; determining a respective second intermediate audio signal based on the location of the respective sound source and further based on acoustic properties of the virtual environment, the respective second intermediate audio signal corresponding to reverberations of the input audio signal within the virtual environment; associating the separate second intermediate audio signal with a second bus; presenting the output audio signal to the listener via the first bus and the second bus; A method comprising: (Item 2) Item 10. The method of item 1, wherein the acoustic properties of the virtual environment are determined via one or more sensors associated with the listener. (Item 3) Item 3. The method of item 2, wherein the one or more sensors comprise one or more microphones. (Item 4) the one or more sensors are associated with a wearable head unit configured to be worn by the listener; the output signal is presented to the listener via one or more speakers associated with the wearable head unit; The method described in item 2. (Item 5) Item 5. The method of item 4, wherein the wearable head unit comprises a display configured to display a view of the virtual environment to the listener in parallel with the presentation of the output signal. (Item 6) Item 5. The method of item 4, further comprising retrieving the acoustic properties from a database, the acoustic properties including acoustic properties determined via one or more sensors of the wearable head unit. (Item 7) Reading out the acoustic property includes: determining a location of the listener based on an output of the one or more sensors; and identifying the acoustic properties based on the location of the listener; Item 7. The method according to item 6, comprising: (Item 8) A wearable device, a display configured to display a view of the virtual environment; one or more sensors; one or more speakers; one or more processors, receiving an input audio signal; For each of the plurality of sound sources in the virtual environment, determining a respective first intermediate audio signal corresponding to the input audio signal based on a location of a respective sound source within the virtual environment; associating the respective first intermediate audio signals with a first bus; determining a respective second intermediate audio signal based on the location of the respective sound source and further based on acoustic properties of the virtual environment, the respective second intermediate audio signal corresponding to reverberations of the input audio signal within the virtual environment; associating the separate second intermediate audio signal with a second bus; presenting the output audio signal to a listener via the speaker and via the first bus and the second bus; one or more processors configured to perform a method comprising: A wearable device comprising: (Item 9) Item 9. The wearable device of item 8, wherein the acoustic properties of the virtual environment are determined via the one or more sensors. (Item 10) Item 9. The wearable device of item 8, wherein the one or more sensors comprise one or more microphones. (Item 11) 9. The method of claim 8, further comprising displaying a view of the virtual environment via the display in parallel with the presentation of the output signal. (Item 12) 9. The method of claim 8, further comprising retrieving the acoustic properties from a database, the acoustic properties including acoustic properties determined via one or more sensors. (Item 13) Reading out the acoustic property includes: determining a location of the listener based on an output of the one or more sensors; and identifying the acoustic properties based on the location of the listener; Item 13. The method according to item 12, comprising: (Item 14) For each of the plurality of sound sources in the virtual environment, determining a respective third intermediate audio signal based on the location of the respective sound source and further based on a second acoustic property of the virtual environment, the respective third intermediate audio signal corresponding to a reflection of the input audio signal off a surface of the virtual environment; associating the separate third intermediate audio signal with a second bus; further comprising the second bus comprises a reflection bus and a reverberation bus; associating the respective second intermediate audio signals with the second bus includes associating the respective second intermediate audio signals with the reverberant bus; associating the respective third intermediate audio signals with the second bus includes associating the respective third intermediate audio signals with the reflection bus. The method according to item 1. (Item 15) The method further comprises: For each of the plurality of sound sources in the virtual environment, determining a respective third intermediate audio signal based on the location of the respective sound source and further based on a second acoustic property of the virtual environment, the respective third intermediate audio signal corresponding to a reflection of the input audio signal off a surface of the virtual environment; associating the separate third intermediate audio signal with the second bus; Including, the second bus comprises a reflection bus and a reverberation bus; associating the respective second intermediate audio signals with the second bus includes associating the respective second intermediate audio signals with the reverberant bus; associating the respective third intermediate audio signals with the second bus includes associating the respective third intermediate audio signals with the reflection bus. Item 9. The wearable device according to item 8. (Item 16) Item 10. The method of claim 1, wherein determining the individual first intermediate audio signal includes applying a first individual filter to the input audio signal, the first individual filter comprising one or more of a source directivity model, a distance model, and an orientation model. (Item 17) Item 17. The method of item 16, wherein determining the individual first intermediate audio signals further comprises applying one or more of an individual gain and an individual panning process to the input audio signals. (Item 18) Item 18. The method of item 17, wherein the individual panning process includes panning the input audio signal based on the geometry of a loudspeaker array. (Item 19) Item 10. The method of item 1, wherein determining the individual second intermediate audio signal comprises applying a second individual filter to the input audio signal, the second individual filter comprising a source directivity model. (Item 20) 20. The method of claim 19, wherein determining the respective second intermediate audio signals further comprises applying one or more of a respective delay, a respective gain, and a respective reverberation process to the input audio signal. (Item 21) Item 21. The method of item 20, wherein applying the individual reverberation process includes applying one or more of a reverberation delay and a reverberation level. (Item 22) 21. The method of claim 20, wherein the individual reverberation processes are frequency dependent. (Item 23) 21. The method of claim 20, wherein the individual reverberation processes are position independent. (Item 24) 22. The method of claim 21, wherein applying the reverberation delay comprises applying one or more of a per-source delay and a global delay. (Item 25) 21. The method of claim 20, wherein applying the reverberation delay includes applying gain normalization. (Item 26) 9. The wearable device of claim 8, wherein determining the individual first intermediate audio signal includes applying a first individual filter to the input audio signal, the first individual filter comprising one or more of a sound source directivity model, a distance model, and an orientation model. (Item 27) 27. The wearable device of claim 26, wherein determining the individual first intermediate audio signal further includes applying one or more of an individual gain and an individual panning process to the input audio signal. (Item 28) Item 28. The wearable device of item 27, wherein the individual panning process includes panning the input audio signal based on the geometry of a loudspeaker array. (Item 29) 9. The wearable device of claim 8, wherein determining the individual second intermediate audio signal includes applying a second individual filter to the input audio signal, the second individual filter comprising a sound source directivity model. (Item 30) 30. The wearable device of item 29, wherein determining the individual second intermediate audio signal further includes applying one or more of an individual delay, an individual gain, and an individual reverberation process to the input audio signal. (Item 31) Item 31. The wearable device of item 30, wherein applying the individual reverberation process includes applying one or more of a reverberation delay and a reverberation level. (Item 32) Item 31. The wearable device of item 30, wherein the individual reverberation processes are frequency dependent. (Item 33) Item 31. The wearable device of item 30, wherein the individual reverberation processes are position-independent. (Item 34) Item 32. The wearable device of item 31, wherein applying the reverberation delay includes applying one or more of a per-source delay and a global delay. (Item 35) Item 31. The wearable device of item 30, wherein applying the reverberation delay includes applying gain normalization. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an exemplary wearable system, according to some embodiments.

[0008] [Figure 2] FIG. 2 illustrates an example handheld controller that may be used with an example wearable system, according to some embodiments.

[0009] [Figure 3] FIG. 3 illustrates an example auxiliary unit that may be used in conjunction with an example wearable system, according to some embodiments.

[0010] [Figure 4] FIG. 4 illustrates an example functional block diagram for an example wearable system, according to some embodiments.

[0011] [Figure 5] FIG. 5 illustrates an example geometric room representation, according to some embodiments.

[0012] [Figure 6] FIG. 6 illustrates an exemplary model of room response measured from a source in a room to a listener, according to some embodiments.

[0013] [Figure 7] FIG. 7 illustrates example factors that affect a user's perception of direct sound, reflections, and reverberations, according to some embodiments.

[0014] [Figure 8] FIG. 8 illustrates an exemplary audio mixing architecture for rendering multiple virtual sound sources in a virtual room, according to some embodiments.

[0015] [Figure 9] FIG. 9 illustrates an exemplary audio mixing architecture for rendering multiple virtual sound sources in a virtual room, according to some embodiments.

[0016] [Figure 10] FIG. 10 illustrates an exemplary per-source processing module, according to some embodiments.

[0017] [Figure 11] FIG. 11 illustrates an exemplary per-source reflect pan module, according to some embodiments.

[0018] [Figure 12] FIG. 12 illustrates an exemplary room processing algorithm, according to some embodiments.

[0019] [Figure 13] FIG. 13 illustrates an exemplary reflection module, according to some embodiments.

[0020] [Figure 14] FIG. 14 illustrates an example spatial distribution of apparent arrival directions of reflections, according to some embodiments.

[0021] [Figure 15] FIG. 15 illustrates an example of direct gain, reflected gain, and reverberant gain as a function of distance, according to some embodiments.

[0022] [Figure 16] FIG. 16 illustrates an example relationship between distance and spatial focus, according to some embodiments.

[0023] [Figure 17] FIG. 17 illustrates an example relationship between time and signal amplitude, according to some embodiments.

[0024] [Figure 18] FIG. 18 illustrates an exemplary system for processing spatial audio, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0025] In the following description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific embodiments which may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

[0026] Exemplary Wearable System

[0027] 1 illustrates an exemplary wearable head device 100 configured to be worn on a user's head. Wearable head device 100 may be part of a broader wearable system that includes one or more components, such as a head device (e.g., wearable head device 100), a handheld controller (e.g., handheld controller 200 described below), and / or an auxiliary unit (e.g., auxiliary unit 300 described below). In some examples, wearable head device 100 can be used for virtual reality, augmented reality, or mixed reality systems or applications. The wearable head device 100 includes one or more displays, such as displays 110A and 110B (which may comprise left and right transmissive displays and associated components for coupling light from the displays to the user's eyes, such as orthogonal pupil expansion (OPE) grating sets 112A / 112B and exit pupil expansion (EPE) grating sets 114A / 114B), left and right acoustic structures, such as speakers 120A and 120B (which may be mounted on temple arms 122A and 122B, respectively, and positioned adjacent the user's left and right ears), and an infrared sensor. The wearable head device 100 may include one or more sensors, such as a microphone, an accelerometer, a GPS unit, an inertial measurement unit (IMU) (e.g., IMU 126), an acoustic sensor (e.g., microphone 150), a quadrature coil electromagnetic receiver (e.g., receiver 127 shown mounted on left temple arm 122A), left and right cameras (e.g., depth (time-of-flight) cameras 130A and 130B) oriented away from the user, and left and right eye cameras (e.g., for detecting the user's eye movements) (e.g., eye cameras 128 and 128B) oriented toward the user. However, the wearable head device 100 may incorporate any suitable display technology and any suitable number, type, or combination of sensors or other components without departing from the scope of the invention.In some examples, wearable head device 100 may incorporate one or more microphones 150 configured to detect audio signals generated by the user's voice, and such microphones may be positioned within the wearable head device adjacent to the user's mouth. In some examples, wearable head device 100 may incorporate networking features (e.g., Wi-Fi capabilities) for communicating with other devices and systems, including other wearable systems. Wearable head device 100 may further include components such as a battery, a processor, memory, a storage unit, or various input devices (e.g., buttons, touchpad), or may be coupled to a handheld controller (e.g., handheld controller 200) or auxiliary unit (e.g., auxiliary unit 300) that comprises one or more such components. In some examples, sensors may be configured to output a set of coordinates of the head-mounted unit relative to the user's environment and may provide input to a processor to implement a simultaneous localization and mapping (SLAM) procedure and / or a visual odometry algorithm. In some embodiments, the wearable head device 100 may be coupled to a handheld controller 200 and / or an auxiliary unit 300, as described further below.

[0028] 2 illustrates an exemplary mobile handheld controller component 200 of an exemplary wearable system. In some examples, handheld controller 200 may communicate wired or wirelessly with wearable head device 100 and / or auxiliary unit 300, described below. In some examples, handheld controller 200 includes a handle portion 220 to be held by a user and one or more buttons 240 disposed along a top surface 210. In some examples, handheld controller 200 may be configured for use as an optical tracking target; for example, a sensor (e.g., a camera or other optical sensor) of wearable head device 100 can be configured to detect the position and / or orientation of handheld controller 200, which in turn may indicate the position and / or orientation of a user's hand holding handheld controller 200. In some examples, handheld controller 200 may include a processor, memory, a storage unit, a display, or one or more input devices, such as those described above. In some examples, the handheld controller 200 includes one or more sensors (e.g., any of the sensors or tracking components described above with respect to the wearable head device 100). In some examples, the sensors can detect the position or orientation of the handheld controller 200 relative to the wearable head device 100 or relative to another component of the wearable system. In some examples, the sensors may be positioned within the handle portion 220 of the handheld controller 200 and / or may be mechanically coupled to the handheld controller. The handheld controller 200 can be configured to provide one or more output signals corresponding, for example, to the press state of the button 240 or the position, orientation, and / or movement of the handheld controller 200 (e.g., via an IMU). Such output signals may be used as inputs to a processor of the wearable head device 100, to the auxiliary unit 300, or to another component of the wearable system.In some embodiments, the handheld controller 200 may include one or more microphones to detect sounds (e.g., a user's speech, environmental sounds) and, in some cases, provide signals corresponding to the detected sounds to a processor (e.g., a processor of the wearable head device 100).

[0029] 3 illustrates an exemplary auxiliary unit 300 of an exemplary wearable system. In some examples, the auxiliary unit 300 may communicate wired or wirelessly with the wearable head device 100 and / or the handheld controller 200. The auxiliary unit 300 may include a battery to provide energy for operating one or more components of the wearable system, such as the wearable head device 100 and / or the handheld controller 200 (including a display, sensors, an acoustic structure, a processor, a microphone, and / or other components of the wearable head device 100 or the handheld controller 200). In some examples, the auxiliary unit 300 may include a processor, memory, a storage unit, a display, one or more input devices, and / or one or more sensors, such as those described above. In some examples, the auxiliary unit 300 includes a clip 310 for attaching the auxiliary unit to a user (e.g., to a belt worn by the user). An advantage of using auxiliary unit 300 to store one or more components of a wearable system is that doing so may allow large or heavy components to be carried on the user's waist, chest, or back, which are relatively better suited to supporting large, heavy objects, rather than being mounted on the user's head (e.g., when stored in wearable head device 100) or carried by the user's hands (e.g., when stored in handheld controller 200). This may be particularly advantageous with respect to relatively heavy or bulky components, such as batteries.

[0030] 4 shows an example functional block diagram that may correspond to an example wearable system 400, such as may include the example wearable head device 100, handheld controller 200, and auxiliary unit 300 described above. In some examples, the wearable system 400 may be used for virtual reality, augmented reality, or mixed reality applications. As shown in FIG. 4 , the wearable system 400 may include an example handheld controller 400B, referred to herein as a “totem” (and which may correspond to the handheld controller 200 described above), which may include a totem / headgear six-degree-of-freedom (6DOF) totem subsystem 404A. The wearable system 400 may also include an example wearable head device 400A (which may correspond to the wearable headgear device 100 described above), which includes a totem / headgear 6DOF headgear subsystem 404B. In some embodiments, the 6DOF totem subsystem 404A and the 6DOF headgear subsystem 404B cooperate to determine six coordinates (e.g., offsets in three translational directions and rotations along three axes) of the handheld controller 400B relative to the wearable head device 400A. The six degrees of freedom may be expressed relative to the coordinate system of the wearable head device 400A. The three translational offsets may be expressed as X, Y, and Z offsets within such a coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom may be expressed as a sequence of yaw, pitch, and roll rotations, a vector, a rotation matrix, a quaternion, or some other representation. In some embodiments, one or more depth cameras 444 (and / or one or more non-depth cameras) and / or one or more optical targets (e.g., buttons 240 of the handheld controller 200 as described above or dedicated optical targets included in the handheld controller) included within the wearable head device 400A can be used for 6DOF tracking.In some embodiments, the handheld controller 400B can include a camera as described above, and the headgear 400A can include optical targets for optical tracking in conjunction with the camera. In some embodiments, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids used to wirelessly transmit and receive three distinguishable signals. By measuring the relative magnitudes of the three distinguishable signals received in each of the coils used for receiving, the 6DOF of the handheld controller 400B relative to the wearable head device 400A can be determined. In some embodiments, the 6DOF totem subsystem 404A can include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding high-speed movement of the handheld controller 400B.

[0031] In some examples involving augmented reality or mixed reality applications, it may be desirable to transform coordinates from a local coordinate space (e.g., a coordinate space that is fixed relative to the wearable head device 400A) to an inertial coordinate space or to an environmental coordinate space. For example, such a transformation may be necessary for the display of the wearable head device 400A to present virtual objects in an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the wearable head device 400A), rather than in a fixed position and orientation on the display (e.g., at the same position on the display of the wearable head device 400A). This can maintain the illusion that the virtual objects exist in the real environment (and do not appear unnaturally positioned in the real environment, e.g., as the wearable head device 400A shifts and rotates). In some examples, a compensatory transformation between coordinate spaces can be determined by processing images from the depth camera 444 (e.g., using simultaneous localization and mapping (SLAM) and / or visual odometry procedures) to determine a transformation of the wearable head device 400A relative to an inertial or environmental coordinate system. In the example shown in FIG. 4 , the depth camera 444 can be coupled to the SLAM / visual odometry block 406 and can provide images to the block 406. The SLAM / visual odometry block 406 implementation can include a processor configured to process the images and then determine the position and orientation of the user's head, which can be used to identify a transformation between the head coordinate space and the real coordinate space. Similarly, in some examples, an additional source of information regarding the user's head pose and location is obtained from the IMU 409 of the wearable head device 400A. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information regarding rapid adjustments of the user's head pose and position.

[0032] In some examples, depth camera 444 can provide 3D images to hand gesture tracker 411, which can be implemented within a processor of wearable head device 400A. Hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D images received from depth camera 444 to stored patterns representing hand gestures. Other suitable techniques for identifying the user's hand gestures will also be apparent.

[0033] In some embodiments, one or more processors 416 may be configured to receive data from the headgear subsystem 404B, the IMU 409, the SLAM / visual odometry block 406, the depth camera 444, a microphone (not shown), and / or the hand gesture tracker 411. The processor 416 may also send and receive control signals to and from the 6DOF totem system 404A. The processor 416 may be wirelessly coupled to the 6DOF totem system 404A, such as in embodiments in which the handheld controller 400B is untethered. The processor 416 may further communicate with additional components, such as an audiovisual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source of imagewise modulated light 424 and a right channel output coupled to a right source of imagewise modulated light 426. The GPU 420 may output stereoscopic image data to the sources of imagewise modulated light 424, 426. The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 416 indicating a direction vector from the user to a virtual sound source (which may be moved by the user, e.g., via the handheld controller 400B). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing an HRTF or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the believability and realism of virtual sounds by incorporating the user's relative position and orientation to the virtual sounds in the mixed reality environment, i.e., by presenting virtual sounds that match the user's expectations of what they would hear if the virtual sounds were real sounds in a real environment.

[0034] 4 , one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included within auxiliary unit 400C (which may correspond to auxiliary unit 300 described above). Auxiliary unit 400C may include battery 427 to power its components and / or provide power to wearable head device 400A and / or handheld controller 400B. Including such components within an auxiliary unit, which may be mounted on the user's waist, can limit the size and weight of wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0035] While FIG. 4 presents elements corresponding to various components of exemplary wearable system 400, various other suitable arrangements of these components will be apparent to those skilled in the art. For example, elements shown in FIG. 4 as associated with auxiliary unit 400C may instead be associated with wearable head device 400A or handheld controller 400B. Furthermore, some wearable systems may dispense with handheld controller 400B or auxiliary unit 400C entirely. Such variations and modifications are understood to be within the scope of the disclosed embodiments.

[0036] Mixed Reality Environment

[0037] Like all people, users of mixed reality systems exist in a real environment, i.e., the three-dimensional portion of the "real world" and all of its contents that are perceptible to the user. For example, users perceive the real environment using their normal human senses, i.e., sight, hearing, touch, taste, and smell, and interact with the real environment by moving their body within the real environment. Locations within the real environment can be described as coordinates within a coordinate space; for example, coordinates can include latitude, longitude, and altitude relative to sea level, distance in three orthogonal dimensions from a reference point, or other suitable values. Similarly, vectors can describe quantities that have direction and magnitude within the coordinate space.

[0038] A computing device can maintain a representation of a virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computer representation of a three-dimensional space. A virtual environment can include representations of any objects, actions, signals, parameters, coordinates, vectors, or other properties associated with that space. In some examples, the circuitry (e.g., a processor) of a computing device can maintain and update the state of the virtual environment; i.e., the processor can determine the state of the virtual environment at a second time based on data associated with the virtual environment and / or input provided by a user at a first time. For example, if an object in the virtual environment is located at a first coordinate and has certain programmed physical parameters (e.g., mass, coefficient of friction) at a certain time, and input received from a user indicates that a force should be applied to the object in a certain directional vector, the processor can apply the laws of kinematics and use basic mechanics to determine the location of the object at that time. The processor can use any suitable known information about the virtual environment and / or any suitable input to determine the state of the virtual environment at a certain time. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to creating and deleting virtual objects within the virtual environment, software (e.g., scripts) for defining the behavior of virtual objects or characters within the virtual environment, software for defining the behavior of signals (e.g., audio signals) within the virtual environment, software for creating and updating parameters associated with the virtual environment, software for generating audio signals within the virtual environment, software for handling input and output, software for implementing network operations, software for applying asset data (e.g., animation data for moving virtual objects over time), or many other possibilities.

[0039] Output devices, such as a display or speakers, can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, lights, etc.) that can be presented to the user. A processor can determine a view of the virtual environment (e.g., corresponding to a "camera" with origin coordinates, viewing axis, and frustum) and render on the display a viewable scene of the virtual environment corresponding to that view. Any suitable rendering technique may be used for this purpose. In some examples, the viewable scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, the virtual environment may include audio aspects that can be presented to the user as one or more audio signals. For example, virtual objects in the virtual environment may generate sounds originating from the object's location coordinates (e.g., a virtual character may speak or trigger a sound effect), or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a particular location. A processor can determine audio signals corresponding to "listener" coordinates, e.g., audio signals that correspond to a composite of sounds in the virtual environment and that are mixed and processed to simulate audio signals that would be heard by a listener at the listener coordinates, and present the audio signals to the user via one or more speakers.

[0040] Because the virtual environment exists only as a computer construct, the user cannot directly perceive the virtual environment using their normal senses. Instead, the user can only indirectly perceive the virtual environment as presented to the user, for example, by a display, speakers, haptic output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment, but can provide input data via input devices or sensors to a processor, which can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data indicating that the user is attempting to move an object in the virtual environment, and the processor can use that data to cause the object to respond accordingly in the virtual environment.

[0041] Reflections and reverberations

[0042] Aspects of a listener's audio experience within a space (e.g., a room) of a virtual environment include the listener's perception of direct sound, the listener's perception of reflections of that direct sound off room surfaces, and the listener's perception of reverberations of the direct sound within the room ("reverb"). FIG. 5 illustrates a geometric room representation 500, according to some embodiments. The geometric room representation 500 shows example propagation paths for direct sound (502), reflections (504), and reverberations (506). These paths represent the paths an audio signal may take within a room from a source to a listener. The room shown in FIG. 5 may be any suitable type of environment associated with one or more acoustic properties. For example, the room 500 may be a concert hall and may include a stage with a piano player and an auditorium section with an audience. As shown, direct sound is sound that originates at a source (e.g., the piano player) and travels directly toward the listener (e.g., the audience). A reflection is sound that originates at a source, bounces off a surface (e.g., the walls of a room), and travels to the listener. Echoes are sound that contain fading signals, including many reflections that arrive close together at a given time.

[0043] FIG. 6 illustrates an example model 600 of room response measured from a source in a room to a listener, according to some embodiments. The model of room response shows the amplitude of the direct sound (610), reflections of the direct sound (620), and reverberations of the direct sound (630) from the perspective of a listener at a distance from the direct sound source. As illustrated in FIG. 6, the direct sound generally arrives at the listener before the reflections (with the illustrated reflection delay (622) showing the difference in time between the direct sound and the reflections), which in turn arrive before the reverberations (with the illustrated reverberation delay (632) showing the difference in time between the direct sound and the reverberations). The reflections and reverberations may be perceptually different to the listener. Reflections can be modeled separately from reverberations, for example, to better control the time, attenuation, spectral shape, and arrival direction of individual reflections. Reflections may be modeled using a reflection model, and reverberations may be modeled using a reverberation model that may be different from the reflection model.

[0044] The reverberation characteristics (e.g., reverberation decay) for the same sound source may differ between two different acoustic environments (e.g., rooms) for the same sound source, and it is desirable to realistically reproduce the sound source according to the characteristics of the current room in the listener's virtual environment. That is, when a virtual sound source is presented in a mixed reality system, the reflection and reverberation characteristics of the listener's real environment should be accurately reproduced. L. Savioja, J. Huopaniemi, T. Lokki, and R. Vaananen, "Creating Interactive Virtual Acoustic Environments," J. Audio Eng. Soc. 47(9): 675-705 (1999), describes a method for reproducing direct path, individual reflections, and acoustic reverberations in a real-time virtual 3D audio reproduction system for video games, simulations, or AR / VR. In the method disclosed by Savioja et al., the direction of arrival, delay, amplitude, and spectral equalization of each individual reflection are derived from a geometric and physical model of the room (e.g., a real room, a virtual room, or some combination thereof), which may require a complex rendering system. These methods may be computationally complex and perhaps prohibitively complex for mobile applications where computing resources may be limited.

[0045] In some room acoustics simulation algorithms, reverberation can be implemented by downmixing all sound sources to a mono signal and sending the mono signal to a reverberation simulation module. The gain used for downmixing and sending can depend on dynamic parameters, such as source distance, and manual parameters, such as reverberation gain.

[0046] Source directivity or radiation pattern may refer to a measure of the amount of energy a sound source is emitting in different directions. Source directivity affects all parts of the room impulse response (e.g., direct, reflected, and reverberant). Different sound sources may exhibit different directivities; for example, human speech may have a different directional pattern than a trumpet performance. A room simulation model may take source directivity into account when generating an accurate simulation of an acoustic signal. For example, a model incorporating source directivity may include a function of the direction of a line from the source to the listener relative to the front direction (or principal acoustic axis) of the source. The directional pattern is axisymmetric about the principal acoustic axis of the source. In some embodiments, a parametric gain model may be defined using a frequency-dependent filter. In some embodiments, to determine how much audio from a given source should be transmitted into the reverberant bath, the average diffuse power of the source may be calculated (e.g., by integrating over a sphere centered at the acoustic center of the source).

[0047] Two-way audio engines and sound design tools may make assumptions about the acoustic system to be modeled. For example, some two-way audio engines may model sound source directivity as a function independent of frequency, which may have two potential drawbacks. First, this may ignore the frequency-dependent attenuation for direct sound propagation from the sound source to the listener. Second, this may ignore the frequency-dependent attenuation for reflected and reverberant transmissions. These effects may be important from a psychoacoustic perspective, and not reproducing them may lead to a room simulation that is perceived as unnatural and different from what listeners are accustomed to experiencing in a real acoustic environment.

[0048] In some cases, a room simulation system or two-way audio engine may not completely separate sound sources, listeners, and acoustic environment parameters such as reflections and reverberations. Instead, the room simulation system may be tuned as a whole for a specific virtual environment and may not adapt to different playback scenarios. For example, reverberations within the simulated environment may not match the environment in which the user / listener is physically present when listening to the rendered content.

[0049] In augmented or mixed reality applications, computer-generated audio objects may be rendered through an acoustically transparent playback system to blend with the physical environment as naturally heard by the user / listener. This may require binaural artificial reverberation processing to match the local environmental acoustics, and thus the synthetic audio objects may not be distinguishable from naturally occurring or reproduced sounds over loudspeakers. For example, approaches involving measuring or calculating room impulse responses based on estimating the geometry of the environment may be limited in consumer environments by practical obstacles and complexity. In addition, physical models may not necessarily provide the most compelling listening experience because they do not consider psychoacoustic acoustic principles or provide suitable audio scene parameterization for sound designers to fine-tune the listening experience.

[0050] Matching some specific physical properties of a target acoustic environment may not provide a simulation that perceptually closely matches the listener's environment or the application designer's intent. A perceptually relevant model of the target acoustic environment that can be characterized using a practical audio environment description interface may be desired.

[0051] For example, a rendering model that separates the contributions of sources, listeners, and room properties may be desirable. A rendering model that separates contributions may allow components to be adapted or swapped at runtime according to the local environment and properties of the end user. For example, a listener may be in a physical room with different acoustic characteristics than the virtual environment in which the content was originally created. Modifying the early reflections and / or reverberations of the simulation to match the listening environment may lead to a more convincing listening experience. Matching the listening environment may be particularly important in mixed reality applications, where the desired effect may be for the listener to be unable to distinguish between sounds that are simulated around them and sounds that exist in the real surrounding environment.

[0052] It may be desirable to create convincing effects without requiring detailed knowledge of the geometry of the real surroundings and / or the acoustic properties of the surrounding surfaces, which may not be available or may be complex to estimate, especially on a portable device. Instead, models based on perceptual and psychoacoustic principles may be a much more practical tool for characterizing acoustic environments.

[0053] FIG. 7 illustrates a table 700 containing several objective acoustic and geometric parameters characterizing each section in a binaural room impulse model that distinguishes between source, listener, and room properties, according to some embodiments. Some source properties, including free-field and diffuse-field transfer functions, may be independent of how and where content will be rendered, while other properties, including position and orientation, may need to be dynamically updated during playback. Similarly, some listener properties, including free-field and diffuse-field head-related transfer functions or diffuse-field interaural coherence (IACC), may be independent of where content will be rendered, while other properties, including position and orientation, may be dynamically updated during playback. Some room properties, particularly those that contribute to late reverberations, may be entirely environment-dependent. Representations of reverberation decay rates and room cube volumes may allow the spatial audio rendering system to adapt to the listener's playback environment.

[0054] The source and the listener's ear may be modeled as emitting and receiving transducers, respectively, characterized by a set of directionally dependent free-field transfer functions, including the listener's head-related transfer function (HRTF).

[0055] FIG. 8 illustrates an exemplary audio mixing system 800 for rendering multiple virtual sound sources in a virtual room, such as in an XR environment, according to some embodiments. For example, the audio mixing architecture may include a rendering engine for room acoustics simulation of multiple virtual sound sources 810 (i.e., objects 1-N). The system 800 comprises a room send bus 830 that feeds a module 850 that renders reflections and reverberations (e.g., a shared reverberation and reflection module). Aspects of this general process are described, for example, in IA-SIG 3D Audio Rendering Guidelines (Level 2), www.iasig.net (1999). The room send bus combines contributions from all sources, e.g., sound sources 810, each processed by a corresponding module 820, and derives an input signal for the room module. The room send bus may comprise a mono room send bus. The format of the main mix bus 840 may be a two-channel or multi-channel format that matches the final output rendering method, which may include, for example, a binaural renderer for headphone playback, an Ambisonic decoder, and / or a multi-channel loudspeaker system. The main mix bus combines contributions from all sources with the room module outputs to derive the output rendering signal 860.

[0056] With reference to the exemplary system 800, each of the N objects may represent a virtual source signal and may be assigned an apparent location in the environment, such as by a panning algorithm. For example, each object may be assigned an angular position on a sphere centered on the virtual listener's position. The panning algorithm may calculate each object's contribution to each channel of the main mix. This general process is described, for example, in J.-M. Jot, V. Larcher, and J.-M. Pernaux, "A comparative study of 3-D audio encoding and rendering techniques," Proc. AES 16th International Conference on Spatial Sound Reproduction (1999). Each object may be input to a pan and gain module 820, which may implement a panning algorithm and perform additional signal processing, such as adjusting gain levels for each object.

[0057] In some embodiments, system 800 (e.g., via module 820) may assign each virtual sound source an apparent distance relative to the virtual listener's position, from which the rendering engine may derive a per-source direct gain and a per-source room gain for each object. The direct and room gains may affect the audio signal power contributed by the virtual sound source to the main mix bus 840 and the room send bus 830, respectively. A minimum distance parameter may be assigned to each virtual sound source, and the direct gain and room gain may roll off at different rates as distance increases beyond this minimum distance.

[0058] In some embodiments, the system 800 of FIG. 8 may be used for audio recording and two-way audio production targeted at conventional two-channel front stereo loudspeaker playback systems. However, when applied in binaural or immersive 3D audio systems that allow for simulated reverberation and a spatially diffuse distribution of reflections, the system 800 may not provide sufficiently convincing sound localization cues when rendering virtual sound sources, especially those far from the listener. This can be addressed by the inclusion of a clustered reflection rendering module shared among the virtual sound sources 810 while supporting per-source control of the spatial distribution of reflections. It is desirable for such a module to incorporate per-source early reflection processing algorithms and dynamic control of early reflection parameters according to virtual sound source and listener position.

[0059] In some embodiments, it may be desirable to have a spatial audio processing model / system and method that can accurately reproduce position-dependent room acoustic cues without computationally complex rendering of individual early reflections for each virtual sound source or detailed descriptions of acoustic reflector geometry and physics.

[0060] The reflection processing model may dynamically consider the position of the listener and virtual sound sources in a real or virtual room / environment without associated physical and geometric descriptions. Perceptual models for per-source clustered reflection panning and control of early reflection processing parameters may be efficiently implemented.

[0061] 9 illustrates an audio mixing system 900 for rendering multiple virtual sound sources in a virtual room, according to some embodiments. For example, system 900 may include a rendering engine for room acoustics simulation of multiple virtual sound sources 910 (e.g., objects 1-N). Compared to system 800 described above, system 900 may include separate control of reverberant and reflected transmit channels for each virtual sound source. Each object may be input to a separate per-source processing module 920, and a room transmit bus 930 may feed a room processing module 950.

[0062] 10 illustrates a per-source processing module 1020 according to some embodiments. Module 1020 may correspond to one or more of modules 920 shown in FIG. 9 and in exemplary system 900. Per-source processing module 1020 may perform processing specific to an individual source (e.g., 1010, which may correspond to one of sources 910) of the overall system (e.g., system 900). Per-source processing modules may include a direct processing path (e.g., 1030A) and / or a room processing path (e.g., 1030B).

[0063] In some embodiments, individual direct and room filters may be applied separately for each sound source. Applying filters separately may allow for more refined and accurate control over how each source radiates sound toward the listener and into the surrounding environment. The use of filters, as opposed to broadband gain, may allow for matching a desired sound radiation pattern as a function of frequency. This is beneficial because radiation properties may vary across sound source types and may be frequency-dependent. The angle between the main acoustic axis of the sound source and the listener's position may affect the sound pressure level perceived by the listener. Furthermore, source radiation characteristics may affect the average diffuse power of the source.

[0064] In some embodiments, the frequency-dependent filter may be implemented using the double shelving approach disclosed in U.S. Patent Application No. 62 / 678,259, entitled "INDEX SCHEMING FOR FILTER PARAMETERS," the contents of which are incorporated by reference in their entirety. In some embodiments, the frequency-dependent filter may be applied in the frequency domain and / or using a finite impulse response filter.

[0065] As shown in the example, the direct processing path may include a direct transmit filter 1040 followed by a direct pan module 1044. The direct transmit filter 1040 may model one or more acoustic effects, such as one or more of sound source directivity, distance, and / or orientation. The direct pan module 1044 may spatialize the audio signal to correspond to an apparent position within the environment (e.g., a 3D location within a virtual environment, such as an XR environment). The direct pan module 1044 may be amplitude and / or intensity based and may depend on the geometry of the loudspeaker array. In some embodiments, the direct processing path may include a direct transmit gain 1042 along with the direct transmit filter and direct pan module. The direct pan module 1044 may output to a main mix bus 1090, which may correspond to the main mix bus 940 described above with respect to exemplary system 900.

[0066] In some embodiments, the room processing path comprises a room delay 1050 and a room transmit filter 1052, followed by a reflection path (e.g., 1060A) and a reverberation path (e.g., 1060B). The room transmit filter may be used to model the effect of source directionality on signals traveling in the reflection and reverberation paths. The reflection path may comprise a reflection transmit gain 1070 and may send its signal to a reflection transmit bus 1074 via a reflection pan module 1072. The reflection pan module 1072 may be similar to the direct pan module 1044 in that it may spatialize the audio signal but may operate on reflections instead of the direct signal. The reverberation path 1060B may comprise a reverberation gain 1080 and may send its signal to a reverberation transmit bus 1084. The reflection transmit bus 1074 and the reverberation transmit bus 1084 may be grouped into a room transmit bus 1092, which may correspond to the room transmit bus 930 described above with respect to exemplary system 900.

[0067] 11 illustrates an example of a per-source reflection pan module 1100, which may correspond to the reflection pan module 1072 described above, according to some embodiments. As shown in the figure, the input signal may be encoded into a three-channel Ambisonic B-format signal, as described, for example, in J.-M. Jot, V. Larcher, and J.-M. Pernaux, "A comparative study of 3-D audio encoding and rendering techniques," Proc. AES 16th International Conference on Spatial Sound Reproduction (1999). The encoding coefficients 1110 may be calculated according to Equations 1-3. [ka]

[0068] In Equations 1-3, k is [ka] where F is a spatial focus parameter with a value between [0, 2 / 3] and Az is an angle in degrees between [0, 360]. The encoder may encode the input signal into a three-channel Ambisonic B-format signal.

[0069] Az may be the azimuth angle defined by the projection of the primary direction of arrival of the reflection onto a head-relative horizontal plane (e.g., the plane perpendicular to the "up" vector of the listener's head and containing the listener's ears). The spatial focus parameter F may indicate the spatial concentration of the reflected signal energy arriving at the listener. When F is zero, the spatial distribution of the reflected energy arrival may be uniform around the listener. As F increases, the spatial distribution may become increasingly concentrated around the primary direction determined by the azimuth angle Az. The maximum theoretical value of F is 1.0, which may indicate that all energy is arriving from the primary direction determined by the azimuth angle Az.

[0070] In one embodiment of the present invention, the spatial focus parameter F may be defined as the magnitude of the Garzon energy vector, as described, for example, in J.-M. Jot, V. Larcher, and J.-M. Pernaux, "A comparative study of 3-D audio encoding and rendering techniques," Proc. AES 16th International Conference on Spatial Sound Reproduction (1999).

[0071] The output of the reflect pan module 1100 may be provided to a reflect transmit bus 1174, which may correspond to the reflect transmit bus 1074 described above with respect to FIG.

[0072] 12 illustrates an exemplary room processing module 1200, according to some embodiments. The room processing module 1200 may correspond to the room processing module 950 described above with respect to FIG. 9 and the exemplary system 900. As shown in FIG. 9, the room processing module 1200 may include a reflection processing path 1210A and / or a reverberation processing path 1210B.

[0073] The reflection processing path 1210A may receive signals from the reflection transmit bus 1202 (which may correspond to the reflection transmit bus 1074 described above) and output signals into the main mix bus 1290 (which may correspond to the main mix bus 940 described above). The reflection processing path 1210A may include a reflection global gain 1220, a reflection global delay 1222, and / or a reflection module 1224 that may simulate / render reflections.

[0074] The reverberation processing path 1210B may receive a signal from the reverberation transmit bus 1204 (which may correspond to the reverberation transmit bus 1084 described above) and output the signal into the main mix bus 1290. The reverberation processing path 1210B may include a reverberation global gain 1230, a reverberation global delay 1232, and / or a reverberation module 1234.

[0075] FIG. 13 illustrates an exemplary reflection module 1300, according to some embodiments. The reflection module input 1310 may be output by a reflection pan module 1100, such as that described above, and presented to the reflection module 1300 via a reflection transmit bus 1174. The reflection transmit bus may carry a three-channel Ambisonic B-format signal combining contributions from all virtual sound sources (e.g., sound source 910 (objects 1-N) described above with respect to FIG. 9). In the example shown, three channels, denoted (W, X, Y), are fed to an Ambisonic decoder 1320. According to the example, the Ambisonic decoder generates six output signals, which feed six mono input / output elementary reflection modules 1330 (R1-R6), respectively, generating a set of six reflection output signals 1340 (s1-s6). (The example shows six signal and reflection modules, but any suitable number may be used.) The reflected output signal 1340 is presented to a main mixing bus 1350, which may correspond to the main mixing bus 940 described above.

[0076] 14 illustrates a spatial distribution 1400 of apparent directions of arrival of reflections, according to some embodiments, as detected by a listener 1402. For example, the reflections shown may be those generated by the reflection module 1300 described above for a sound source that is assigned particular values ​​of the reflection pan parameters Az and F, e.g., as described above with respect to FIG.

[0077] 14, the effect of the reflection module 1300 in combination with the reflection pan module 1100 is to generate a series of reflections, each of which may arrive at a different time (e.g., as shown in model 600) from each of the virtual loudspeaker directions 1410 (e.g., 1411-1416, which may correspond to the reflected output signals s1-s6 described above). The effect of the reflection pan module of 1100 in combination with the Ambisonic decoder 1320 is to adjust the relative magnitudes of the reflected output signals 1340 to create for the listener the sensation that the reflections are emanating from a primary direction angle Az with a spatial distribution (e.g., more or less concentrated around that primary direction) determined by the setting of the spatial focus parameter F.

[0078] In some embodiments, the reflection principal direction angle Az corresponds, for each source, to the apparent direction of arrival of the direct path, which can be controlled for each source by the direct pan module 1020. The simulated reflections may enhance the perception of the directional location of the virtual sound source as perceived by the listener.

[0079] In some embodiments, the main mix bus 940 and direct pan module 1020 may enable three-dimensional reconstruction of the sound direction. In these embodiments, the reflection principal direction angle Az may correspond to the projection of the apparent direction onto the plane from which the reflection principal angle Az is measured.

[0080] FIG. 15 illustrates a model 1500 of exemplary direct gain, reflected gain, and reverberant gain as a function of distance (e.g., to a listener) according to some embodiments. Model 1500 illustrates an example of the variation of direct, reflected, and reverberant transmit gain, for example, as shown in FIG. 10, with respect to source distance. As shown in the figure, there are cases where the direct sound, its reflection, and its reverberation can have significantly different falloff curves with distance. In some cases, per-source processing such as that described above may enable achieving a faster distance-based rolloff for reflections than for reverberations. Psychoacoustically, this may enable robust directionality and distance perception, especially for distant sources.

[0081] 16 illustrates an exemplary model 1600 of spatial focus versus source distance for direct and reflected components, according to some embodiments. In this example, the direct pan module 1020 is configured to generate maximum spatial concentration of direct path components in the direction of the sound source, regardless of its distance. Meanwhile, the reflected spatial focus parameter F may be set to an exemplary value of 2 / 3 to enhance directional perception in a realistic manner for all distances longer than a limit distance (e.g., reflection minimum distance 1610). As illustrated by the exemplary model 1600, the reflected spatial focus parameter value decreases toward zero as the source approaches the listener.

[0082] 17 shows an example model 1700 of the amplitude of an audio signal as a function of time. As described above, a reflection processing path (e.g., 1210A) may receive a signal from a reflection transmit bus and output a signal on a main mix bus. The reflection processing path may include a reflection global gain (e.g., 1220), a reflection global delay (e.g., 1222) for controlling the parameter Der as shown in model 1700, and / or a reflection module (e.g., 1224), such as those described above.

[0083] As described above, a reverberation processing path (e.g., 1210B) may receive a signal from the reverberation transmit bus and output a signal into the main mix bus. The reverberation processing path 1210B may include a reverberation global gain (e.g., 1230) for controlling the parameter Lgo as shown in model 1700, a reverberation global delay (e.g., 1232) for controlling the parameter Drev as shown in model 1700, and / or a reverberation module (e.g., 1234). The processing blocks within the reverberation processing path may be implemented in any suitable order. Examples of reverberation modules are described in U.S. Patent Application No. 62 / 685,235, entitled "REVERBERATION GAIN NORMALIZATION," and U.S. Patent Application No. 62 / 684,086, entitled "LOW-FREQUENCY INTERCHANNEL COHERENCE CONTROL," the contents of each of which are incorporated herein by reference in their entirety.

[0084] Model 1700 in Figure 17 illustrates how per-source parameters, including distance and reverberation delay, can be taken into account to dynamically adjust reverberation delay and level according to some embodiments. In the figure, Dtof represents the delay due to time of flight for a given object, i.e., Dtof = ObjDist / c, where ObjDist is the object distance from the center of the listener's head and c is the speed of sound in air. Drm represents the room delay per object. Dobj represents the total per-object delay, i.e., Dobj = Dtof + Drm. Der represents the global early reflection delay. Drev represents the global reverberation delay. Dtotal represents the total delay for a given object, i.e., Dtotal = Dobj + Dglobal.

[0085] Lref represents the level of reverberation relative to Dtotal=0. Lgo represents the global level offset due to global delay, which may be calculated according to Equation 10, and T60 is the reverberation time of the reverberation algorithm. Loo represents the level offset per object due to global delay, which may be calculated according to Equation 11. Lto represents the total level offset for a given object, which may be calculated according to Equation 12 (assuming dB values). [ka]

[0086] In some embodiments, the reverberation level is calibrated independently of object position, reverberation time, and other user-controllable parameters. Thus, Lrev can be the extrapolated level of decaying reverberation at the initial time of sound emission. Lrev is the "REVERBERATION" parameter. Lrev may be the same quantity as the reverberation initial power (RIP) defined in U.S. patent application Ser. No. 62 / 685,235, entitled "Gain Normalization," the contents of which are incorporated herein by reference in their entirety. Lrev may be calculated according to Equation 13. [ka]

[0087] In some embodiments, T60 may be a function of frequency, and therefore Lgo, Loo, and consequently Lto, are frequency dependent.

[0088] FIG. 18 illustrates an example system 1800 for determining spatial audio properties based on an acoustic environment. The example system 1800 can be used to determine spatial audio properties related to reflections and / or reverberations, such as those described above. As examples, such properties may include room volume, reverberation time as a function of frequency, the listener's position relative to the room, the presence of objects (e.g., sound-attenuating objects) in the room, surface materials, or other suitable properties. In some examples, these spatial audio properties may be retrieved locally by capturing a single impulse response with a microphone and loudspeaker freely positioned in the local environment, or may be adaptively derived by continuously monitoring and analyzing sound captured by a mobile device microphone. In some examples, such as when the acoustic environment can be sensed via sensors in an XR system (e.g., an augmented reality system including one or more of the wearable head unit 100, handheld controller 200, and auxiliary unit 300 described above), the user's location can be used to present audio reflections and reverberations corresponding to the environment presented to the user (e.g., via a display).

[0089] In exemplary system 1800, acoustic environment sensing module 1810 identifies spatial audio properties of the acoustic environment, such as those described above. In some examples, acoustic environment sensing module 1810 may capture data corresponding to the acoustic environment (stage 1812). For example, the data captured in stage 1812 may include audio data from one or more microphones, camera data from a camera, such as an RGB camera or a depth camera, LIDAR data, sonar data, radar data, GPS data, or other suitable data that may convey information about the acoustic environment. In some instances, the data captured in stage 1812 may include data related to the user, such as the user's position or orientation relative to the acoustic environment. The data captured in stage 1812 may be captured via one or more sensors of a wearable device, such as wearable head unit 100, described above.

[0090] In some embodiments, the local environment in which the head-mounted display device resides may include one or more microphones. In some embodiments, one or more microphones may be employed, mounted on the mobile device, positioned in the environment, or both. Benefits of such an arrangement may include gathering directional information about room reverberations or mitigating poor signal quality of any one microphone within the one or more microphones. Signal quality may be poor on a given microphone due to, for example, occlusion, overload, wind noise, transducer damage, and the like.

[0091] In step 1814 of module 1810, features may be extracted from the data captured in step 1812. For example, room dimensions may be determined from sensor data such as camera data, LIDAR data, sonar data, etc. The features extracted in step 1814 may be used to determine one or more acoustic properties of the room, e.g., frequency-dependent reverberation time, which may be stored in step 1816 and associated with the current acoustic environment.

[0092] In some examples, module 1810 can communicate with database 1840 to store and retrieve acoustic properties related to the acoustic environment. In some embodiments, the database may be stored locally on the device's memory. In some embodiments, the database may be stored online as a cloud-based service. The database may assign geographic locations to room properties for easy access at a later time based on the listener's location. In some embodiments, the database may contain additional information to identify the listener's location and / or determine reverberant properties within the database that are close approximations of the listener's environmental properties. For example, room properties may be categorized by room type, so that a set of parameters can be used as soon as a listener is identified as being in a room of a known type (e.g., a bedroom or living room) even if the absolute geographic location cannot be known.

[0093] Storing echo properties in a database may be related to U.S. Patent Application No. 62 / 573,448, entitled "PERSISTENT WORLD MODEL SUPPORTING AUGMENTED REALITY AND INCLUDING AUDIO COMPONENT," the contents of which are incorporated herein by reference in their entirety.

[0094] In some examples, system 1800 may include a reflection adaptation module 1820 to retrieve acoustic properties related to a room and apply those properties to audio reflections (e.g., audio reflections presented to a user of wearable head unit 100 via headphones or via speakers). At stage 1822, the user's current acoustic environment may be determined. For example, GPS data may indicate the user's location in GPS coordinates, which may in turn indicate the user's current acoustic environment (e.g., the room located at those GPS coordinates). As another example, camera data combined with optical recognition software may be used to identify the user's current environment. Reflection adaptation module 1820 may then communicate with database 1840 to retrieve acoustic properties associated with the determined environment, which may be used at stage 1824 to update the audio rendering accordingly. That is, acoustic properties associated with the reflections (e.g., a directional pattern or falloff curve such as those described above) may be applied to the reflected audio signals presented to the user such that the presented reflected audio signals incorporate those acoustic properties.

[0095] Similarly, in some embodiments, system 1800 may include a reflection adaptation module 1830 to retrieve acoustic properties related to the room and apply those properties to audio reverberations (e.g., audio reflections presented to a user of wearable head unit 100 via headphones or via speakers). The acoustic properties of interest for reverberations may be different from those of interest for reflections (e.g., in table 700 for FIG. 7 ), such as those described above. In stage 1832, the user's current acoustic environment may be determined, as described above. For example, GPS data may indicate the user's location within GPS coordinates, which may in turn indicate the user's current acoustic environment (e.g., the room located at those GPS coordinates). As another example, camera data combined with optical recognition software may be used to identify the user's current environment. The reflection adaptation module 1830 may then communicate with database 1840 to retrieve acoustic properties associated with the determined environment, which may be used in stage 1824 to update the audio rendering accordingly. That is, acoustic properties related to reverberation (e.g., reverberation decay times such as those described above) can be applied to the reverberant audio signal presented to the user such that the presented reverberant audio signal incorporates those acoustic properties.

[0096] With respect to the systems and methods described above, elements of the systems and methods can be implemented by one or more computer processors (e.g., CPUs or DSPs), as appropriate. The present disclosure is not limited to any particular configuration of computer hardware, including computer processors, used to implement these elements. In some cases, multiple computer systems can be employed to implement the systems and methods described above. For example, a first computer processor (e.g., a processor of a wearable device coupled to a microphone) can be utilized to receive input microphone signals and perform initial processing of those signals (e.g., signal conditioning and / or segmentation, such as those described above). A second (perhaps more computationally powerful) processor can then be utilized to perform more computationally intensive processing, such as determining probability values ​​associated with speech segments in those signals. Another computer device, such as a cloud server, can host a speech recognition engine, to which the input signals are ultimately provided. Other suitable configurations will become apparent and are within the scope of the present disclosure.

[0097] Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

1. A method, wherein the method is Based on the location of the sound source within the virtual environment, a first intermediate audio signal corresponding to the input audio signal is determined, Associating the first intermediate audio signal with the first bus, The process involves determining a second intermediate audio signal based on the location of the sound source and further based on the acoustic properties of the virtual environment, wherein the second intermediate audio signal corresponds to the reverberation of the input audio signal within the virtual environment. Associating the second intermediate audio signal with the second bus, A third intermediate audio signal is determined based on the location of the sound source and further based on the second acoustic properties of the virtual environment, wherein the third intermediate audio signal corresponds to the reflection of the input audio signal on the surface of the virtual environment. Associating the third intermediate audio signal with the second bus, The output audio signal is determined based on the first bus and the second bus, To present the aforementioned output audio signal to the listener. Includes, The second bus comprises a reflective bus and an echo bus, Associating the second intermediate audio signal with the second bus includes associating the second intermediate audio signal with the reverberation bus, A method comprising associating the third intermediate audio signal with the second bus, wherein the third intermediate audio signal is associated with the reflection bus.

2. The method according to claim 1, wherein the acoustic properties of the virtual environment are determined via one or more sensors associated with the listener.

3. The method according to claim 2, wherein the one or more sensors comprises one or more microphones.

4. The one or more sensors are associated with a wearable head unit configured to be worn by the hearing person, The method according to claim 2, wherein the output audio signal is presented to the listener via one or more speakers associated with the wearable head unit.

5. The method according to claim 4, wherein the wearable head unit comprises a display configured to display a view of the virtual environment to the listener in parallel with the presentation of the output audio signal.

6. Further comprising reading the acoustic properties from a database, The method according to claim 2, wherein the acoustic properties include acoustic properties determined via one or more sensors.

7. Reading out the acoustic properties is The location of the listener is determined based on the output of one or more of the sensors, Identifying the acoustic properties based on the location of the listener. The method according to claim 6, including the method described in claim 6.

8. The method according to claim 1, wherein determining the first intermediate audio signal comprises applying a first filter to the input audio signal, the first filter comprising one or more of a sound source directivity model, a distance model, and an orientation model.

9. The method according to claim 8, wherein determining the second intermediate audio signal comprises applying a second filter to the input audio signal, the filter comprising one or more of a sound source directivity model, a distance model, and an orientation model.

10. The method according to claim 1, wherein determining the first intermediate audio signal comprises applying one or more of the gain and panning processes to the input audio signal.

11. The method according to claim 10, wherein the panning process comprises panning the input audio signal based on the geometric shape of the loudspeaker array.

12. The method according to claim 1, wherein determining the second intermediate audio signal comprises applying one or more of the delay, gain, and reverberation processes to the input audio signal.

13. The method according to claim 12, wherein applying the reverberation process includes applying one or more of the reverberation delay and reverberation level.

14. The method according to claim 12, wherein the reverberation process is frequency-dependent.

15. The method according to claim 12, wherein the reverberation process is position-independent.

16. The method according to claim 13, wherein applying the echo delay includes applying one or more of the source-specific delay and the global delay.

17. A device, wherein the device is One or more speakers, One or more processors configured to perform the method and Equipped with, The aforementioned method, Based on the location of the sound source within the virtual environment, a first intermediate audio signal corresponding to the input audio signal is determined, Associating the first intermediate audio signal with the first bus, The process involves determining a second intermediate audio signal based on the location of the sound source and further based on the acoustic properties of the virtual environment, wherein the second intermediate audio signal corresponds to the reverberation of the input audio signal within the virtual environment. Associating the second intermediate audio signal with the second bus, A third intermediate audio signal is determined based on the location of the sound source and further based on the second acoustic properties of the virtual environment, wherein the third intermediate audio signal corresponds to the reflection of the input audio signal on the surface of the virtual environment. Associating the third intermediate audio signal with the second bus, The output audio signal is determined based on the first bus and the second bus, The output audio signal is presented to the listener via the speaker. Includes, The second bus comprises a reflective bus and an echo bus, Associating the second intermediate audio signal with the second bus includes associating the second intermediate audio signal with the reverberation bus, A device in which associating the third intermediate audio signal with the second bus includes associating the third intermediate audio signal with the reflection bus.

18. The device according to claim 17, wherein the acoustic properties of the virtual environment are determined via one or more sensors of the device.

19. The device according to claim 18, wherein the one or more sensors comprises one or more microphones.

20. The device according to claim 17, further comprising the method of displaying a view of the virtual environment via a head-mounted display of the device in parallel with the presentation of the output audio signal.

21. The method further includes: This includes reading the aforementioned acoustic properties from a database, The device according to claim 17, wherein the acoustic properties include acoustic properties determined via one or more sensors of the device.

22. Reading out the acoustic properties means that The location of the listener is determined based on the output of one or more of the sensors, Identifying the acoustic properties based on the location of the listener. The device according to claim 21, including the device described in claim 21.

23. The device according to claim 17, wherein determining the first intermediate audio signal comprises applying a first filter to the input audio signal, the first filter comprising one or more of a sound source directivity model, a distance model, and an orientation model.

24. The device according to claim 23, wherein determining the second intermediate audio signal comprises applying a second filter to the input audio signal, the second filter comprising one or more of a sound source directivity model, a distance model, and an orientation model.

25. The device according to claim 17, wherein determining the first intermediate audio signal comprises applying one or more of the gain and panning processes to the input audio signal.

26. The device according to claim 25, wherein the panning process comprises panning the input audio signal based on the geometric shape of the loudspeaker array.

27. ​​The device according to claim 17, wherein determining the second intermediate audio signal comprises applying one or more of the delay, gain, and reverberation processes to the input audio signal.

28. The device according to claim 27, wherein applying the reverberation process includes applying one or more of the reverberation delay and reverberation level.

29. The device according to claim 27, wherein the reverberation process is frequency-dependent.

30. The device according to claim 27, wherein the reverberation process is position-independent.

31. The device according to claim 28, wherein applying the echo delay includes applying one or more of the source-specific delay and the global delay.

32. A non-temporary computer-readable medium storing instructions, wherein, when an instruction is executed by one or more processors, the one or more processors cause the one or more processors to execute a method. The aforementioned method, Based on the location of the sound source within the virtual environment, a first intermediate audio signal corresponding to the input audio signal is determined, Associating the first intermediate audio signal with the first bus, The process involves determining a second intermediate audio signal based on the location of the sound source and further based on the acoustic properties of the virtual environment, wherein the second intermediate audio signal corresponds to the reverberation of the input audio signal within the virtual environment. Associating the second intermediate audio signal with the second bus, A third intermediate audio signal is determined based on the location of the sound source and further based on the second acoustic properties of the virtual environment, wherein the third intermediate audio signal corresponds to the reflection of the input audio signal on the surface of the virtual environment. Associating the third intermediate audio signal with the second bus, The output audio signal is determined based on the first bus and the second bus, To present the aforementioned output audio signal to the listener. Includes, The second bus comprises a reflective bus and an echo bus, Associating the second intermediate audio signal with the second bus includes associating the second intermediate audio signal with the reverberation bus, Associating the third intermediate audio signal with the second bus includes associating the third intermediate audio signal with the reflection bus, in a non-transient computer-readable medium.

33. The non-temporary computer-readable medium according to claim 32, wherein the acoustic properties of the virtual environment are determined via one or more sensors associated with the listener.

34. The non-temporary computer-readable medium according to claim 33, wherein the one or more sensors comprises one or more microphones.

35. The one or more sensors are associated with a wearable head unit configured to be worn by the hearing person, The output audio signal is presented to the listener via one or more speakers associated with the wearable head unit in a non-transient, computer-readable medium according to claim 33.

36. The non-temporary computer-readable medium according to claim 35, wherein the wearable head unit comprises a display configured to display a view of the virtual environment to the listener in parallel with the presentation of the output audio signal.

37. The method further includes reading the acoustic properties from a database, The non-transient computer-readable medium according to claim 33, wherein the acoustic properties include acoustic properties determined via one or more sensors.

38. Reading out the acoustic properties means The location of the listener is determined based on the output of one or more of the sensors, Identifying the acoustic properties based on the location of the listener. A non-temporary computer-readable medium according to claim 37, including the above.

39. The non-transient computer-readable medium according to claim 32, wherein determining the first intermediate audio signal comprises applying a first filter to the input audio signal, the first filter comprising one or more of a sound source directivity model, a distance model, and an orientation model.

40. The non-transient computer-readable medium according to claim 39, wherein determining the second intermediate audio signal comprises applying a second filter to the input audio signal, the filter comprising one or more of a sound source directivity model, a distance model, and an orientation model.

41. The non-transient computer-readable medium according to claim 32, wherein determining the first intermediate audio signal comprises applying one or more of the gain and panning processes to the input audio signal.

42. The non-transient computer-readable medium according to claim 41, wherein the panning process comprises panning the input audio signal based on the geometric shape of the loudspeaker array.

43. The non-transient computer-readable medium according to claim 32, wherein determining the second intermediate audio signal comprises applying one or more of the delay, gain, and reverberation processes to the input audio signal.

44. The non-transient computer-readable medium according to claim 43, wherein applying the reverberation process includes applying one or more of the reverberation delay and reverberation level.

45. The non-transient computer-readable medium according to claim 43, wherein the reverberation process is frequency-dependent.

46. The non-transient computer-readable medium according to claim 43, wherein the reverberation process is position-independent.

47. The non-transient computer-readable medium according to claim 44, wherein applying the echo delay includes applying one or more of the source delay and the global delay.

Images