Index scheming in regard to filter parameter

Abstract

To provide a system and a method, for a capture, a processing, and a reproducing of an audio signal.SOLUTION: The present invention provides a method for processing an audio signal. According to an embodiment of the present method, prototype filter size response information is determined. The size response information contains a plurality of gain values, and at least one of them contains a first gain corresponded to a first frequency. The size response information of the prototype filter is stored. The size response information of the prototype filter in the first frequency is read out. Each gain is calculated in regard to a plurality of control frequencies on the basis of the size response information of the prototype filter in the first frequency in regard to the plurality of control frequencies. The gain to be calculated is adopted to the audio signal.SELECTED DRAWING: None

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 678,259, filed May 30, 2018, which is incorporated by reference in its entirety.

[0002] The present disclosure relates generally to systems and methods for capturing, processing and playing audio signals, and more particularly to systems and methods for capturing, processing and playing audio signals for presentation to a user in a virtual or augmented reality system. [Background technology]

[0003] Virtual environments are ubiquitous in computing environments, finding use in video games (where a virtual environment may represent a game world), maps (where a virtual environment may represent a terrain to be navigated), simulations (where a virtual environment may simulate a real environment), digital storytelling (where virtual characters may interact with one another within a virtual environment), and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, a user's experience with a virtual environment may be limited by the technology for presenting the virtual environment. For example, conventional displays (e.g., 2D display screens) and audio systems (e.g., fixed speakers) may be incapable of realizing a virtual environment in a way that creates a compelling, realistic, and immersive experience.

[0004] Virtual reality ("VR"), augmented reality ("AR"), mixed reality ("MR"), and related technologies (collectively, "XR") share the ability to present to a user of an XR system sensory information corresponding to a virtual environment represented by data in a computer system. Such systems can provide a uniquely enhanced sense of immersion and presence by combining virtual visual and audio cues with real sights and sounds. It may therefore be desirable to present digital sounds to a user of an XR system so that the sounds appear to occur naturally in the user's real environment and consistent with the sounds the user expects. Generally speaking, users expect virtual sounds to take on the acoustic properties of the real environment in which they are heard. For example, a user of an XR system in a large concert hall would expect the virtual sounds of the XR system to have a tonal quality similar to a large cavern, and conversely, a user in a small apartment would expect the sounds to be more attenuated, close, and immediate. In addition, users expect the virtual sounds to be presented without delay.

[0005] To meet these expectations, audio signals may need to be processed for precise magnitude response control. One exemplary mechanism used for audio signal processing is the Proportional Parametric Equalizer (PPE). The PPE can provide continuous control over the parameters of the audio signal and over the frequency content of the audio signal. The PPE can be an efficient tool for precise magnitude response control within defined constraints. More specifically, a cascade of shelving filters can be used to create a multi-band (e.g., three-band) parametric equalizer or tone control with little processing overhead. However, significant computing cycles and resources may be required to continuously control such filters in an environment as dynamic as AR or dynamic spatialized audio capture.

[0006] One method for determining the magnitude response of a prototype filter may be to apply the filter to a test signal and measure the output signal. Such an approach may be prohibitive in terms of computing resources. Another method may be to pre-calculate the response of the filter and store it, for example, in a look-up table. At runtime, data corresponding to frequencies of interest can be fetched from a storage device. Although fetching information from a storage device may require very low computing costs, such costs add computer overhead each time new filter data is needed. Therefore, a magnitude response control for a filter signal with increased efficiency is desired. Summary of the Invention [Means for solving the problem]

[0007] A system and method for processing an audio signal using a cascade of shelving filters to create a three-band parametric equalizer is disclosed. In some embodiments, gain values ​​derived from prototype filter parameters can be measured, and then a lookup table that stores known gain values ​​for the designated filter can be used. The lookup table is accessed by a computing device, such as a head-mounted AR display device. The magnitude response of this designated or prototype filter is also stored in the lookup table. The magnitude response is retrieved and then applied and interpolated as necessary for the particular combination of control frequencies used by the user.

[0008] In some embodiments, an indexing scheme for a lookup table is used. The indexing scheme allows for retrieval of filter data without the need to search for the frequency of interest. The indexing scheme can be based on a prototype filter and its associated measured gain value. In some examples, the magnitude response of the filter may be required to calculate the filter parameters. An approximate response can be derived from the magnitude response of the corresponding prototype filter. The response of the prototype filter can then be modified to match the desired filter parameters. Data for the prototype control frequency is indexed in the lookup table, and different values ​​of the control frequency are offset and easy to retrieve. The present invention provides, for example, the following items. (Item 1) 1. A method for processing an audio signal, the method comprising: determining magnitude response information of a prototype filter, the magnitude response information including a plurality of gain values, at least one of the plurality of gain values ​​including a first gain corresponding to a first frequency; storing magnitude response information of the prototype filter; Retrieving magnitude response information of the prototype filter at the first frequency; and calculating gains for a plurality of control frequencies based on the retrieved magnitude response information of the prototype filter at the first frequency; applying the calculated gain to the audio signal; A method comprising: (Item 2) Determining the magnitude response information of the prototype filter comprises: determining a magnitude response information of a high shelving equalizer; scaling the magnitude response information of the high shelving equalizer; shifting the scaled magnitude response information of the high shelving equalizer along a frequency axis by a predetermined frequency amount, the shifted scaled magnitude response being the magnitude response information of the prototype filter; The method according to item 1, comprising: (Item 3) 3. The method of claim 2, wherein the predetermined frequency amount is equal to an amount required to match the scaled magnitude response information of the high shelving equalizer to the magnitude response information of the prototype filter. (Item 4) Determining the magnitude response information of the prototype filter comprises: determining magnitude response information of a low shelving equalizer; scaling the magnitude response information of the low shelving equalizer; shifting the scaled magnitude response information of the low shelving equalizer along a frequency axis by a predetermined frequency amount, the shifted scaled magnitude response being the magnitude response information of the prototype filter; The method according to item 1, comprising: (Item 5) 2. The method of claim 1, wherein determining the magnitude response information of the prototype filter includes inverting the magnitude response of a low shelving equalizer along a frequency axis. (Item 6) Item 1. The method of item 1, wherein determining the magnitude response information of the prototype filter includes inverting the magnitude response of a high shelving equalizer along a frequency axis. (Item 7) 2. The method of claim 1, wherein storing the magnitude response information of the prototype filter comprises storing the magnitude response information of the prototype filter in a lookup table. (Item 8) 8. The method of claim 7, wherein the lookup table contains equally spaced entries of magnitude response information of the prototype filter. (Item 9) 8. The method of claim 7, wherein the first frequency is close to DC or Nyquist, and reading out magnitude response information of the prototype filter at the first frequency includes setting the magnitude response information equal to a saturation value. (Item 10) Determining one or more properties of the environment; and determining one or more parameters based on the determined one or more properties of the environment; and Further comprising: 2. The method of claim 1, wherein the determined one or more parameters are used to retrieve magnitude response information of the prototype filter at the first frequency. (Item 11) Determining one or more characteristics of the environment includes: generating a test audio signal; determining a response to the generated test audio signal; determining one or more characteristics of the environment based on a variation between the test audio signal and the response; and 11. The method according to item 10, comprising: (Item 12) Determining one or more actions of a user of a wearable head device; determining one or more parameters, and reading out magnitude response information of the prototype filter at the first frequency is based on the determined one or more actions; Further comprising: 2. The method of claim 1, wherein the determined one or more parameters are used to retrieve magnitude response information of the prototype filter at the first frequency. (Item 13) Retrieving the magnitude response information of the prototype filter determining an index associated with the first frequency; using said determined index to find a corresponding index in a look-up table for retrieval of said magnitude response information; The method according to item 1, comprising: (Item 14) 1. A system comprising: a wearable head device configured to provide an audio signal to a user; 1. A circuit comprising: determining magnitude response information of a prototype filter, the magnitude response information including a plurality of gain values, at least one of the plurality of gain values ​​including a first gain corresponding to a first frequency; Retrieving magnitude response information of the prototype filter at the first frequency; and calculating gains for a plurality of control frequencies based on the retrieved magnitude response information of the prototype filter at the first frequency; processing the audio signal by applying the calculated gain to the audio signal; transmitting the processed audio signal to the wearable head device; A circuit configured to: A system comprising: (Item 15) Item 15. The system of item 14, further comprising a memory that stores magnitude response information of the prototype filter. (Item 16) Item 16. The system of item 15, wherein the magnitude response information of the prototype filter is stored in a lookup table that includes equally spaced entries of the magnitude response information of the prototype filter. (Item 17) Item 16. The system of item 15, wherein the magnitude response information of the prototype filter is stored in a lookup table, the lookup table including a plurality of indexes, each index associated with a plurality of frequencies. (Item 18) the wearable head device comprises one or more sensors; The system is configured to determine one or more properties of an environment; the circuitry is further configured to determine one or more parameters based on the determined one or more properties of the environment; retrieving the magnitude response information of the prototype filter at the first frequency based on the determined one or more parameters. Item 15. The system according to item 14. (Item 19) The wearable head device comprises: one or more speakers configured to generate a test audio signal; one or more sensors configured to determine a response to the generated test audio signal; Equipped with 20. The system of claim 18, wherein one or more characteristics of the environment are determined by the circuitry based on a change between the test audio signal and the response. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an exemplary wearable system, according to some embodiments.

[0010] [Diagram 2] FIG. 2 illustrates an example handheld controller that may be used with an example wearable system, according to some embodiments.

[0011] [Diagram 3] FIG. 3 illustrates an example auxiliary unit that may be used with an example wearable system, according to some embodiments.

[0012] [Figure 4] FIG. 4 illustrates an example functional block diagram for an example wearable system, according to some embodiments.

[0013] [Diagram 5] FIG. 5 illustrates an example process that may be performed by an XR system, according to some embodiments.

[0014] [Figure 6] FIG. 6 illustrates the magnitude response of an example low shelving prototype filter in accordance with some embodiments.

[0015] [Figure 7A] FIG. 7A illustrates the measured magnitude response of an exemplary low shelving equalizer, in accordance with some embodiments.

[0016] [Figure 7B] FIG. 7B illustrates an approximated magnitude response of an exemplary low shelving equalizer, according to some embodiments.

[0017] [Figure 7C] FIG. 7C illustrates FIG. 7A overlaid on FIG. 7B, according to some embodiments.

[0018] [Figure 8A] FIG. 8A illustrates the measured magnitude response of an exemplary high shelving equalizer in accordance with some embodiments.

[0019] [Figure 8B] FIG. 8B illustrates an approximated magnitude response of an exemplary high shelving equalizer in accordance with some embodiments.

[0020] [Figure 8C] FIG. 8C illustrates FIG. 8A overlaid on FIG. 8B, according to some embodiments.

[0021] [Figure 9A] FIG. 9A illustrates an example lookup table including frequencies and associated gain values ​​for prototype filters, according to some embodiments.

[0022] [Figure 9B]FIG. 9B illustrates an example lookup table including indexes and associated frequency and gain values ​​according to some embodiments.

[0023] [Figure 9C] FIG. 9C illustrates an example lookup table including half number indexes according to some embodiments.

[0024] [Figure 10] FIG. 10 shows example magnitude responses as a function of frequency for a low shelving equalizer, a high shelving equalizer, and a dual shelving equalizer in accordance with some embodiments.

[0025] [Figure 11] 11A and 11B illustrate an example magnitude response and an example phase response, respectively, of a dual shelving equalizer having a target magnitude response that decreases monotonically with frequency in accordance with some embodiments.

[0026] [Figure 12] 12A and 12B illustrate an example magnitude response and an example phase response, respectively, of a dual shelving equalizer having a target magnitude response that is not monotonically decreasing with frequency in accordance with some embodiments.

[0027] [Figure 13] 13A and 13B illustrate an example magnitude response and an example phase response, respectively, of a dual shelving equalizer operated outside its operating range in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] In the following description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific embodiments which may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

[0029] US Patent Application No. 15 / 907,155 is incorporated herein by reference in its entirety.

[0030] Exemplary Wearable System

[0031] 1 illustrates an exemplary wearable head device 100 configured to be worn on a user's head. The wearable head device 100 may be part of a broader wearable system that includes one or more components, such as a head device (e.g., wearable head device 100), a handheld controller (e.g., handheld controller 200 described below), and / or an auxiliary unit (e.g., auxiliary unit 300 described below). In some examples, the wearable head device 100 can be used for virtual reality, augmented reality, or mixed reality systems or applications. The wearable head device 100 includes one or more displays, such as displays 110A and 110B (which may comprise left and right transmissive displays and associated components for coupling light from the displays to the user's eyes, such as orthogonal pupil expansion (OPE) grating set 112A / 112B and exit pupil expansion (EPE) grating set 114A / 114B), left and right acoustic structures, such as speakers 120A and 120B (which may be mounted on temple arms 122A and 122B, respectively, and positioned adjacent the user's left and right ears), and an infrared sensor. The wearable head device 100 may include one or more sensors, such as a GPS unit, an accelerometer, a GPS unit, an inertial measurement unit (IMU) (e.g., IMU 126), an acoustic sensor (e.g., microphone 150), a quadrature coil electromagnetic receiver (e.g., receiver 127 shown mounted on left temple arm 122A), left and right cameras oriented away from the user (e.g., depth (time of flight) cameras 130A and 130B), and left and right eye cameras oriented towards the user (e.g., for detecting the user's eye movements) (e.g., eye cameras 128 and 128B). However, the wearable head device 100 may incorporate any suitable display technology and any suitable number, type, or combination of sensors or other components without departing from the scope of the invention.In some embodiments, the wearable head device 100 may incorporate one or more microphones 150 configured to detect audio signals generated by the user's voice, and such microphones may be positioned within the wearable head device adjacent to the user's mouth. In some embodiments, the wearable head device 100 may incorporate networking features (e.g., Wi-Fi capabilities) for communicating with other devices and systems, including other wearable systems. The wearable head device 100 may further include components such as a battery, a processor, memory, a storage unit, or various input devices (e.g., buttons, touchpads), or may be coupled to a handheld controller (e.g., handheld controller 200) or auxiliary unit (e.g., auxiliary unit 300) that comprises one or more such components. In some embodiments, the sensors may be configured to output a set of coordinates of the head-mounted unit relative to the user's environment and may provide input to a processor to implement a simultaneous localization and mapping (SLAM) procedure and / or a visual odometry algorithm. In some embodiments, the wearable head device 100 may be coupled to a handheld controller 200 and / or an auxiliary unit 300, as described further below.

[0032] FIG. 2 illustrates an exemplary mobile handheld controller component 200 of an exemplary wearable system. In some examples, the handheld controller 200 may communicate wired or wirelessly with the wearable head device 100 and / or the auxiliary unit 300 described below. In some examples, the handheld controller 200 includes a handle portion 220 to be held by a user and one or more buttons 240 disposed along a top surface 210. In some examples, the handheld controller 200 may be configured for use as an optical tracking target, e.g., a sensor (e.g., a camera or other optical sensor) of the wearable head device 100 may be configured to detect the position and / or orientation of the handheld controller 200, which in turn may indicate the position and / or orientation of a user's hand holding the handheld controller 200. In some examples, the handheld controller 200 may include a processor, a memory, a storage unit, a display, or one or more input devices such as those described above. In some examples, the handheld controller 200 includes one or more sensors (e.g., any of the sensors or tracking components described above with respect to the wearable head device 100). In some examples, the sensors can detect a position or orientation of the handheld controller 200 relative to the wearable head device 100 or relative to another component of the wearable system. In some examples, the sensors may be located in the handle portion 220 of the handheld controller 200 and / or may be mechanically coupled to the handheld controller. The handheld controller 200 can be configured to provide one or more output signals corresponding, for example, to a press state of the button 240, or to a position, orientation, and / or movement of the handheld controller 200 (e.g., via an IMU). Such output signals may be used as inputs to a processor of the wearable head device 100, to the auxiliary unit 300, or to another component of the wearable system.In some embodiments, the handheld controller 200 may include one or more microphones to detect sounds (e.g., a user's speech, environmental sounds) and, in some cases, provide signals corresponding to the detected sounds to a processor (e.g., a processor of the wearable head device 100).

[0033] FIG. 3 illustrates an exemplary auxiliary unit 300 of an exemplary wearable system. In some examples, the auxiliary unit 300 may communicate wired or wirelessly with the wearable head device 100 and / or the handheld controller 200. The auxiliary unit 300 may include a battery to provide energy to operate one or more components of the wearable system, such as the wearable head device 100 and / or the handheld controller 200 (including a display, a sensor, an acoustic structure, a processor, a microphone, and / or other components of the wearable head device 100 or the handheld controller 200). In some examples, the auxiliary unit 300 may include a processor, a memory, a storage unit, a display, one or more input devices, and / or one or more sensors, such as those described above. In some examples, the auxiliary unit 300 includes a clip 310 for attaching the auxiliary unit to a user (e.g., a belt worn by the user). An advantage of using the auxiliary unit 300 to store one or more components of a wearable system is that doing so may allow a large or heavy component to be carried on the user's waist, chest, or back, which are relatively better suited for supporting large and heavy objects, rather than being mounted on the user's head (e.g., when stored in the wearable head device 100) or carried by the user's hands (e.g., when stored in the handheld controller 200). This may be particularly advantageous with respect to relatively heavy or bulky components, such as batteries.

[0034] FIG. 4 illustrates an example functional block diagram that may correspond to an example wearable system 400, such as may include the example wearable head device 100, handheld controller 200, and auxiliary unit 300 described above. In some examples, the wearable system 400 may be used for virtual reality, augmented reality, or mixed reality applications. As shown in FIG. 4, the wearable system 400 may include an example handheld controller 400B, referred to herein as a "totem" (and that may correspond to the handheld controller 200 described above), which may include a totem / headgear six degree of freedom (6DOF) totem subsystem 404A. The wearable system 400 may also include an example wearable head device 400A (which may correspond to the wearable headgear device 100 described above), which may include a totem / headgear 6DOF headgear subsystem 404B. In an embodiment, the 6DOF totem subsystem 404A and the 6DOF headgear subsystem 404B cooperate to determine six coordinates (e.g., offsets in three translation directions and rotations along three axes) of the handheld controller 400B relative to the wearable head device 400A. The six degrees of freedom may be expressed relative to the coordinate system of the wearable head device 400A. The three translation offsets may be expressed as X, Y, and Z offsets in such coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom may be expressed as a sequence of yaw, pitch, and roll rotations, a vector, a rotation matrix, a quaternion, or some other representation. In some embodiments, one or more depth cameras 444 (and / or one or more non-depth cameras) and / or one or more optical targets (e.g., buttons 240 of the handheld controller 200 as described above or dedicated optical targets included in the handheld controller) included in the wearable head device 400A can be used for 6DOF tracking.In some embodiments, the handheld controller 400B can include a camera as described above, and the headgear 400A can include an optical target for optical tracking in conjunction with the camera. In some embodiments, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids that are used to wirelessly transmit and receive three distinguishable signals. By measuring the relative magnitudes of the three distinguishable signals received in each of the coils used to receive, the 6DOF of the handheld controller 400B relative to the wearable head device 400A can be determined. In some embodiments, the 6DOF totem subsystem 404A can include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding high speed movements of the handheld controller 400B.

[0035] In some implementations involving augmented or mixed reality applications, it may be desirable to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the wearable head device 400A) to an inertial coordinate space or to an environmental coordinate space. For example, such a transformation may be necessary for the display of the wearable head device 400A to present virtual objects in an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the wearable head device 400A), rather than in a fixed position and orientation on the display (e.g., in the same position on the display of the wearable head device 400A). This can maintain the illusion that the virtual objects exist in the real environment (and do not appear unnaturally positioned in the real environment, e.g., as the wearable head device 400A shifts and rotates). In some examples, a compensation transformation between coordinate spaces can be determined by processing images from the depth camera 444 (e.g., using simultaneous localization and mapping (SLAM) and / or visual odometry procedures) to determine a transformation of the wearable head device 400A relative to an inertial or environmental coordinate system. In the example shown in FIG. 4, the depth camera 444 can be coupled to the SLAM / visual odometry block 406 and can provide images to the block 406. The SLAM / visual odometry block 406 implementation can include a processor configured to process the images and then determine a position and orientation of the user's head, which can be used to identify a transformation between the head coordinate space and the real coordinate space. Similarly, in some examples, an additional source of information regarding the user's head pose and location is obtained from the IMU 409 of the wearable head device 400A. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information regarding fast adjustments of the user's head pose and position.

[0036] In some examples, the depth camera 444 can provide 3D images to a hand gesture tracker 411, which can be implemented within a processor of the wearable head device 400A. The hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D images received from the depth camera 444 to stored patterns representing hand gestures. Other suitable techniques for identifying the user's hand gestures will also be apparent.

[0037] In some embodiments, one or more processors 416 may be configured to receive data from the headgear subsystem 404B, the IMU 409, the SLAM / visual odometry block 406, the depth camera 444, the microphone (not shown), and / or the hand gesture tracker 411. The processor 416 may also send and receive control signals to and from the 6DOF totem system 404A. The processor 416 may be wirelessly coupled to the 6DOF totem system 404A, such as in embodiments where the handheld controller 400B is not tethered. The processor 416 may further communicate with additional components, such as an audiovisual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source of imagewise modulated light 424 and a right channel output coupled to a right source of imagewise modulated light 426. The GPU 420 may output stereoscopic image data to the imagewise modulated light sources 424, 426. The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 416 indicating a direction vector from the user to a virtual sound source (which may be moved by the user, e.g., via the handheld controller 400B). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing an HRTF or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the believability and realism of virtual sounds by incorporating the user's relative position and orientation to the virtual sounds in the mixed reality environment, i.e., by presenting virtual sounds that match the user's expectations of what they would hear if the virtual sounds were real sounds in a real environment.

[0038] 4, one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included within an auxiliary unit 400C (which may correspond to auxiliary unit 300 described above). The auxiliary unit 400C may include a battery 427 to power its components and / or provide power to the wearable head device 400A and / or the handheld controller 400B. Including such components within an auxiliary unit, which may be mounted on the user's waist, can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0039] 4 presents elements corresponding to various components of the exemplary wearable system 400, various other suitable arrangements of these components will be apparent to those skilled in the art. For example, elements presented in FIG. 4 as being associated with the auxiliary unit 400C may instead be associated with the wearable head device 400A or the handheld controller 400B. Furthermore, some wearable systems may dispense with the handheld controller 400B or the auxiliary unit 400C entirely. Such variations and modifications are to be understood as falling within the scope of the disclosed embodiments.

[0040] Mixed Reality Environment

[0041] Like all people, users of a mixed reality system exist in a real environment, i.e., the three-dimensional portion of the "real world" and all of its contents that are perceptible by the user. For example, the user perceives the real environment using his or her normal human senses, i.e., sight, hearing, touch, taste, and smell, and interacts with the real environment by moving his or her body within the real environment. Locations within the real environment can be described as coordinates in a coordinate space, where, for example, the coordinates can include latitude, longitude, and altitude relative to sea level, distance in three orthogonal dimensions from a reference point, or other suitable values. Similarly, vectors can describe qualities that have direction and magnitude within a coordinate space.

[0042] The computing device can maintain a representation of the virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computer representation of a three-dimensional space. A virtual environment can include a representation of any objects, actions, signals, parameters, coordinates, vectors, or other properties associated with that space. In some examples, the circuitry (e.g., a processor) of the computing device can maintain and update the state of the virtual environment, i.e., the processor can determine the state of the virtual environment at a second time based on data associated with the virtual environment and / or input provided by a user at a first time. For example, if an object in the virtual environment is located at a first coordinate at a time and has certain programmed physical parameters (e.g., mass, coefficient of friction), and input received from a user indicates that a force should be applied to the object in a certain directional vector, the processor can apply the laws of kinematics and use fundamental mechanics to determine the location of the object at that time. The processor can use any suitable information known about the virtual environment and / or any suitable input to determine the state of the virtual environment at a time. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to creating and deleting virtual objects in the virtual environment, software (e.g., scripts) for defining behavior of virtual objects or characters in the virtual environment, software for defining behavior of signals (e.g., audio signals) in the virtual environment, software for creating and updating parameters associated with the virtual environment, software for generating audio signals in the virtual environment, software for handling inputs and outputs, software for implementing network operations, software for applying asset data (e.g., animation data for moving a virtual object over time), or many other possibilities.

[0043] An output device, such as a display or speaker, can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, lights, etc.) that may be presented to the user. The processor can determine a view of the virtual environment (e.g., corresponding to a "camera" with origin coordinates, viewing axis, and frustum) and render on the display a viewable scene of the virtual environment corresponding to that view. Any suitable rendering technique may be used for this purpose. In some examples, the viewable scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, the virtual environment may include audio aspects that may be presented to the user as one or more audio signals. For example, virtual objects in the virtual environment may generate sounds (e.g., a virtual character may speak or cause a sound effect) originating from the object's location coordinates, or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a particular location. A processor can determine audio signals corresponding to "listener" coordinates, e.g., audio signals that correspond to a composite of sounds in the virtual environment and that are mixed and processed to simulate an audio signal that would be heard by a listener at the listener coordinates, and present the audio signals to a user via one or more speakers.

[0044] Because the virtual environment exists only as a computer construct, the user cannot directly perceive the virtual environment using his or her normal senses. Instead, the user can only indirectly perceive the virtual environment as presented to the user, for example, by a display, a speaker, a haptic output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment, but can provide input data via input devices or sensors to a processor that can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data indicating that the user is attempting to move an object in the virtual environment, and the processor can use that data to cause the object to respond accordingly in the virtual environment.

[0045] Digital reverberation and ambient audio processing

[0046] The XR system can present the user with audio signals that originate at a sound source with origin coordinates and appear to travel in the direction of an orientation vector within the system. The user can perceive these audio signals as if they were real audio signals originating from the origin coordinates of the sound source and traveling along the orientation vector.

[0047] In some cases, audio signals may be considered virtual in that they correspond to computer signals in a virtual environment and not necessarily to real sounds in a real environment, however, virtual audio signals can be presented to a user as real audio signals detectable by the human ear, for example, as generated via speakers 120A and 120B of wearable head device 100 in FIG.

[0048] Some virtual or mixed reality environments suffer from the perception that the environment does not feel real or authentic. One reason for this perception is that audio and visual cues do not always match each other in the virtual environment. The entire virtual experience may feel fake and inauthentic, in part because it does not fit one's expectations based on real-world interactions. It is desirable to improve the user's experience by presenting audio signals that appear to interact realistically with objects in the user's environment, even in minor respects. The more consistent such audio signals are with one's expectations based on real-world experiences, the more immersive and engaging the user's experience will be.

[0049] As discussed above, the processor can determine an audio signal corresponding to a composition of sounds in the virtual environment. The composition of sounds can be generated based on the properties of the user's current environment. Exemplary properties include, but are not limited to, size, shape, material, and acoustic properties. For example, a brick wall may cause a different sound than a glass wall. As another example, the acoustic properties of the sound may be different when a couch is located in the current environment versus when the couch is absent. The processor may use information (e.g., one or more properties) about the user's current environment to set various parameters related to audio signal processing, which are discussed in detail below. The parameters can be used to determine information from a lookup table. Advantages of the embodiments disclosed below include reduced memory requirements, reduced network bandwidth, reduced power consumption, reduced computational complexity, and reduced computational delay. These advantages may be particularly significant for mobile systems, including wearable systems, where processing resources, networking resources, battery capacity, and physical size and weight are often limited.

[0050] In some embodiments, the processor may dynamically determine the parameters (e.g., calculate the impulse response on the fly). For example, the system may store one or more predefined signals in memory. The wearable head unit may generate a test audio signal and determine its response in the user's current environment, e.g., via a sensor in the wearable head unit. The response may be, for example, a reflected audio signal propagated through the user's current environment. The processor may determine the parameters based on a change between the test audio signal and the reflected audio signal. The reflected audio signal may be responsive to the generated test audio signal.

[0051] In some embodiments, the processor may determine the parameters based on one or more actions of the user. For example, the processor may use sensors on the wearable head device to determine whether the user has changed their gaze target, whether the user has changed their vital signs, etc. The processor may use the determined sensor information to determine parameters in the current environment that will result in the user's action.

[0052] In an environment as dynamic as AR, the filters used for audio signal processing must be continuously controlled, which can be achieved using the PPE, and more specifically, a cascade of shelving filters that create a three-band parametric equalizer or tone control with little processing overhead.

[0053] The system may use a second-order infinite impulse response (IIR) filter topology that facilitates parameter equalization. One such topology is the Regalia-Mitra topology. The Regalia-Mitra topology may be modified to obtain a parametric shelving filter with a "mutually similar" response for a given value ω of the control frequency.

[0054] In some embodiments, a true three-band parametric equalizer (e.g., bass / mid / treble) may be formed by cascading two proportional shelving filters. Cascading two filters may be equivalent to using one filter whose gain k is the product of the gains of the two filters. One filter may be a parametric low shelving equalizer and the other filter may be a parametric high shelving equalizer. Cascading the low shelving equalizer with the high shelving equalizer can result in a dual shelving equalizer. The dual shelving equalizer may have adjustable crossover frequencies and may be efficiently implemented as a biquadratic HR filter.

[0055] Example Implementation

[0056] 5 illustrates an example process 500 that may be performed by an XR system, such as by one or more processors of the XR system. The example process 500 uses a prototype filter to determine parametric filter parameters, a lookup table to store corresponding gain values, and an indexing scheme to efficiently retrieve gain information from the lookup table. Once the gain information is obtained from the lookup table, data (e.g., gain values) related to the control frequency are calculated. Each step is discussed in more detail below and illustrated as a non-limiting example.

[0057] In step 510, the system determines the magnitude response of a filter at a frequency. In some embodiments, this step includes calculating the magnitude response of one or more filters. The filters may be two separate filters, such as a low shelving equalizer and a high shelving equalizer. As discussed above, the low shelving equalizer has a control frequency F l and the high shelving equalizer can have a control frequency F h may have the following structure:

[0058] In some embodiments, the magnitude response of the first filter can be determined (e.g., approximately derived) from the magnitude response of the second filter. This determination can include scaling the magnitude response information (e.g., gain) of the second filter and shifting the data (e.g., the scaled magnitude response information) along the frequency axis by a predetermined frequency amount. The predetermined frequency amount can be an amount required to match the scaled magnitude response information of the second filter to the first filter.

[0059] In some embodiments, the filters may be symmetric. Thus, the magnitude response of a first filter (e.g., a high shelving equalizer) can be determined by inverting the magnitude response of a second filter (e.g., a low shelving equalizer) along the frequency axis. Examples of the present disclosure further include the first filter being a low shelving equalizer and the second filter being a high shelving equalizer.

[0060] In some embodiments, the frequency response of the prototype filter can be pre-computed. The corresponding magnitude response can also be pre-computed and stored in memory (step 520). The magnitude response can be stored in a look-up table along with other information such as frequency values ​​and associated gain values.

[0061] At run-time, the system retrieves the magnitude response information from the look-up table in step 530. In step 540, the system uses the magnitude response information to determine the control frequency F l , F m , and F h The gain G for the desired combination of hl , G hm , G lm , and G lhThe system may then process the audio signal by implementing a filter and applying the calculated gain to the audio signal (step 550). In some embodiments, process 500 may include the additional step of transmitting the processed audio signal to a wearable head device.

[0062] Exemplary Magnitude Response Determinations

[0063] For example purposes only, a prototype filter with a control frequency of 640 Hz may be selected. One advantage of the 640 Hz control frequency may be its suitability for audio applications. 640 Hz is approximately midway between 20 Hz and 20 kHz on a logarithmic scale, spanning the useful human listening distance. Another advantage of the 640 Hz control frequency may be that it is far enough away from DC and Nyquist (assuming a 44.1 kHz or 48 kHz sample rate) to avoid warping problems. Examples of the present disclosure include control frequencies other than 640 Hz.

[0064] FIG. 6 illustrates the magnitude response of an example low shelving prototype filter. The thin vertical lines indicate sampling frequency points. The sampling frequency points may be equally spaced entries in a lookup table. For example, the sampling frequency points may have 12 octave intervals. As used throughout this disclosure, the term "filter gain" refers to the gain at a control frequency. In some instances, the low shelving prototype filter may have a 1 dB gain at a control frequency of 640 Hz and a 2 dB gain at DC, as shown in the figure.

[0065] In some embodiments, the magnitude responses of the prototype filters at those 12 octave frequency points may be stored in a look-up table (step 520).

[0066] In some cases, this lookup table may be used later for filters with control frequencies close to DC or Nyquist (step 530). The data from the magnitude response determination may not cover a wide enough frequency range. In some embodiments, the system may set the magnitude response of such filters to be equal to a saturation value. For example, the saturation value may be 2 dB when the control frequency is below 20 Hz, or 0 dB when the control frequency is above 20 kHz. This assumed information may be stored in the lookup table (at step 520). Alternatively, the system may determine that the control frequency is outside a threshold range for the lookup table and use the assumed information as a result of the determination.

[0067] FIG. 7A illustrates a measured magnitude response of an exemplary low shelving equalizer, and FIG. 7B illustrates an approximated magnitude response of an exemplary low shelving equalizer. FIG. 7C illustrates FIG. 7A overlaid on FIG. 7B. The approximated magnitude response shown in FIG. 7B can be obtained by shifting a prototype filter by a predetermined frequency amount along the frequency axis. As shown in the figure, the prototype filter can provide a very good approximation. In some embodiments, there may be an approximation error for the low shelving equalizer when the control frequency approaches Nyquist. In some instances, the approximation error may be a function of the control frequency F l However, it may not affect performance because it is unlikely to be close to Nyquist. l is F m Lower than F m is F h lower than.)

[0068] FIG. 8A illustrates a measured magnitude response of an exemplary high shelving equalizer, and FIG. 8B illustrates an approximated magnitude response of an exemplary high shelving equalizer. FIG. 8C illustrates FIG. 8A overlaid on FIG. 8B. The approximated magnitude response shown in FIG. 8B can be obtained by shifting a prototype filter along the frequency axis. As shown in the figure, the prototype filter can provide a very good approximation. In some embodiments, two gains G hl and G hm There may be an approximation error for the high shelving equalizer that affects the calculation of the control frequency F h , F l , and F m can be noticeable if it is set too high (eg, above 2 kHz).

[0069] Lookup Tables and Indexing Schemes

[0070] As discussed above, in step 520, the magnitude response at a given frequency can be stored in a lookup table. The magnitude response can indicate an associated gain value of a prototype filter. FIG. 9A illustrates an exemplary lookup table including frequencies and associated gain values ​​for a prototype filter. Returning to the above example of a magnitude response having 12 octave intervals, the lookup table can include an entry for each sampling frequency point. For example, as shown in the figure, frequency F1 can have an associated gain G1 stored in the table, frequency F2 can have an associated gain G2 stored in the table, and frequency F3 can have an associated gain G3 stored in the table. 12 is the associated gain G stored in a table 12 Thus, for a given frequency point of interest, the system may retrieve a corresponding gain value from a look-up table in step 530.

[0071] In some embodiments, the system can retrieve the gain information using an index. An index for each frequency and corresponding gain value can be stored in a lookup table. FIG. 9B illustrates an exemplary lookup table including an index and associated frequency and gain value. Returning to the previous example of a magnitude response with 12 octave intervals, the lookup table can include an index for each sampling frequency point. The index id for any two frequency points (e.g., frequency points F1 and F2) can be stored in a lookup table. FIG. 9B illustrates an exemplary lookup table including an index and associated frequency and gain value. Returning to the previous example of a magnitude response with 12 octave intervals, the lookup table can include an index for each sampling frequency point. F2 and id F1 The relationship between can be expressed as follows: [ka]

[0072] Thus, the gain may be accessed from a table (in FIG. 9B) by using equation (1) to calculate the relative index. For example, G lm (When the gain is set to +1 dB, F m The dB gain of the low shelving equalizer at may be derived by calculating its index: [ka] In the formula, id Fcp is the index of the control frequency in the lookup table. In some embodiments, the index in the lookup table may be an integer value, as shown in the figure. As an example, frequency F6 in the table of FIG. 9B may be 640 Hz, which may correspond to index 6.

[0073] In some embodiments, the index relationship may be generalized to a prototype filter sampled on n octave intervals. The index relationship can be expressed as follows: [ka]

[0074] In some embodiments, the lookup table of FIG. 9B can be used for multiple filters, such as a low shelving filter and a high shelving filter. For example, the lookup table can store values ​​from a low shelving filter, and the high shelving filter response can be obtained by inverting the prototype filter data along the frequency axis. For example, G hl The index can be calculated as follows: [ka]

[0075] In some embodiments, the lookup table can include half the number of indexes used for retrieving the magnitude response information. Returning to the above example of a magnitude response with 12 octave intervals, the lookup table can include half (e.g., six) indexes. The six indexes can store the magnitude response information for a first filter (e.g., a low shelving filter). The magnitude response information for a second filter (e.g., a high shelving filter) can be obtained by using the information from the first filter stored in the table by using Equation (4). In this way, each index in the table can be used for multiple frequencies.

[0076] 9C illustrates an example lookup table including half the number of indexes. Index 1 corresponds to frequency F1 (first filter) and frequency F 12 (second filter), index 2 can be used for frequency F2 (first filter) and frequency F 11 (second filter), and so on.

[0077] Equations (1)-(4) above are indexing formulas that allow the system to retrieve the gain value corresponding to the nearest control frequency. Embodiments of the present disclosure may include using one or more interpolation methods on the retrieved gain information to convert it to a more accurate value corresponding to the actual frequency.

[0078] For example, the remaining index ids rem can be expressed as follows: [ka] Flooring Index ID F can be expressed as follows: [ka]

[0079] Linear interpolation may then generate the target index using: [ka]

[0080] Profit calculation

[0081] As discussed above, in step 530, the system retrieves magnitude response information from a lookup table. The magnitude response information may be a gain value. The desired dB gain at the low, mid, and high control frequencies of the dual shelving equalizer may be expressed as follows: [ka] In the formula, K l and K. h are the dB gains of the low and high shelving filters at their control frequencies, respectively (as shown in FIG. 1), K is the additive wideband gain, and G is the gain transformation matrix.

[0082] The gain transformation matrix G can be written as: [ka] In the formula, (1)G hl When the gain is set to +1 dB, the control frequency F l (2) is the dB gain of the high shelving equalizer at G hm When the gain is set to +1 dB, the control frequency F m (3) is the dB gain of the high shelving equalizer at lm When the gain is set to +1 dB, the control frequency F m (4) is the dB gain of the low shelving equalizer at lh When the gain is set to +1 dB, the control frequency F h is the dB gain of the low shelving equalizer at

[0083] From the matrix inversion of equation (9), a closed form solution for the internal gain can be determined and expressed as follows: [ka]

[0084] The inverse of the gain matrix can be expressed as: [ka] During the ceremony, [ka] and, [ka] It is.

[0085] From equations (11)-(13), the system can calculate the low and high shelving equalizer gains.

[0086] Independent Control Frequency

[0087] In some embodiments, the control frequency of the three-band parametric equalizer may be different from the control frequency of the dual shelving filter. For example, the control frequency of the three-band parametric equalizer may be related to one or more characteristics of the user, such as head size. Meanwhile, the control frequency of the shelving filter may be controlled through the system, which may not be based on the characteristics of the user. In this way, the control frequency of the three-band parametric equalizer may be independent from the control frequency of the dual shelving filter.

[0088] The desired dB gains at the low, mid, and high control frequencies of the dual shelving equalizer can be expressed as follows: [ka] In the formula, K lc and K. hc are the dB gains of the low and high shelving filters at their control frequencies, respectively, K is the additive wideband gain, and G is the gain transformation matrix.

[0089] From equation (14), the gain transformation matrix G can be written as: [ka] In the formula, (1)G hcl When the gain is set to +1 dB, the control frequency F l (2) is the dB gain of the high shelving equalizer at G hcm When the gain is set to +1 dB, the control frequency F m (3) is the dB gain of the high shelving equalizer at hchWhen the gain is set to +1 dB, the control frequency F h (4) is the dB gain of the high shelving equalizer at lcl When the gain is set to +1 dB, the control frequency F l (5) is the dB gain of the low shelving equalizer at lcm When the gain is set to +1 dB, the control frequency F m is the dB gain of the low shelving equalizer at lch When the gain is set to +1 dB, the control frequency F h is the dB gain of the low shelving equalizer at

[0090] From the matrix inversion of equation (15), a closed form solution for the internal gain can be determined and expressed as follows: [ka]

[0091] The inverse of the gain matrix can be expressed as: [ka] During the ceremony, [ka] and, [ka] It is.

[0092] From equations (17)-(19), the system can calculate the low and high shelving equalizer gains.

[0093] Implementing a Filter

[0094] Filters can then be easily implemented based on their transfer functions. Figure 10 shows example magnitude responses as a function of frequency for a low shelving equalizer, a high shelving equalizer, and a dual shelving equalizer. The dual shelving equalizer has multiple control frequencies, namely a low control frequency F l , Medium control frequency F m , and high control frequency F h In some embodiments, the control frequencies of the low and high shelving equalizers are equal to or greater than the low control frequency F of the dual shelving equalizer. l and high control frequency F h can be matched.

[0095] The transfer function of the parametric low shelving equalizer can be expressed as: [ka] During the ceremony, [ka] where k is the filter gain at DC and F C is the control frequency of the low shelving equalizer, and F S is the sampling frequency. In some embodiments, the gain at the control frequency ω is √k, which is half the decibel gain at DC.

[0096] The transfer function of the parametric high shelving equalizer can be expressed as follows: [ka] where k is the filter gain at Nyquist. In some embodiments, the gain at the control frequency ω is k, which is half the decibel gain at Nyquist.

[0097] 11A and 11B respectively illustrate an example magnitude response and an example phase response of a dual shelving equalizer having a target magnitude response that decreases monotonically with frequency. The present dual shelving equalizer may be useful in many applications, such as environmental acoustic modeling applications. As shown in the figure, the magnitude response may decrease continuously as frequency increases. The figure also shows the magnitude response of a low shelving equalizer and a high shelving equalizer. Using the method disclosed above, the gain values ​​of the dual shelving equalizer at the three control frequencies are determined to be −2.5 dB at 200 Hz, −5.0 dB at 1,000 Hz, and −12.0 dB at 5,000 Hz. Thus, the disclosed three-band parametric equalizer achieves a defined magnitude response at the control frequencies with high accuracy.

[0098] 12A and 12B respectively illustrate an example magnitude response and an example phase response of a dual shelving equalizer having a target magnitude response that does not decrease monotonically with frequency. As shown in the figure, the two shelving equalizers can be cascaded shelving equalizers with identical dB gain codes. The figure also shows the magnitude response of the low shelving equalizer and the high shelving equalizer. Using the method disclosed above, the gain values ​​of the dual shelving equalizer at the three control frequencies are determined to be −3.0 dB at 75 Hz, 4.5 dB at 1,500 Hz, and −6.0 dB at 12,000 Hz. Thus, the disclosed three-band parametric equalizer achieves a defined magnitude response at the control frequencies with high accuracy.

[0099] 13A and 13B respectively illustrate an example magnitude response and an example phase response of a dual shelving equalizer operated outside its operating range. A dual shelving equalizer may be operated outside its operating range when the required gains are too far apart for the control points and therefore close in frequency. The figures also show the magnitude responses of the low and high shelving equalizers. Using the method disclosed above, the gain values ​​of the dual shelving equalizer at the three control frequencies are determined to be 4.0 dB at 300 Hz, −2.0 dB at 1,200 Hz, and 10.0 dB at 4,000 Hz. As shown in the figures, the intermediate gain G m cannot be achieved (offset by about 5 dB).

[0100] As shown above, the modification of the Ragalia-Mitra structure provides a design that strictly respects the proportionality property for the shelving filter at three points: DC, Nyquist, and the control frequency of the filter. At other frequencies, the proportionality is verified to be approximately. In practice, for settings of the gain k within [-12 dB, +12 dB], the accuracy is sufficient for many audio applications.

[0101] With respect to the systems and methods described above, elements of the systems and methods can be implemented by one or more computer processors (e.g., CPU or DSP) as appropriate. The present disclosure is not limited to any particular configuration of computer hardware, including computer processors, used to implement these elements. In some cases, multiple computer systems can be employed to implement the systems and methods described above. For example, a first computer processor (e.g., a processor of a wearable device coupled to a microphone) can be utilized to receive input microphone signals and perform initial processing of those signals (e.g., signal conditioning and / or segmentation, such as those described above). A second (possibly more computationally powerful) processor can then be utilized to perform more computationally intensive processing, such as determining probability values ​​associated with speech segments of those signals. Another computing device, such as a cloud server, can host a speech recognition engine, to which the input signals are ultimately provided. Other suitable configurations will become apparent and are within the scope of the present disclosure.

[0102] Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

[Claim 1] The invention described in this specification.

Images