Customized binaural rendering of audio content

Abstract

Methods, systems, and media for processing audio are provided. In some embodiments, a method for processing audio may involve receiving a stereo audio signal. The method may involve separating the stereo audio signal into a pilot signal and a spread signal. The method may involve determining one or more diffusion signal modification parameters based on a current listening context, wherein the one or more diffusion signal modification parameters indicate a proportion of the diffusion signal to be reassigned to one or more output channels in an output multi-channel signal or a degree of attenuation to be applied to the diffusion signal. The method may involve generating the output multi-channel signal based on the pilot signal, the diffusion signal, and the one or more diffusion signal modification parameters. The method may involve providing the output multi-channel signal to a virtualizer for rendering as a binaural audio signal for playback on a wearable device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 497,025 filed on April 19, 2023 and PCT Application No. PCT / CN2022 / 0901993 filed on May 5, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to systems, methods, and media for customized binaural rendering of audio content. Background Art

[0004] Media content viewers are increasingly interested in spatial audio that can create a sense of immersion. For example, when listening to immersive audio content, listeners can feel as if the audio content is surrounding them. However, rendering spatial audio content can be difficult, especially when the sound is rendered binaurally via headphones or earbuds.

[0005] Symbols and nomenclature

[0006] Throughout this disclosure, including in the claims, the terms "speaker," "horn," and "audio reproduction transducer" are used synonymously to refer to any sound-producing transducer (or set of transducers). A typical set of headphones includes two speakers. Speakers can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds can undergo different processing in different circuit branches coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, the expression "performing an operation on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) is used in a broad sense to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0008] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio, or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the invention

[0010] A method, system, and medium for customized binaural rendering of audio content are provided. In some embodiments, the method involves receiving a stereo audio signal. The method may further involve separating the stereo audio signal into a directional signal and a diffuse signal, wherein the directional signal corresponds to the directional content in the stereo audio signal, and wherein the diffuse signal corresponds to the background content in the stereo audio signal. The method may further involve determining one or more diffuse signal modification parameters based on a current listening situation, wherein the one or more diffuse signal modification parameters indicate the proportion of the diffuse signal to be reallocated to one or more output channels in an output multi-channel signal or the degree of attenuation to be applied to the diffuse signal. The method may further involve generating the output multi-channel signal based on the directional signal, the diffuse signal, and the one or more diffuse signal modification parameters. The method may further involve providing the output multi-channel signal to a virtualizer for rendering as a binaural audio signal for playback on a wearable device.

[0011] In some examples, the one or more output channels include at least one of a left channel, a right channel, or a center channel.

[0012] In some examples, generating the output multi-channel signal includes: obtaining an expansion matrix; generating a modified expansion matrix using the expansion matrix and the one or more diffuse signal modification parameters; generating a diffuse multi-channel signal using the modified expansion matrix; and generating the output multi-channel signal based on the diffuse multi-channel signal and the steered signal. In some examples, the one or more diffuse signal modification parameters cause the proportion of the diffuse signal to be redistributed among the one or more output channels, and wherein generating the modified expansion matrix includes determining a matrix dot product of: a norm associated with the expansion matrix and a matrix representing the diffuse signal modification parameters, a matrix associated with the one or more diffuse signal modification parameters, and the expansion matrix. In some examples, the one or more diffuse signal modification parameters include a diffuse signal redistribution modification parameter indicating a redistribution of the diffuse signal in the multi-channel output, and wherein the norm normalizes the energy of the diffuse signal. In some examples, the one or more diffuse signal modification parameters cause the degree of attenuation to be applied to the diffuse signal, and wherein generating the output multi-channel signal includes performing energy normalization, the energy normalization being configured to cause the energy of the output multi-channel signal to be the same as the energy of the stereo audio signal. In some examples, normalizing the energy is performed by one of: an upmixer that generates the output multi-channel signal, or the virtualizer.

[0013] In some examples, the current listening context includes one of the following: watching movie content mode, listening to music mode, or playing game mode. In some examples, the current listening context is watching movie content mode, and wherein the one or more diffuse signal modification parameters are in the range of about 0.8-1. In some examples, the current listening context is listening to music mode, and wherein the one or more diffuse signal modification parameters are in the range of about 0-0.2. In some examples, the current listening context is playing game mode, and wherein the one or more diffuse signal modification parameters have a value less than a value associated with watching movie content mode.

[0014] In some examples, the one or more diffuse signal modification parameters are received from a user of the wearable device. In some examples, the one or more diffuse signal modification parameters are received via a user interface.

[0015] In some examples, generating the output multi-channel signal occurs on a companion user device associated with the wearable device, and wherein the virtualizer includes one or more components executed on the wearable device. In some examples, the method further involves transmitting data from the companion user device to the wearable device via a Bluetooth communication protocol.

[0016] In some examples, the wearable device includes one of an earbud or a headset.

[0017] In some examples, the wearable device includes one or more sensors that collect sensor data that can be used to generate head tracking information associated with a wearer of the wearable device.

[0018] In some examples, the virtualizer is configured to render the binaural audio signal based on the output multi-channel signal and the head tracking information.

[0019] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices, such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0020] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

[0021] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a diagram of an example system including an upmixer that generates a customized multi-channel output signal according to some implementations.

[0023] Figure 2A , Figure 2B and Figure 2C Depicted are graphs illustrating the effects of various values ​​of a diffusion signal modification parameter according to some implementations.

[0024] Figure 3 is a diagram of an example upmixer according to some implementations.

[0025] Figure 4 is a flow chart of an example process for generating a multi-channel output signal according to some implementations.

[0026] Figure 5 is a flow chart of an example process for generating a diffuse multi-channel signal using diffuse attenuation signal parameters according to some implementations.

[0027] Figure 6 is a flow chart of an example process for generating a diffuse multi-channel signal that distributes a diffuse signal according to some implementations.

[0028] Figure 7 A block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure is shown.

[0029] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0030] Media content viewers are increasingly interested in spatial audio that creates a sense of immersion. For example, when listening to immersive audio content, listeners can feel as if the audio content is surrounding them. However, rendering spatial audio content can be difficult, especially when the sound is rendered binaurally via headphones or earplugs. For example, when the user moves their head, spatial audio can cause a perception of instability in the audio scene when rendered binaurally via headphones or earplugs. For example, if the listener is listening to music and thus expects the singer and / or instrumentalist to be perceived as being in front of the listener, then when the spatial audio is rendered binaurally and the listener moves their head (e.g., looking around them), the listener may perceive a jump or discontinuity. Rendering spatial audio via headphones and / or earplugs that perform head orientation determination can be particularly challenging because listeners may have different preferences for whether to prioritize immersion or scene stability (which may additionally depend on the type of audio content being listened to). For example, listeners may prioritize scene stability when listening to music, where direct or directed signals (such as vocals or instrumental music) are perceived as fixed in front of the listener. In other words, such rendering can make listeners perceive that they are listening to music in front of the front speakers. In turn, listeners may prioritize immersion when listening to audio content associated with movies. It is particularly challenging to customize the listening experience of spatial audio when presented via headphones or earplugs. As used herein, "immersion" refers to rendering audio data in a way that is perceived as three-dimensional and surrounds the user. Immersive audio content may involve rendering an audio object as having a given spatial position, which has a specific azimuth and / or elevation relative to the listener. For example, audio data rendered in an immersive manner may produce a listening experience in which audio sounds are rendered in a way that is perceived as surrounding the listener rather than just in front of the listener. As a specific example, audio content rendered in an immersive manner may include the sound of an airplane or helicopter, which is rendered so that the listener perceives the sound as being above the head. The techniques disclosed herein allow users to adjust the audio scene, for example, by allowing audio objects to be fixed at a particular perceived location (e.g., at the screen where the content is rendered), or by allowing audio objects to be perceived as surrounding or encircling the user. For example, the techniques described herein allow a listener to balance scene stability, which generally refers to a listening experience in which audio objects do not change position as the listener moves their head (e.g., from side to side as they look around), with immersion. Allowing a listener to balance scene stability with immersion can allow a listener to customize the listening experience based on the type of content the listener is listening to.

[0031] Disclosed herein is a technique for generating a customized multi-channel output signal based on a current listening situation. The customized multi-channel output signal can be generated by taking into account a diffuse signal modification parameter, or by reallocating at least a portion of the diffuse signal to one or more output channels (such as a left, right, or center channel), the diffuse signal modification parameter causing the diffuse signal to be attenuated (thereby causing the direct or directed signal to be perceived as more prominent, which in turn can increase the perception of scene stability). In the case where at least a portion of the diffuse signal is reallocated, the extent to which the diffuse signal is allocated can depend on the listening situation. For example, in the case where the listening situation indicates that the listener is watching a movie, no diffuse signal, or a relatively small proportion of the diffuse signal, can be allocated to the left, right, and center channels, thereby maintaining a sense of immersion. Conversely, in the case where the listening situation indicates that the listener is listening to music or playing a game, a larger proportion of the diffuse signal can be allocated to one or more output channels, thereby increasing the perception of scene stability when the user moves their head.

[0032] In some implementations, the customized multi-channel output signal can be generated by the upmixer component of the device. The device can be a user device (e.g., a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, a television set, etc.) that makes audio content, for example, presented via paired or connected headphones or earplugs. The customized multi-channel output signal can then be rendered as a binaural audio signal by the virtualizer component. In some embodiments, the virtualizer component can be a part of headphones or earplugs, so that rendering as a binaural audio signal can depend on the user's head orientation.

[0033] Figure 1 1 is a block diagram of a system configured to generate and utilize customized binaural audio rendering according to some implementations. As shown, an upmixer 102 receives a stereo audio signal including a left audio signal and a right audio signal. The upmixer 102 additionally receives diffuse signal modification information. The diffuse signal modification information may indicate the manner in which the diffuse signal is to be attenuated or reassigned to one or more channels (such as one or more of the left, right, and center channels of a customized multi-channel output signal generated by the upmixer 102). The diffuse signal modification information may correspond to a current listening context. Example listening contexts include a user listening to music, watching a movie, playing a game (e.g., a computer game), etc. The diffuse signal modification information may include one or more parameters, each set of one or more parameters being associated with a given listening context. Multiple sets of one or more parameters may be stored in a device (e.g., a memory) used to present audio content, such that the device retrieves a set of diffuse signal modification parameters corresponding to the current listening context. The upmixer 102 may then generate a multi-channel audio signal. The example upmixer system in Figure 3 is shown in and combined below Figure 3A technique for generating a customized multi-channel audio signal based on diffuse signal attenuation information is described in Figure 4 , Figure 5 and Figure 6 is shown in and combined below Figure 4 , Figure 5 and Figure 6 Give a description.

[0034] The upmixer 102 may provide the multi-channel audio signal to the virtualizer 104. Note that in some implementations, the upmixer 102 may be executed on a companion device (e.g., a mobile phone, a tablet computer, a laptop computer, etc.) that provides the audio signal for playback by a paired set of headphones or earbuds, and the virtualizer 104 may be executed on the paired headphones or earbuds. In such an implementation, the upmixer 102 may transmit the multi-channel audio signal to the virtualizer 104 via Bluetooth or another wireless communication protocol. Note that in some implementations, the upmixer 102 and the virtualizer 104 may be implemented on the same device.

[0035] The virtualizer 104 can receive a multi-channel audio signal and can render the multi-channel audio signal as a binaural audio signal suitable for playback via, for example, headphones or earplugs. Note that the virtualizer 104 can render the multi-channel audio signal based on head tracking information obtained using one or more sensors (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetometers), etc. One or more sensors can be deployed in or on headphones or earplugs. It should be noted that because the multi-channel audio signal is generated based on the current listening context (e.g., indicating the type of audio content or video content that the user is consuming), the diffuse signal can be attenuated (e.g., so that the audio signal to be rendered in front of the user is promoted relative to the diffuse signal) or reassigned to one or more channels (such as one or more of the left, right and center channels of the upmixed multi-channel audio signal). Subsequently, the virtualizer can render the multi-channel audio signal as a binaural audio signal in a manner dependent on the head orientation of the listener. Thus, binaural audio signals can be presented in a manner that depends on the listener's head orientation and the listening context, so that diffuse signals are attenuated or redistributed in a customized manner consistent with the user's listening preferences for various types of audio content. For example, where the listening context corresponds to watching a movie, the binaural audio signals can be rendered in a manner that is substantially immersive regardless of the user's head orientation. As another example, where the listening context corresponds to listening to music, the binaural audio signals can be rendered in a manner that makes vocals and instrumental music perceived as being in front of the user regardless of the user's head orientation, thereby improving scene stability.

[0036] Figure 2A , Figure 2B and Figure 2C The effect of varying values ​​of a diffuse signal modification parameter (generally denoted herein as β) is illustrated. In general, β indicates the degree to which the diffuse signal is expanded or reassigned to other channels (e.g., left, right, and / or center channels) of a multi-channel mix. Note that Figure 2A , Figure 2B and Figure 2C In, L s and R s Denote the left and right surround signals, respectively, and L, R, and C denote the left, right, and center channels of the 5.1 upmix signal, respectively. Figure 2A , when β is 1, the diffuse signal from the left and right surround channels is not expanded or reassigned to the left, right, and center channels. Figure 2B , when β is between 0 and 1, some portion of the diffuse signal from the left and right surround channels is expanded or reassigned to the left, right, and center channels, but some portion of the diffuse signal remains in the left and right surround channels. Figure 2C , in the case where β is 0, all of the diffuse signals from the left and right surround channels are expanded or reassigned to the left, right and center channels. Note that the allocation of diffuse signals can be implemented for other upmix formats (such as 7.1 upmix, etc.).

[0037] In some implementations, the diffuse signal modification can be performed by an upmixer. In some implementations, the upmixer can be a component or module of a user device that provides audio content for playback. For example, the user device can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, a television set, etc. The upmixer can be configured to receive a stereo audio signal (e.g., a left signal and a right signal) and generate a customized multi-channel output signal based on a diffuse signal modification parameter. The diffuse signal modification parameter can be received (e.g., obtained and / or identified) by the upmixer based on the current listening situation of the user of the user device. The upmixer can be configured to separate a direct signal from a diffuse signal, wherein the direct signal corresponds to, for example, a human voice and / or an instrumental sound, and the diffuse signal corresponds to general surrounding and / or environmental sound. The upmixer can be configured to pan the direct signal so that the direct signal is rendered as if it is placed at a single point. The upmixer can be configured to decorrelate and expand the diffuse signal so that the diffuse signal is reassigned or attenuated around the output channel, thereby affecting the immersion and / or scene stability of the audio content experienced by the listener. The upmixer can be configured to combine the panned pilot signal and the diffuse signal of expansion and / or attenuation into a customized multi-channel output signal. It is noted that the upmixer can be configured to receive the stereo signal in the time domain and generate the multi-channel output signal in the time domain. However, in some implementations, the upmixer can be configured to generally perform processing in the frequency domain. In other words, in some embodiments, the upmixer can convert the received stereo signal to the frequency domain before separating the pilot signal and the diffuse signal, expanding and / or attenuating the diffuse signal to other channels, generating the combined multi-channel output signal, etc. The upmixer can then transform the multi-channel output signal from the frequency domain to the time domain before providing the multi-channel output signal to the virtualizer for rendering.

[0038] Figure 3 is a block diagram of an example upmixer 300 according to some implementations. In some implementations, the upmixer 300 may be implemented using, for example, one or more processors or controllers of a user device (e.g., a mobile phone, a tablet computer, a laptop computer, a desktop computer, a gaming console, etc.). An example of such a controller is Figure 7 A control system 710 is provided.

[0039] As shown, the upmixer 300 may receive a stereo audio signal, which is generally denoted herein as L T (n) (e.g., left stereo signal) and R T (n) (e.g., a right stereo signal), where n represents the current audio frame. The stereo signal may be transformed from the time domain to the frequency domain using time-frequency transform blocks 302a and 302b. For example, the time-frequency transform block 302a may output a frequency domain representation of the left stereo signal, which is generally denoted herein as LT (m, k), and the time-to-frequency transform block 302b may output a frequency domain representation of the right stereo signal, which is generally denoted herein as R T (m, k), where m represents the time block index and k represents the frequency index.

[0040] The frequency domain representation of the stereo audio signal may be provided to the statistical estimation block 304. The statistical estimation block 304 may generate estimated parameters X(m,b), Y(m,b), and T(m,b), which may be provided to the separation block 306. The frequency domain representation of the stereo audio signal may also be provided to the separation block 306, such as Figure 3 as shown in .

[0041] The separation block 306 may be configured to separate the direct signal and the diffuse signal. The direct signal may be represented as S for the left and right direct signals, respectively. L (m,k) and S R (m,k). The diffusion signal can be expressed as d L (m,k) and d R (m, k). In some implementations, the diffuse signal can be obtained by subtracting the estimated steered signal from the input stereo signal, where the subtraction is performed in the frequency domain. For example, in some implementations, the left and right diffuse signals can be determined by the following formulas:

[0042]

[0043] In the equation given above, W(m,b) represents the steering signal separation matrix.

[0044] Direct signal S L (m,k) and S R (m, k) may be provided to a panning block 308. The panning block 308 may be configured to position the direct signal at a particular location. Note that the panning block 308 may utilize parameters generated by the statistical estimation block 304. Further, note that the panning block 308 is configured to output a multi-channel output of the direct signal. For example, the multi-channel direct signal may have N channels, where N is the total number of channels. For example, for a 5.1 upmix, N is 5, and for a 7.1 upmix, N is 7.

[0045] The decorrelation and expansion block 310 may receive the diffuse signal d L (m,k) and d R(m, k). The decorrelation and expansion block 310 can be additionally configured to receive, obtain or determine diffuse signal modification parameters, which can be specified in the diffuse energy adjustment matrix B. The diffuse signal modification parameters can be received or determined based on the current listening situation. The decorrelation and expansion block 310 can be configured to modify the diffuse signal so that the diffuse signal is attenuated (thereby making the direct signal more prominent) or so that the diffuse signal is reassigned to other output channels (for example, the left, right and center channels in a 5.1 upmix). The decorrelation and expansion block 310 can modify the diffuse signal by generating a modified expansion matrix that controls the degree to which the diffuse signal exists in each output channel. The expansion matrix, generally represented as O in this document, can be obtained based on the parameters generated by the statistical estimation block 304 and can be modified based on the diffuse signal modification parameters. The technique for generating the modified expansion matrix is ​​described in Figure 5 and Figure 6 is shown in and combined below Figure 5 and Figure 6 The modified diffusion signal is generally denoted as Z in this article. d1 (m,k),...Z dN (m, k), where N is the number of channels in the multi-channel output signal.

[0046] The panned multi-channel direct signal may be combined with the modified diffuse signal to generate an output multi-channel signal in the frequency domain, which is generally denoted herein as Z d1 (m,k),...Z dN (m, k), where N is the number of channels in the multi-channel output signal. The output signal can then be transformed into the time domain to generate a multi-channel output signal in the time domain, which is denoted herein as Z 1 (n),...Z N (n), where n is the audio frame number and N is the number of output channels. The transformation to the time domain may be achieved via a set of frequency-time transformation blocks, such as blocks 312a and 312b.

[0047] As described above, the upmixer can generate a customized multi-channel output based on the current listening situation. The current listening situation can indicate the type of content being presented by the user device (e.g., whether the content is music content, video content (such as a movie or TV show), game content, etc.), whether the audio content is being presented via paired headphones or earplugs, etc. For example, the diffuse signal modification parameters can be applied in the case where the audio content is being presented via paired headphones or earplugs, and may not be applied in the case where the audio content is presented directly by the user device (e.g., via the speaker of the user device). In some implementations, the diffuse signal modification parameters can be obtained, retrieved or otherwise determined based on the type of content being presented. For example, the user device can store different groups of diffuse signal modification parameters that can be applied to different types of content. For example, the diffuse signal modification parameters may include diffuse signal attenuation parameters, which, in response to determining that the current listening situation corresponds to the music content being played, cause the diffuse signal to be attenuated (thereby rendering the vocals and instrumental sounds more prominent). As another example, the diffuse signal modification parameters may include a first set of diffuse signal modification parameters that, in response to determining that the current listening situation corresponds to playback of a movie, cause the diffuse signal to not be reassigned to other output channels at all, thereby causing the movie audio content to be rendered in a more immersive manner. As yet another example, the diffuse signal modification parameters may include a second set of diffuse signal modification parameters that, in response to determining that the current listening situation corresponds to playing a video game, cause a portion of the diffuse signal to be reassigned or reallocated to other output channels, thereby causing increased scene stability when the user moves their head at the expense of less immersive perception.

[0048] Regardless of whether the diffuse signal is attenuated or reassigned to other output channels, the diffuse signal modification parameters can be applied by modifying the expansion matrix to generate a modified expansion matrix. The modified expansion matrix can therefore indicate the degree to which the diffuse signal is attenuated or redistributed. The modified diffuse signal can then be determined based on the modified expansion matrix. The customized multi-channel output signal can then be determined by combining the multi-channel modified diffuse signal with the multi-channel direct signal and transforming the combined signal into the time domain. It is noted that in the case where the diffuse signal is attenuated (for example, in the case where the diffuse signal modification parameters correspond to parameters that make the direct signal (such as vocals and / or instrumental music) rendered more prominently), the multi-channel output signal may have less energy than the input stereo signal. Thus, in such a case, in some embodiments, the multi-channel output signal can be normalized so that the output signal has the same energy as the input stereo signal.

[0049] Figure 44 is a flow chart of an example process 400 for generating a customized multi-channel output signal according to some embodiments. In some implementations, the blocks of process 400 may be performed on a user device. For example, the user device may be a device that enables playback of audio content (or audio content associated with video content) via paired headphones or earbuds. Examples of such user devices include mobile phones, tablet computers, laptop computers, desktop computers, game consoles, televisions, etc. The blocks of process 400 may be performed by one or more processors or controllers of the user device. An example of such a controller is control system 710, which is Figure 7 is shown in and combined below Figure 7 In some implementations, the blocks of process 400 may be performed in accordance with Figure 4 In some implementations, two or more blocks of process 400 may be performed substantially in parallel. In some implementations, one or more blocks of process 400 may be omitted.

[0050] Process 400 may begin at 402 by receiving a stereo audio signal. The stereo audio signal may be received by an upmixer. As described above, the stereo audio signal may be generally represented herein as L T (n) (e.g., left stereo signal) and R T (n) (e.g., right stereo signal), where n represents the current audio frame. Note that after receiving the stereo audio signal, the process 400 can transform the stereo audio signal from the time domain to the frequency domain. For example, the process 400 can utilize a short-time Fourier transform (STFT).

[0051] At 404, process 400 may separate the stereo audio signal into a direct signal and a diffuse signal. Process 400 may separate the stereo signal into a direct signal and a diffuse signal based on a statistical estimation performed using a frequency domain representation of the stereo audio signal, such as Figure 3 shown in and combined above Figure 3 The direct signal is generally referred to as S for the left and right direct signals in this article. L (m,k) and S R (m, k), and the diffuse signals are generally referred to herein as d for the left and right diffuse signals, respectively. L (m,k) and d R In some implementations, the diffuse signal can be obtained by subtracting the estimated steered signal from the input stereo signal (e.g., using a steered signal separation matrix), as described above in conjunction with Figure 3 Described.

[0052] At 406, process 400 can determine one or more diffuse signal modification parameters based on the current listening context. As described above, the current listening context can indicate the type of audio content being presented (e.g., whether the audio content is music content, audio content associated with a movie or television program, audio content associated with a video game, etc.) and / or whether the audio content is being played back via headphones and / or earbuds.

[0053] In some implementations, the diffuse signal modification parameters may include parameters configured to attenuate the diffuse signal. Attenuation may be performed where increasing scene stability during user head movement is a priority. For example, attenuation of the diffuse signal may be performed where the current listening context indicates that the audio content is music being listened to using paired headphones or earbuds.

[0054] In some implementations, the diffuse signal modification parameters may include parameters indicating the extent to which the diffuse signal is to be reallocated to one or more output channels (e.g., one or more of the left, right, and / or center channels), thereby causing a change in the degree of immersion perceived by the user. For example, where the diffuse signal modification parameters indicate that the diffuse signal is not to be reallocated to one or more output channels (e.g., where the current listening context indicates that the audio content is associated with a movie or television program and a high perception of immersion is therefore desired), the diffuse signal modification parameters may not reallocate any portion of the diffuse signal. Conversely, where the diffuse signal modification parameters indicate that the diffuse signal is to be at least partially reallocated to other output channels (e.g., where the current listening context indicates that the audio content is associated with content other than a movie or television program (such as music, video games, podcasts, etc.)), at least a portion of the diffuse signal may be reallocated, for example, to the left, right, and center channels, thereby increasing the perception of scene stability at the expense of the perception of immersion.

[0055] It is noted that the diffuse signal modification parameters may be specified by a user of the user device, or may be programmed into the user device by, for example, a manufacturer of the user device. The diffuse signal modification parameters may include different groups of diffuse signal modification parameters, each group being applicable to different listening situations. Process 400 may then retrieve diffuse signal modification parameters applicable to the current listening situation. In the case where the diffuse signal modification parameters are specified and / or modified by a user of the user device, the parameters may be specified or modified via, for example, a user interface presented on the user device. For example, the user interface may include a slider control or other user interface control that allows the user to adjust the diffuse signal modification parameters for different listening situations. The settings may then be stored in, for example, a memory of the user device for use during future listening sessions. It is noted that in some embodiments, the diffuse signal modification parameters may be received from a wearable device (e.g., a pair of earplugs or headphones) and / or from a matching user device (e.g., a paired mobile device).

[0056] At 408, process 400 may generate an output multi-channel signal based on the direct signal, the diffuse signal, and one or more diffuse signal modification parameters. For example, process 400 may apply the diffuse signal modification parameters by applying the diffuse signal modification parameters to an expansion matrix to generate a modified expansion matrix. The modified expansion matrix may then be used to generate a modified diffuse signal, which may be attenuated or redistributed relative to the original diffuse signal. The modified diffuse signal may then be combined with the direct signal to generate an output multi-channel signal. In some implementations, the output multi-channel signal may then be transformed into the time domain. An example of a process for generating an attenuated diffuse signal by generating a modified expansion matrix is ​​provided in Figure 5 is shown in and combined below Figure 5 An example of a process for allocating diffuse signals to other output channels by generating a modified expansion matrix is ​​described in Figure 6 is shown in and combined below Figure 6 Give a description.

[0057] In some embodiments, at 410, process 400 may optionally perform energy normalization. For example, energy normalization may be performed in the case where the diffuse signal at block 408 is attenuated to account for the reduction in total energy in the output multi-channel signal when utilizing the attenuated diffuse signal. Energy normalization may cause the normalized multi-channel signal to have a total energy similar to the total energy of the stereo audio signal received at block 402. Note that, in the case where the diffuse signal is not attenuated but redistributed to other output channels, energy normalization does not need to be performed and block 410 may be omitted. In addition, it should be noted that energy normalization may be performed by an upmixer or by a virtualizer. In the case where energy normalization is performed by a virtualizer, block 410 may be performed after block 412 described below. Energy normalization may be performed in the time domain or in the frequency domain.

[0058] At 412, process 400 may provide the output multi-channel signal to a virtualizer for rendering as a binaural audio signal for playback on a wearable device. The wearable device may include headphones or earplugs. The virtualizer may be implemented on a user device that provides audio content or on a wearable device (e.g., headphones or earplugs). In some implementations, the wearable device may include one or more sensors that can be used to determine the user's head orientation. The virtualizer may then render the binaural audio signal based on the head orientation. Thus, in the case where the diffuse signal modification parameter is a parameter that prioritizes scene stability, the binaural audio signal may be rendered so that the direct signal (e.g., vocals, instrumental music, etc.) is rendered to be perceived in front of the user regardless of the user's head orientation. Conversely, in the case where the diffuse signal modification parameter is a parameter that prioritizes immersion, the binaural audio signal may be rendered so that the diffuse signal is distributed in a manner that causes an immersive perception even when the user moves their head. Note that because the diffuse signal modification parameters are specific to a listening context that may indicate the type of audio content being presented, scene stability may be prioritized for some types of audio content, while immersion may be prioritized for other types of audio content. Furthermore, because the diffuse signal modification parameters may be set or adjusted by an end user of a user device in some implementations, the listener may control whether scene stability or immersion is prioritized for different types of audio content, and the degree to which each is prioritized.

[0059] In some implementations, the diffuse signal modification parameter may include a diffuse signal attenuation parameter that causes the diffuse signal to be attenuated. This may cause the steer or direct signal to be rendered in a manner that is perceived as fixed in front of the listener, even when the listener moves their head while wearing headphones or earplugs. In other words, the steer or direct signal may be rendered more prominently and rendered in a manner that is perceived as fixed to the front, thereby increasing the listener's sense of scene stability even when the listener moves their head. In the case where the current listening context is listening to music content, the attenuation of the diffuse signal may be performed because the steer or direct signal may include vocals or instrumental music that are advantageously rendered more prominently. In some implementations, the attenuation of the diffuse signal may involve applying the diffuse signal attenuation parameter to the expansion matrix to generate a modified expansion matrix. The modified expansion matrix may be used to generate a modified diffuse signal that is attenuated relative to the original diffuse signal. Note that because the diffuse signal is attenuated, the multi-channel output signal including the attenuated diffuse signal and the multi-channel direct signal may be lower in energy relative to the original stereo signal. Thus, normalization may be performed to normalize the energy of the multi-channel output signal to the energy of the original stereo signal so that the rendered binaural signal does not drop below a desired value in volume when presented.

[0060] Figure 5 5 is a flow chart of an example process 500 for attenuating diffuse signals according to some implementations. In some implementations, blocks of process 500 may be performed on a user device. For example, the user device may be a device that enables playback of audio content (or audio content associated with video content) via paired headphones or earbuds. Examples of such user devices include mobile phones, tablet computers, laptop computers, desktop computers, game consoles, televisions, etc. Blocks of process 500 may be performed by one or more processors or controllers of the user device. An example of such a controller is control system 710, which is Figure 7 is shown in and combined below Figure 7 In some implementations, the blocks of process 500 may be performed in accordance with Figure 5 In some implementations, two or more blocks of process 500 may be performed substantially in parallel. In some implementations, one or more blocks of process 500 may be omitted.

[0061] Process 500 may begin at 502 by obtaining one or more diffuse signal attenuation parameters, an expansion matrix, and a diffuse signal. Note that the diffuse signal may be provided in the frequency domain. Figure 3 As described above, the diffusion signal can be expressed as d 1 (m,k),…d j(m, k), where j is the number of diffuse signals, m is the time block index, and k is the frequency band. The expansion matrix may be denoted herein as O, which may have dimensions N×j, where N is the number of channels. For example, for a 5.1 channel upmix, N may be 5, and j may be 2. In other examples, N may be 7, 9, etc., and j may be 2, 3, 4, etc. In some embodiments, the expansion matrix may be based on statistical estimation parameters (e.g., as estimated by statistical estimation block 304, such as Figure 3 shown in and combined above Figure 3 The diffuse signal attenuation parameters may be generally represented herein as a matrix B, which may have dimensions N×j, similar to the matrix O. The diffuse signal attenuation parameters may be obtained from a memory of the user device and may have been set based on user preferences related to the extent to which the diffuse signal is to be attenuated for various listening scenarios (e.g., for various types of audio content). Note that in some implementations, each element of the matrix β may be between 0 and 1, with a value of 0 indicating that the diffuse signal will be attenuated entirely and a value of 1 indicating that the diffuse signal will not be attenuated to any extent.

[0062] At 504, process 500 may generate a modified spreading matrix using the spreading matrix and one or more diffuse signal attenuation parameters. For example, in some implementations, the modified spreading matrix O' may be determined by:

[0063]

[0064] At 506, process 500 may generate an attenuated diffuse multi-channel signal using the modified expansion matrix and the diffuse multi-channel signal. Note that the diffuse multi-channel signal (which is generally denoted herein as Z d1 ,…Z dN ) is conventionally generated by multiplying an expansion matrix by a vector formed by diffuse signals. However, to generate an attenuated diffuse multi-channel signal, process 500 may multiply a modified expansion matrix incorporating diffuse signal attenuation parameters by a vector formed by diffuse signals. For example, in some embodiments, the diffuse multi-channel signal may be determined by:

[0065]

[0066] like Figure 3 and Figure 4 shown in and combined above Figure 3 and Figure 4 As described above, the diffuse multi-channel signal can then be combined with the multi-channel direct signal to generate a multi-channel output signal in the frequency domain. Figure 3 and Figure 4 As described, the frequency domain signal can then be transformed into the time domain.

[0067] Note that because the total energy in the output multi-channel signal is attenuated due to the attenuation of the diffuse signal, the energy can be normalized before rendering the binaural audio signal. For example, the energy can be normalized based on the stereo signal. Such normalization can be performed by the upmixer, virtualizer, or any other component.

[0068] In some implementations, the diffuse signal modification parameters may include parameters that cause at least a portion of the diffuse signal to be reallocated to one or more output channels (such as left, right and / or center channels). Such reallocation can be used to effectively balance scene stability and immersion for the presentation of immersive audio content. For example, when spatial audio is rendered from a device such as a tablet computer or mobile phone and presented via a user's headphones or earbuds, allocating at least a portion of the diffuse signal to other output channels can increase the perception of scene stability, especially when the user's head orientation changes. In some cases, the extent to which the diffuse signal is reallocated can depend on the listening context. For example, for a listening context in which a user is watching a movie, no diffuse signal, or a relatively low proportion of diffuse signals (e.g., 10%, 15%, etc.) can be reallocated to prioritize the feeling of immersion. As another example, for a listening context in which a user is watching or playing a game, a relatively large proportion of diffuse signals (e.g., a larger proportion than that allocated for a context in which a movie is watched) can be reallocated. For example, when watching or playing a game, 60%, 70%, 80%, etc. of diffuse signals can be reallocated.

[0069] Combined with the above Figure 5 Similar to the description, in some embodiments, the modified diffusion signal may be determined by generating a modified expansion matrix using the diffusion signal modification parameters. Figure 5 Unlike the description, because the total energy of the multi-channel output signal should not be attenuated, a normalization parameter used to maintain the norm (e.g., Frobenius norm) of the expansion matrix can be used to generate the modified expansion matrix. In addition, in some implementations, the diffuse signal modification parameter may include a single diffuse signal modification parameter. The use of a single diffuse signal modification parameter may allow user configuration to be easier.

[0070] Figure 6600 is a flow chart of an example process 600 for distributing a diffuse signal according to some implementations. In some implementations, blocks of process 600 may be performed on a user device. For example, the user device may be a device that enables playback of audio content (or audio content associated with video content) via paired headphones or earbuds. Examples of such user devices include mobile phones, tablet computers, laptop computers, desktop computers, game consoles, televisions, etc. Blocks of process 600 may be performed by one or more processors or controllers of the user device. An example of such a controller is control system 710, which is Figure 7 is shown in and combined below Figure 7 In some implementations, the blocks of process 600 may be performed in accordance with Figure 6 In some implementations, two or more blocks of process 600 may be performed substantially in parallel. In some implementations, one or more blocks of process 600 may be omitted.

[0071] Process 600 may begin at 602 by obtaining one or more diffuse signal modification parameters, an expansion matrix, and a diffuse stereo signal. Note that the diffuse stereo signal may be provided in the frequency domain. Figure 3 As described above, the diffuse stereo signal can be expressed as d 1 (m,k),…d j (m, k), where j is the number of diffuse signals, m is the time block index, and k is the frequency band. The expansion matrix may be denoted herein as O, which may have dimensions N×j, where N is the number of channels. For example, for a 5.1 channel upmix, N may be 5, and j may be 2. In other examples, N may be 7, 9, etc., and j may be 2, 3, 4, etc. In some embodiments, the expansion matrix may be based on statistical estimation parameters (e.g., as estimated by statistical estimation block 304, such as Figure 3 shown in and combined above Figure 3 described) is generated.

[0072] In some implementations, the diffuse signal modification parameters may be generally represented herein as a matrix B, which may have dimensions N×j, similar to the matrix O. Alternatively, in some implementations, the diffuse signal modification parameters may be a single parameter, which is generally represented herein as β. The use of a single parameter may reduce the complexity of allowing a user to configure the diffuse signal modification parameters for various listening scenarios. The diffuse signal modification parameters may be obtained from a memory of a user device and may have been set based on user preferences related to the extent to which diffuse signals are to be allocated to other output channels for various listening scenarios (e.g., for various types of audio content). Note that in some implementations, each element of the matrix B may be between 0 and 1, where a value of 0 indicates that the diffuse signal in a specified channel (e.g., left surround and right surround) is completely reallocated to other output channels (e.g., left, right, and / or center channels), and a value of 1 indicates that the diffuse signal will not be reallocated.

[0073] At 604, process 600 may generate a modified expansion matrix using the expansion matrix and one or more diffuse signal modification parameters. Because the multi-channel output signal should not be attenuated when the diffuse signal is reallocated to other output channels, the modified expansion matrix may be determined using one or more normalization parameters. For example, the modified expansion matrix may be determined by a dot product of a normalization matrix, a matrix representing the diffuse signal modification parameters, and the expansion matrix. For example, in the case where the diffuse signal modification parameters include a plurality of diffuse signal modification parameters (each diffuse signal modification parameter corresponds to a specific output channel and diffuse signal, and is arranged in a matrix B), the modified expansion matrix may be determined by the following formula:

[0074]

[0075] In the equation given above, the elements of the NORM matrix can be determined as values ​​that make the Frobenius norm of O' the same as the Frobenius norm of O.

[0076] As another example, in the case where the diffuse signal modification parameters include a single diffuse signal modification parameter β, the modified spreading matrix may be determined by the following formula:

[0077]

[0078] In the example equations given above, the normalization matrix, the diffuse signal modification matrix, and the expansion matrix may each have dimensions of N times j, where N represents the number of output channels in the upmix and j represents the number of diffuse signals. In some implementations, the normalization matrix may be used to keep the Frobenius norm of the expansion matrix O unchanged. In the equations given above, and for the example case of five output channels (N) and two diffuse signals (j), the value of norm used in the normalization matrix may be determined by the following formula:

[0079]

[0080] At 606, process 600 may generate a diffuse multi-channel signal using the modified expansion matrix, wherein at least a portion of the diffuse stereo signal has been assigned to the output channels. The modified diffuse multi-channel signal is generally denoted herein as Z d1 ,…Z dN , and is conventionally generated by multiplying the expansion matrix by a vector formed by the diffuse signal. However, to generate a modified diffuse multi-channel signal, the process 500 may multiply the modified expansion matrix incorporating the diffuse signal modification parameter by a vector formed by the diffuse signal. For example, in some embodiments, the diffuse multi-channel signal may be determined by:

[0081]

[0082] For example, in the presence of five output channels (N) and two diffuse signals (j), the diffuse multi-channel signal can be determined by:

[0083]

[0084] like Figure 3 and Figure 4 shown in and combined above Figure 3 and Figure 4 As described above, the diffuse multi-channel signal can then be combined with the multi-channel direct signal to generate a multi-channel output signal in the frequency domain. Figure 3 and Figure 4 As described, the frequency domain signal can then be transformed into the time domain.

[0085] Figure 7 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. Figure 1 Sample, Figure 7The types and quantities of the elements shown in are provided only by way of example. Other implementations may include more, less and / or different types and quantities of elements. According to some examples, device 700 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 700 may be or may include a television set, one or more components of an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, an intelligent speaker or another type of equipment.

[0086] According to some alternative implementations, the apparatus 700 may be or may include a server. In some such examples, the apparatus 700 may be or may include an encoder. Thus, in some cases, the apparatus 700 may be a device configured for use within an audio environment such as a home audio environment, while in other cases, the apparatus 700 may be a device configured for use in the "cloud", e.g., a server.

[0087] In this example, the device 700 includes an interface system 705 and a control system 710. The interface system 705 can be configured to communicate with one or more other devices of the audio environment in some implementations. The audio environment can be a home audio environment in some examples. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. The interface system 705 can be configured to exchange control information and associated data with the audio devices of the audio environment in some implementations. The control information and associated data can be related to one or more software applications being executed by the device 700 in some examples.

[0088] The interface system 705 may be configured in some implementations to receive or provide a content stream. The content stream may include audio data. The audio data may include, but may not be limited to, an audio signal. In some cases, the audio data may include spatial data, such as channel data and / or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0089] The interface system 705 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 705 may include one or more wireless interfaces. The interface system 705 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 705 may include a control system 710 and a memory system (such as Figure 7The interface system 705 may be configured to receive input from one or more microphones in the environment.

[0090] The control system 710 may include, for example, a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0091] In some implementations, the control system 710 may reside in more than one device. For example, in some implementations, a portion of the control system 710 may reside in a device within one of the environments depicted herein, and another portion of the control system 710 may reside in a device outside the environment (such as a server, a mobile device (e.g., a smart phone or tablet computer), etc.). In other examples, a portion of the control system 710 may reside in a device within an environment, and another portion of the control system 710 may reside in one or more other devices of the environment. For example, a portion of the control system 710 may reside in a device (such as a server) that implements a cloud-based service, and another portion of the control system 710 may reside in another device (such as another server, a memory device, etc.) that implements a cloud-based service. The interface system 705 may also reside in more than one device in some examples.

[0092] In some implementations, the control system 710 can be configured to at least partially perform the methods disclosed herein.According to some examples, the control system 710 can be configured to implement methods of attenuating diffuse signals, distributing diffuse signals to other output channels, and the like.

[0093] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices, such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 7 710 and / or in the control system 710. Thus, various innovative aspects of the subject matter described in the present disclosure can be implemented in one or more non-transitory media having software stored thereon. The software can, for example, determine or obtain diffuse signal attenuation parameters, generate an output multi-channel signal based on the diffuse signal attenuation parameters, etc. The software can, for example, be executed by a control system such as Figure 7The system is executed by one or more components of the control system 710).

[0094] In some examples, apparatus 700 may include Figure 7 Optional microphone system 720 shown in . Optional microphone system 720 may include one or more microphones. In some implementations, one or more of the microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, device 700 may not include microphone system 720. However, in some such implementations, device 700 may still be configured to receive microphone data of one or more microphones in an audio environment via interface system 710. In some such implementations, a cloud-based implementation of device 700 may be configured to receive microphone data or a noise metric corresponding at least in part to the microphone data from one or more microphones in an audio environment via interface system 710.

[0095] According to some implementations, the apparatus 700 may include Figure 7 Optional speaker system 725 is shown in . Optional speaker system 725 may include one or more speakers, which may also be referred to herein as "speakers", or more generally as "audio reproduction transducers". In some examples (e.g., cloud-based implementations), device 700 may not include speaker system 725. In some implementations, device 700 may include headphones. The headphones may be connected or coupled to device 700 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0096] Some aspects of the present disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0097] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed or otherwise configured) to perform the desired processing on the audio signal, including the execution of one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed and / or otherwise configured using software or firmware to perform any of a variety of operations, including one or more examples of the disclosed method. Alternatively, elements of some embodiments of the inventive system are implemented as a general-purpose processor or DSP that is configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more speakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0098] Another aspect of the disclosure is a computer-readable medium (eg, a disk or other tangible storage medium) storing code for executing (eg, an encoder executable to perform) one or more examples of the disclosed method or steps thereof.

[0099] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be clear to those of ordinary skill in the art that many variations of the embodiments and applications described herein are possible without departing from the scope of the disclosure described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure will not be limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method for processing audio, the method comprising: Receiving stereo audio signals; separating the stereo audio signal into a directional signal and a diffuse signal, wherein the directional signal corresponds to directional content in the stereo audio signal, and wherein the diffuse signal corresponds to background content in the stereo audio signal; determining one or more diffuse signal modification parameters based on a current listening situation, wherein the one or more diffuse signal modification parameters indicate a proportion of the diffuse signal to be reallocated to one or more output channels of an output multi-channel signal or a degree of attenuation to be applied to the diffuse signal; generating the output multi-channel signal based on the steerable signal, the diffuse signal, and the one or more diffuse signal modification parameters; as well as The output multi-channel signal is provided to a virtualizer for rendering into a binaural audio signal for playback on a wearable device. 2 . The method of claim 1 , wherein the one or more output channels include at least one of a left channel, a right channel, or a center channel.

3. The method of claim 1 , wherein generating the output multi-channel signal comprises: Get the expansion matrix; generating a modified spreading matrix using the spreading matrix and the one or more diffusion signal modification parameters; generating a diffuse multi-channel signal using the modified expansion matrix; as well as The output multi-channel signal is generated based on the diffuse multi-channel signal and the steerable signal.

4. The method of claim 3, wherein the one or more diffuse signal modification parameters cause the proportion of the diffuse signal to be redistributed among the one or more output channels, and wherein generating the modified expansion matrix comprises determining a matrix dot product of: a norm associated with the expansion matrix and a matrix representing the diffuse signal modification parameters, a matrix associated with the one or more diffuse signal modification parameters, and the expansion matrix.

5. The method of claim 4, wherein the one or more diffuse signal modification parameters include a diffuse signal redistribution modification parameter indicating a redistribution of the diffuse signal in a multi-channel output, and wherein the norm normalizes the energy of the diffuse signal.

6. The method of claim 3, wherein the one or more diffuse signal modification parameters cause the degree of attenuation to be applied to the diffuse signal, and wherein generating the output multi-channel signal comprises performing energy normalization configured to cause the energy of the output multi-channel signal to be the same as the energy of the stereo audio signal.

7. The method of claim 6, wherein normalizing the energy is performed by one of: an upmixer generating the output multi-channel signal, or the virtualizer.

8. The method of any one of claims 1-7, wherein the current listening context comprises one of the following: a movie content viewing mode, a music listening mode, or a game playing mode.

9. The method of claim 8, wherein the current listening context is a movie content viewing mode, and wherein the one or more diffuse signal modification parameters are in the range of approximately 0.8-1.

10. The method of claim 8, wherein the current listening context is a music listening mode, and wherein the one or more diffuse signal modification parameters are in the range of approximately 0-0.

2.

11. The method of claim 8, wherein the current listening context is a game playing mode, and wherein the one or more diffuse signal modification parameters have values ​​that are less than values ​​associated with a movie content viewing mode.

12. The method of any of claims 1-11, wherein the one or more diffusion signal modification parameters are received from a user of the wearable device.

13. The method of claim 12, wherein the one or more diffuse signal modification parameters are received via a user interface.

14. The method of any of claims 1-13, wherein generating the output multi-channel signal occurs on a companion user device associated with the wearable device, and wherein the virtualizer comprises one or more components executing on the wearable device.

15. The method of claim 14, further comprising transmitting data from the companion user device to the wearable device via a Bluetooth communication protocol.

16. The method of any of claims 1-15, wherein the wearable device comprises one of an earbud or a headset.

17. The method of any of claims 1-16, wherein the wearable device comprises one or more sensors that collect sensor data that can be used to generate head tracking information associated with a wearer of the wearable device.

18. The method of claim 17, wherein the virtualizer is configured to render the binaural audio signal based on the output multi-channel signal and the head tracking information.

19. A system comprising: one or more processors; as well as A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of claims 1-18.

20. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of claims 1-18.

Images