Colorless generation of elevation-aware cues using all-pass filter networks

Abstract

A system includes one or more computing devices that encode spatially aware cues into a monaural channel to generate a plurality of output channels. The computing device determines a target amplitude response for a middle channel and a side channel of the plurality of output channels, thereby defining spatial awareness associated with the one or more frequency dependent phase shifts. The computing device determines a transfer function of the single-input multiple-output all-pass filter based on the target amplitude response, determines coefficients of the all-pass filter based on the transfer function, and processes the mono channel with the coefficients of the all-pass filter to generate a plurality of channels with encoded spatially aware cues. The all-pass filter is configured to be colorless with respect to each output channel, allowing spatial cues to be placed into the audio stream to separate from the overall coloring of the audio.

Description

[0001] This application is a divisional application of a patent application with an application date of July 7, 2022, application number 202280047861.8, and invention name “Colorless Generation of Elevation Perception Cues Using an All-Pass Filter Network”. Technical Field

[0002] The present disclosure relates generally to audio processing, and more particularly to encoding spatial cues into audio content. Background Art

[0003] Audio content may be encoded to include spatial properties of a sound field, thereby allowing a user to perceive a sense of space in the sound field. For example, audio from a particular sound source (e.g., such as speech or an instrument) may be mixed into the audio content in a manner that creates a sense of space associated with the audio (e.g., the perception that the audio is arriving at the user from a particular direction of arrival or is located in a particular type of location (e.g., a small room, a large auditorium, etc.) Summary of the invention

[0004] Some embodiments include a method for encoding spatial cues along a sagittal plane into a monophonic signal to generate multiple resultant channels. The method includes, by a processing circuit, determining a target amplitude response of a mid component or a side component of the multiple resultant channels based on the spatial cues associated with a frequency-dependent phase shift; converting the target amplitude response of the mid component or the side component into a transfer function of a single-input multiple-output all-pass filter; and processing the monophonic signal using the all-pass filter, wherein the all-pass filter is configured based on the transfer function.

[0005] Some embodiments include a system for generating multiple channels from a monophonic channel, wherein the multiple channels are encoded using one or more spatial cues. The system includes one or more computing devices configured to determine a target amplitude response of a mid component or a side component of the multiple channels based on the spatial cues associated with a frequency-dependent phase shift. The one or more computers are further configured to convert the target amplitude response of the mid component or the side component into a transfer function of a single-input multiple-output all-pass filter, and process the monophonic signal using the all-pass filter, wherein the all-pass filter is configured according to the transfer function.

[0006] Some embodiments include a non-transitory computer-readable medium comprising stored instructions for generating multiple channels from a monaural channel, wherein the multiple channels are encoded using one or more spatial cues, which instructions, when executed by at least one processor, configure the at least one processor to: determine a target amplitude response of a mid component or a side component of the multiple resulting channels based on the spatial cues associated with a frequency-dependent phase shift; convert the target amplitude response of the mid component or the side component into a transfer function of a single-input multiple-output all-pass filter; and process the monaural signal using the all-pass filter, wherein the all-pass filter is configured based on the transfer function.

[0007] Some embodiments relate to spatially shifting a portion of audio content (e.g., speech) using a series of Hilbert transforms. Some embodiments include one or more processors and a non-transitory computer-readable medium. The computer-readable medium includes stored program code that, when executed by one or more processors, configures the one or more processors to: separate an audio channel into a low-frequency component and a high-frequency component; apply a first Hilbert transform to the high-frequency component to generate a first left leg component and a first right leg component, the first left leg component being 90 degrees out of phase with respect to the first right leg component; apply a second Hilbert transform to the first right leg component to generate a second left leg component and a second right leg component, the second left leg component being 90 degrees out of phase with respect to the second right leg component; combine the first left branch component with the low-frequency component to generate a left channel; and combine the second right branch component with the low-frequency component to generate a right channel.

[0008] Some embodiments include a non-transitory computer-readable medium containing stored program code. The program code, when executed by one or more processors, configures the one or more processors to: separate an audio channel into a low-frequency component and a high-frequency component; apply a first Hilbert transform to the high-frequency component to generate a first left-branch component and a first right-branch component, the first left-branch component being 90 degrees out of phase with respect to the first right-branch component; apply a second Hilbert transform to the first right-branch component to generate a second left-branch component and a second right-branch component, the second left-branch component being 90 degrees out of phase with respect to the second right-branch component; combine the first left-branch component with the low-frequency component to generate a left channel; and combine the second right-branch component with the low-frequency component to generate a right channel.

[0009] Some embodiments include a method performed by one or more processors. The method includes: separating an audio channel into a low-frequency component and a high-frequency component; applying a first Hilbert transform to the high-frequency component to generate a first left-branch component and a first right-branch component, the first left-branch component being 90 degrees out of phase with respect to the first right-branch component; applying a second Hilbert transform to the first right-branch component to generate a second left-branch component and a second right-branch component, the second left-branch component being 90 degrees out of phase with respect to the second right-branch component; combining the first left-branch component with the low-frequency component to generate a left channel; and combining the second right-branch component with the low-frequency component to generate a right channel. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of an audio processing system according to some embodiments.

[0011] Figure 2 is a block diagram of a computing system environment according to some embodiments.

[0012] Figure 3 Illustrated is a graph showing sampled HRTFs measured at an elevation angle of 60 degrees, in accordance with some embodiments.

[0013] Figure 4 A graph showing an example of perceptual cues characterized by a target amplitude function corresponding to a narrow region of infinite attenuation at 11 kHz is illustrated in accordance with some embodiments.

[0014] Figure 5 Illustrated is a frequency graph generated by driving a second order all pass filter section having the coefficients shown in Table 1 with white noise, according to some embodiments.

[0015] Figure 6 is a block diagram of a PSM module implemented using a Hilbert transform according to one or more embodiments.

[0016] Figure 7 is a block diagram of a Hilbert transform module according to one or more embodiments.

[0017] Figure 8 The diagram illustrates driving a signal with white noise according to some embodiments. Figure 6 A frequency diagram generated by the HPSM module shows the output frequency response of the sum (middle) and difference (side) of multiple channels.

[0018] Fig. 9 is a block diagram of a PSM module implemented using a FNORD filter network according to some embodiments.

[0019] Fig. 10A is a detailed block diagram of a PSM module 900 according to some embodiments.

[0020] Fig. 10B is a block diagram of a wideband phase rotator implemented within an all-pass filter module of a PSM module according to some embodiments.

[0021] Fig.11 A frequency response graph is illustrated showing the output frequency response of a FNORD filter network configured to achieve a magnitude response for a 60 degree vertical cue, in accordance with some embodiments.

[0022] Fig.12 is a block diagram of an audio processing system 1000 according to one or more embodiments.

[0023] Fig.13A is a block diagram of a quadrature component generator according to one or more embodiments.

[0024] Fig. 13B is a block diagram of a quadrature component generator according to one or more embodiments.

[0025] Fig. 13C is a block diagram of a quadrature component generator according to one or more embodiments.

[0026] Fig.14A is a block diagram of a quadrature component processor module according to one or more embodiments.

[0027] Fig. 14B A block diagram of a quadrature component processor module is illustrated in accordance with one or more embodiments.

[0028] Fig.15 is a block diagram of a sub-band spatial processor module according to one or more embodiments.

[0029] Fig.16 is a block diagram of a crosstalk compensation processor module according to one or more embodiments.

[0030] Fig.17 is a block diagram of a crosstalk simulation processor module according to one or more embodiments.

[0031] Fig.18 is a block diagram of a crosstalk cancellation processor module according to one or more embodiments.

[0032] Fig.19 is a flow chart of a process for PSM processing using a Hilbert Transform Perceptual Sound Field Modification (HPSM) module according to one or more embodiments.

[0033] Fig. 20 is a flow chart of another process for PSM processing using a first-order non-orthogonal rotation-based decorrelation (FNORD) filter network in accordance with some embodiments.

[0034] Fig.21 is a flow chart of a process for spatial processing using at least one of a super mid component, a residual mid component, a super side component, or a residual side component according to one or more embodiments.

[0035] Fig. 22 is a flow chart of a process for performing sub-band spatial processing and crosstalk processing compensation using at least one of a super mid component, a residual mid component, a super side component, or a residual side component according to one or more embodiments.

[0036] Fig.23 is a block diagram of a computer according to some embodiments.

[0037] The drawings depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0038] The drawings and the following description relate to the preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily identified as viable alternatives that can be employed without departing from the principles claimed.

[0039] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. Note that, wherever practicable, like or similar reference numerals may be used in the accompanying drawings and may indicate like or similar functionality. The accompanying drawings depict embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0040] Encoding spatial perception cues into a monophonic audio source may be desirable in various applications involving the presentation of multiple simultaneous streams of audible content. Examples of such applications include:

[0041] Conferencing use cases – where adding spatial awareness cues applied to one or more remote talkers can help improve overall speech intelligibility and enhance the overall immersion of the listener.

[0042] Video and music playback / streaming use cases where one or more audio channels or signal components of one or more audio channels can be enhanced by adding spatial awareness cues to improve the clarity or spatial perception of speech or other elements of the mix.

[0043] Co-viewing entertainment use cases – where the streams are individual content channels, such as one or more remote speakers and entertainment program material, that must be mixed together to form an immersive experience, and applying spatial perception cues to one or more elements can increase the sense of perceived difference between the mixed elements, broadening the listener’s perceptual bandwidth.

[0044] Embodiments are directed to an audio system that modifies the perceived spatial quality (e.g., soundstage and overall position relative to a target listener's head) of one or more audio channels. In some embodiments, modifying the perceived spatial quality of an audio channel can be used to decouple the coloration of a particular source from its perceived spatial position, and / or to reduce the number of amplifiers and speakers required to encode such an effect.

[0045] The audio signal processing performed by the audio system is called the Perceptual Sound Field Modification (PSM) processing. The perceptual result of the PSM processing is referred to as spatial displacement in this article. Users usually experience the psychoacoustic effect as the overall displacement of the sound source above, around or towards the head, thereby perceptually distinguishing the sound source from the other parts of the audio content. This psychoacoustic effect originates from the phase and time relationship between the left channel and the right channel, and is enhanced by an all-pass filter and a delay network. In some embodiments, the filter and delay network can be implemented as one or more second-order all-pass parts, such as a series of Hilbert transforms, or a first-order non-orthogonal rotation-based decorrelation (FNORD) filter network, each of which will be described in more detail below. The perceptual result of the PSM processing may vary depending on different listening configurations (e.g., headphones or speakers, etc.). For some content and algorithm configurations, the result may also produce the impression that the perceptual signal propagates (e.g., diffuses) around the listener's head. For mono input signals (e.g., non-spatial audio signals), the diffusion effect of the PSM processing can be used for upmixing from mono to stereo.

[0046] In some embodiments, an audio system may isolate a target portion of an audio signal from a residual portion of the audio signal, apply various configurations of PSM processing to perceptually shift the target portion, and mix the processed result with the residual portion (e.g., which may be unprocessed or processed in a different manner). Such a system may be understood to clarify, enhance, or otherwise distinguish the target portion in an overall audio mix. In some embodiments, PSM processing is used to perceptually shift a portion of an audio signal that includes sung or spoken speech. By convention, speech in a television, movie, or music audio stream is typically located in the center of the sound field and is therefore part of a middle component (also referred to as a non-spatial or correlated component) of a stereo or multi-channel audio signal. Therefore, PSM processing can be applied to a middle component or a super-middle component of an audio signal that includes the spectral energy of a side component (also referred to as a spatial or non-correlated component) removed from the spectral energy of the middle component.

[0047] PSM processing can be combined with other types of processing. For example, the audio system can apply processing to the shifted portion of the audio signal to perceptually transform the shifted portion and distinguish it from other components within the mix. These additional types of processing can include one or more of single-band or multi-band equalization, single-band or multi-band dynamic processing (e.g., limiting, compression, expansion, etc.), single-band or multi-band gain or delay, crosstalk processing (e.g., crosstalk cancellation and / or crosstalk simulation processing), or compensation for crosstalk processing. In some embodiments, PSM processing can be performed together with mid-processing / side processing, such as sub-band spatial processing, in which the sub-bands of the mid and side components of the audio signal generated via PSM processing are gain adjusted to enhance the spatial sense of the sound field.

[0048] Isolation of audio channels for PSM processing can be achieved in various ways. In some embodiments, PSM processing can be performed on spectrally orthogonal sound components (such as super-mid components of an audio signal). In other embodiments, PSM processing is performed on audio channels associated with a sound source (e.g., speech), and the processed channels are subsequently mixed with other audio content (e.g., background music).

[0049] Although the following discussion focuses primarily on upmixing a mono signal to stereo (i.e., two output channels), since most audio rendering equipment is stereo, it should be understood that the discussed techniques can be easily generalized to include a greater number of channels. Stereo embodiments can be discussed in terms of mid / side processing, where the phase difference between the left and right channels becomes a complementary region of amplification and attenuation in the mid / side space.

[0050] Audio Processing System Example

[0051] Figure 1 is a block diagram of an audio processing system 100 according to one or more embodiments. The system 100 uses PSM processing to spatially shift an audio signal and applies other types of spatial (e.g., mid / side) processing. Some embodiments of the system 100 have different components than those described herein. Similarly, in some cases, functionality may be allocated between components in a manner different than that described herein.

[0052] The system 100 includes a PSM module 102, an L / R to M / S converter module 104, a component processor module 106, an M / S to L / R converter module 108, and a crosstalk processor module 108. The PSM module 102 receives input audio 120 and generates spatially shifted left and right channels 122 and 124. Figure 6-Figure 11 The operation of PSM 102 according to various embodiments is described in more detail.

[0053] The L / R to M / S converter module 104 receives the left channel 122 and the right channel 124 and generates a middle component 126 (e.g., a non-spatial component) and a side component 128 (e.g., a spatial component) based on the channels 122 and 124. In some embodiments, the middle component 126 is generated based on the sum of the left channel 122 and the right channel 122, and the side component 128 is generated based on the difference between the left channel 122 and the right channel 124. In some embodiments, the transformation of a point in the L / R space to a point in the M / S space can be expressed as follows according to equation (1):

[0054]

[0055] The inverse transform can be expressed according to equation (2) as follows:

[0056]

[0057] It should be understood that in other embodiments, other L / R to M / S type transforms may be used to generate the middle component 126 and the side component 128. In some embodiments, due to lower computational complexity, the transforms shown in equations (1) and (2) may be used instead of the true orthogonal form, where both the forward and inverse transforms are scaled by √2. For ease of discussion, no matter which particular transform is used, the convention of transforming the row vector coordinates by multiplication on the right side, and the sign of the transformed coordinates with their basis as the label above them, will be used, as shown in equation (3) below:

[0058]

[0059] The component processor module 106 processes the mid component 126 to generate a processed mid component 130 and processes the side component 128 to generate a processed side component 314. Processing of each of the components 126 and 128 may include various types of filtering, such as spatial cue processing (e.g., amplitude or delay based panning, binaural processing, etc.), single-band or multi-band equalization, single-band or multi-band dynamics processing (e.g., compression, expansion, limiting, etc.), single-band or multi-band gain or delay stages, adding audio effects, or other types of processing. In some embodiments, the component processor module 106 uses the mid component 126 and the side component 128 to perform sub-band spatial processing and / or crosstalk compensation processing. Sub-band spatial processing is processing performed on frequency sub-bands of the mid component and the side component to spatially enhance the audio signal. Crosstalk compensation processing is processing that adjusts for spectral artifacts caused by crosstalk processing, such as crosstalk cancellation for speakers or crosstalk simulation for headphones. Reference will be made to Figure 15-16 Various components that may be included in the component processor module 106 are further described.

[0060] The M / S to L / R converter module 108 receives the processed middle component 130 and the processed side component 132 and generates a processed left component 134 and a processed right component 136. In some embodiments, the M / S to L / R converter module 108 transforms the processed middle component 130 and the side component 132 based on the inverse of the transformation performed by the L / R to M / S converter module 104, e.g., the processed left component 134 is generated based on the sum of the processed middle component 130 and the processed side component 132, and the processed right component 136 is generated based on the difference between the processed middle component 130 and the processed side component 132. Other M / S to L / R types of transformations may be used to generate the processed left component 134 and the processed right component 136.

[0061] The crosstalk processor module 110 receives the processed left component 134 and the processed right component 136 and performs crosstalk processing thereon. Crosstalk processing includes, for example, crosstalk simulation or crosstalk cancellation. Crosstalk simulation is processing performed on an audio signal (e.g., output via headphones) in order to simulate the effect of a speaker. Crosstalk cancellation is processing performed on an audio signal (e.g., output via a speaker) in order to reduce crosstalk caused by a speaker. The crosstalk processor module 110 outputs a left channel 138 and a right output channel 140. In some embodiments, crosstalk processing (e.g., simulation or cancellation) may be performed before component processing, such as before the left channel 122 and the right channel 124 are converted into a mid component and a side component. Fig.17 and Fig.18 Various components that may be included in the crosstalk processor module 110 are further described.

[0062] In some embodiments, PSM module 102 is incorporated into component processor module 106. L / R to M / S converter module 104 receives left and right channels, which can represent (e.g., stereo) input of audio processing system 100. L / R to M / S converter module 104 uses left and right input channels to generate middle component and side component. PSM module 102 of component processor module 106 processes middle component and / or side component as input, such as discussed herein for input audio 102, to generate left and right channels. Component processor module 106 can also perform other types of processing to middle component and side component, and M / S to L / R converter module 108 generates left and right channels according to processed middle component and side component. The left channel generated by PSM module 102 is combined with the left channel generated by M / S to L / R converter module 108 to generate processed left component. The right channel generated by the PSM module 102 is combined with the right channel generated by the M / S to L / R converter module 108 to generate a processed right component.

[0063] The system 100 provides a left channel 138 to a left speaker 112 and a right channel 140 to a right speaker 114. The speakers 112 and 114 may be components of a smartphone, a tablet computer, a smart speaker, a laptop computer, a desktop computer, a fitness machine, etc. The speakers 112 and 114 may be part of a device including the system 100 or may be separate from the system 100, such as connected to the system 100 via a network. The network may include wired and / or wireless connections. The network may include a local area network, a wide area network (e.g., including the Internet), or a combination thereof.

[0064] Figure 2 2 is a block diagram of a computing system environment 200 according to some embodiments. The computing system 200 may include an audio system 202, which may include one or more computing devices (e.g., servers) connected to user devices 210a and 210b via a network 208. The audio system 202 provides audio content to the user devices 210a and 210b (also individually referred to as user devices 210) via the network 208. The network 208 facilitates communication between the system 202 and the user devices 210. The network 106 may include various types of networks, including the Internet.

[0065] The audio system 202 includes one or more processors 204 and a computer readable medium 206. The one or more processors 204 execute program modules that cause the one or more processors 204 to perform functionality, such as generating multiple output channels from a mono channel. The (multiple) processors 204 may include one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a controller, a state machine, other types of processing circuits, or one or more of these combinations. The processor 204 may also include local memory to store program modules, operating system data, and the like.

[0066] The computer readable medium 206 is a non-transitory storage medium that stores program code for the PSM module 102, the component processor module 106, the crosstalk processor module 110, the L / R and M / S conversion modules 104 and 108, and the channel summing module 212. The PSM module 102 generates a plurality of output channels from a mono channel, which may be further processed using the component processor module 106, the crosstalk processor module 110, and / or the L / R and M / S conversion modules 104 and 108. The system 202 provides the plurality of output channels to a user device 210a, which includes a plurality of speakers 214 to render each of the output channels.

[0067] The channel summing module 212 generates a mono output channel by adding together multiple output channels generated by the PSM module 102 and / or other modules. The system 202 provides the mono output channel to the user device 210b, which includes a single speaker 216 for presenting the mono output channel. In some embodiments, the channel summing module 212 is located at the user device 210b. The audio system 202 provides multiple output channels to the user device 210b, and the user device 210b converts the multiple channels into a mono output channel for the speaker 216. The user device 210 presents audio content to the user. The user device 210 can be a computing device of the user, such as a music player, a smart speaker, a smart phone, a wearable device, a tablet computer, a notebook computer, a desktop computer, etc.

[0068] Middle / side space coloring

[0069] In some embodiments, spatial cues are encoded into the audio signal by creating a coloration effect in the middle / side space while avoiding it in the left / right. In some embodiments, this is achieved by applying an all-pass filter in the left / right space that has properties specifically selected to produce the target coloration in the middle / side. For example, in a two-channel system, the relationship between the left / right phase angle and the middle / side gain can be expressed as follows using equation (4):

[0070]

[0071] in is a two-dimensional row vector consisting of the mid and side target gain factors at a specific frequency ω in decibels, and is the objective function of the phase relationship between the left channel and the right channel. According to the following equations (5) and (6), for Solving equation (4) provides the desired frequency-dependent phase difference applied in the left / right space:

[0072]

[0073] Note that if the constraint that the system is colorless in left / right space is applied, only the transfer function of the middle component or the side component can be specified. Thus, the system of equations (5) and (6) is overdetermined, where only one of the above equations can be solved without destroying the required symmetry. In some embodiments, the selection of a particular equation yields control over the middle component or the side component. If the constraint that the system is colorless in left / right space is discarded, additional degrees of freedom can be achieved. In systems with more than two channels, different techniques (such as paired or layered sum and difference transforms) can be used to replace middle and side.

[0074] Example all-pass filter implementation for encoding elevation cues

[0075] In some embodiments, spatial perceptual cues can be encoded into an audio signal by embedding frequency-dependent amplitude cues (i.e., coloration) into the mid / side space while constraining the left / right signals to be colorless. For example, this framework can be used to encode elevation cues (e.g., spatial perceptual cues that are positioned along the sagittal plane) because the left / right cues for elevation are theoretically symmetric in coloration.

[0076] In some embodiments, a distinct feature of the elevation cue based on the head-related transfer function (HRTF) is a notch starting at about 8 kHz and rising monotonically to about 16 kHz as a function of elevation, which can be used to derive appropriate coloring of the intermediate channel with which the elevation is encoded. Using this intermediate encoding cue, a corresponding frequency-dependent phase shift can be derived, which can be further used to derive functions to be implemented via a filter network (e.g., PSM module 102), such as those described below. In some embodiments, the HRTF-based elevation cue can be characterized as a notch starting at about 8 kHz and rising monotonically to about 12 kHz as a function of elevation.

[0077] For ease of discussion, the following example filter framework is discussed with respect to encoding the same perceptual cues according to some embodiments, where the target angle of elevation is 60 degrees (e.g., spatially shifting the audio content 60 degrees above the horizontal plane in the sagittal plane), but it should be understood that in other embodiments, similar techniques may be used to encode perceptual cues having different elevation angles. Figure 3 Illustrated is a graph showing sampled HRTFs measured at an elevation angle of 60 degrees, in accordance with some embodiments. Figure 4 A graph is illustrated showing an example of a perceptual cue characterized by a target amplitude function corresponding to a narrow region of infinite attenuation at approximately 11 kHz, according to some embodiments. Such cues can be used to produce a perception of elevation angle that is consistent with most people in a variety of presentation scenarios. Although Figure 4 The graph of illustrative examples illustrates a simplification of sampled HRTFs, but it should be understood that more complex cues may also be derived based on the framework described herein.

[0078] Design using a second-order all-pass section

[0079] In some embodiments, the PSM module 102 is implemented using two independent cascaded second-order all-pass filters plus a delay element to achieve the desired phase shift in the left / right space to encode perceptual cues, such as those described above with respect to Figure 4 In some embodiments, the second order section is implemented as a biquad section, where coefficients are applied to feedback and feedforward taps of up to two delayed samples. As discussed herein, a convention is used to name the feedback coefficients of one and two samples A1 and A2, respectively, and the feedforward coefficients of zero, one, and two samples B0, B1, and B2, respectively.

[0080] In some embodiments, the PSM module 102 is implemented using a second order all-pass filter configured to perform pole and zero elimination to allow the amplitude component of the transfer function to remain flat while changing the phase response. By using an all-pass filter section on both channels in the left / right space, a specific phase shift across the spectrum can be guaranteed. Another benefit of this is that a given phase offset between left and right is allowed in addition to the desired zero in the middle / side space, which will result in an increase in the sense of spatial extension.

[0081] Table 1 below shows an example set of biquad coefficients that can be used in a second-order all-pass filter framework with an additional 2-sample delay on the right channel according to some embodiments. The biquad coefficients shown in Table 1 can be designed for a 44.1kHz sampling rate, but can also be used for systems with other sampling rates (e.g., 48kHz) sampling rates.

[0082]

[0083] Table 1

[0084] A filter network with the coefficients shown in Table 1 can produce an appropriate phase response in the left / right space, which results in a noticeable null / amplification in the middle / side space at 11 kHz. Figure 5 Illustrated is a frequency graph generated by driving a second order all pass filter section having coefficients shown in Table 1 with white noise, showing the output frequency response of the sum 502 (middle) and the difference 504 (side) of multiple channels, according to some embodiments.

[0085] In some embodiments, the PSM module 102 implemented using a second-order all-pass filter section can be further enhanced by using a crossover network to exclude processing of frequency regions that do not require it. The use of a crossover network can increase the flexibility of the embodiment by allowing further processing of perceptually significant cues to exclude unnecessary auditory data.

[0086] In some embodiments, the PSM module 102 implemented using a second order all-pass filter portion may be implemented using a network of serially linked Hilbert transforms, as will be described in more detail below.

[0087] Example Hilbert Transform Perceptual Sound Field Modification (HPSM) module

[0088] Figure 6 6 is a block diagram of a PSM module implemented using a Hilbert transform according to one or more embodiments. The PSM module 600 may also be referred to as a Hilbert transform-perceived sound field modification (HPSM) module, which applies a network of serially linked Hilbert transforms to input audio 602 (which may correspond to Figure 1 ) to perceptually shift the input audio 602.

[0089] Module 600 includes a cross network module 604, a gain unit 610, a gain unit 612, a Hilbert transform module 614, a Hilbert transform module 620, a delay unit 626, a gain unit 628, a delay unit 630, a gain unit 632, an addition unit 634, and an addition unit 636. Some embodiments of module 600 have different components than those described herein. Similarly, in some cases, functionality may be allocated between components in a manner different than that described herein.

[0090] The crossover network module 604 receives the input audio 602 and generates a low frequency component 606 and a high frequency component 608. The low frequency component includes a subband of the input audio 602 having lower frequencies than a subband of the high frequency component 608. In some embodiments, the low frequency component 606 includes a first portion of the input audio including low frequencies, and the high frequency component 608 includes the remaining portion of the input audio including high frequencies.

[0091] As discussed in more detail below, the high frequency component 608 is processed using a series of Hilbert transforms, while the low frequency component 606 bypasses the series of Hilbert transforms, and then the low frequency component and the processed high frequency component 608 are recombined. The crossover frequency between the frequency component 606 and the high frequency component 608 can be adjustable. For example, more frequencies can be included in the high frequency component 608 to increase the perceived strength of the spatial shift of the HPSM module 600, while more frequencies can be included in the low frequency component 606 to reduce the perceived strength of the shift. In another example, the crossover frequency is set so that the frequency of the sound source of interest (e.g., speech) is included in the high frequency component 608.

[0092] The input audio 602 may include a monophonic channel or may be a mix of a stereo signal or other multi-channel signal (e.g., surround sound, surround stereo, etc.). In some embodiments, the input audio 602 is audio content associated with a sound source to be incorporated into an audio mix. For example, the input audio 602 may be speech processed by the module 600, and the processed result is combined with other audio content (e.g., background music) to generate an audio mix.

[0093] Gain unit 610 applies gain to low frequency component 606, and gain unit 612 applies gain to high frequency component 608. Gain units 610 and 612 can be used to adjust the overall level of low frequency component 606 and high frequency component 608 relative to each other. In some embodiments, gain unit 610 or gain unit 612 can be omitted from module 600.

[0094] The Hilbert transform modules 614 and 620 apply a series of Hilbert transforms to the high frequency components 608. The Hilbert transform module 614 applies the Hilbert transform to the high frequency components 608 to generate a left branch component 616 and a right branch component 618. The left branch component 616 and the right branch component 618 are audio components that are 90 degrees out of phase with respect to each other. In some embodiments, the left branch component 616 and the right branch component 618 are out of phase with respect to each other at an angle other than 90 degrees, such as between 20 degrees and 160 degrees.

[0095] The Hilbert transform module 620 applies the Hilbert transform to the right branch component 618 generated by the Hilbert transform module 614 to generate the left branch component 122 and the right branch component 624. The left branch component 622 and the right branch component 624 are audio components that are 90 degrees out of phase with respect to each other. In some embodiments, the Hilbert transform module 620 generates the right branch component 624 without generating the left branch component 122. In some embodiments, the left branch component 622 and the right branch component 624 are out of phase with respect to each other at an angle other than 90 degrees, such as between 20 degrees and 160 degrees.

[0096] In some embodiments, each of the Hilbert transform modules 614 and 620 is implemented in the time domain and includes a cascade of all-pass filters and delays, as shown below in conjunction with Figure 7 In other embodiments, Hilbert transform modules 614 and 620 are implemented in the frequency domain.

[0097] Delay unit 626, gain unit 628, delay unit 630, and gain unit 632 provide tuning controls for manipulating the perceived results of the process of module 600. Delay unit 626 applies a time delay to the left branch component 616 generated by the Hilbert transform module 614. Gain unit 628 applies a gain to the left branch component 616. In some embodiments, delay unit 626 or gain unit 628 may be omitted from module 600.

[0098] The delay unit 630 applies a time delay to the right branch component 624 generated by the Hilbert transform module 620. The gain unit 632 applies a gain to the right branch component 624. In some embodiments, the delay unit 630 or the gain unit 632 may be omitted from the module 600.

[0099] The summing unit 634 combines the low frequency component 606 with the left branch component 616 to generate the left channel 642. The left branch component 616 is the output from the first Hilbert transform module 614 in the series. The left branch component 616 may include a delay applied by a delay unit 626 and a gain applied by a gain unit 628.

[0100] The summing unit 636 combines the low frequency component 606 with the right branch component 624 to generate the right channel 644. The right branch component 624 is the output from the second Hilbert transform module 620 in the series. The right branch component 624 may include a delay applied by a delay unit 626 and a gain applied by a gain unit 628.

[0101] Figure 7700 according to one or more embodiments. The Hilbert transform module 700 is an example of the Hilbert transform module 614 or the Hilbert transform module 620. The Hilbert transform module 700 receives an input component 702 and uses the input component 702 to generate a left branch component 712 and a right branch component 724. Embodiments of the Hilbert transform module 700 have different components than those described herein. Similarly, in some cases, functionality may be allocated between components in a manner different from that described herein.

[0102] The Hilbert transform module 700 includes an all-pass filter cascade module 740 for generating a left branch component 712 and a delay unit 714 and an all-pass filter cascade module 742 for generating a right branch component 724. The all-pass filter cascade module 714 includes a series of all-pass filters 704, 706, 708 and 710. The delay unit 714 applies a time delay to the input component 702. The all-pass filter cascade module 742 includes a series of all-pass filters 716, 718, 720 and 722. Each of the all-pass filters 704 to 710 and 716 to 722 passes through frequencies with equal gain while changing the phase relationship between different frequencies. In some embodiments, each all-pass filter 704 to 710 and 716 to 722 is a biquad filter as defined by equation (7):

[0103]

[0104] Where z is a complex variable, and a0, a1, a2, b0, b1, and b2 are digital filter coefficients. Different biquad filters may include different coefficients to apply different phase changes.

[0105] All-pass filter cascade modules 740 and 742 can each include a different number of all-pass filters. Hilbert transform module 700 is an eighth-order filter with eight all-pass filters, and each of left branch component 712 and right branch component 724 has four. In other embodiments, Hilbert transform module 700 is an eighth-order filter (e.g., four all-pass filters for each of all-pass filter cascade modules 740 and 742) or a sixth-order filter (e.g., three all-pass filters for each of all-pass filter cascade modules 740 and 742).

[0106] As above combined Figure 6As discussed, module 600 includes a series of Hilbert transform modules 614 and 620. Using Hilbert transform module 700 for each of Hilbert transform modules 614 and 620, left branch component 616 is generated by one all-pass filter cascade module 742 applied to high frequency component 608. Right branch component 624 is generated by two delay units 714 and two all-pass filter cascade modules 742 by passing through Hilbert transform module 700 twice. In some embodiments, Hilbert transform modules 614 and 620 may be different. For example, Hilbert transform modules 614 and 620 may include filters of different orders, such as an eighth order filter for one of the Hilbert transform modules and a sixth order filter for the other of the Hilbert transform modules.

[0107] Using the Hilbert transform module 700 for the Hilbert transform modules 614 and 620, the right branch component 624 includes a phase and delay relationship with the right branch component 618 created by the all-pass filter and the delay of the Hilbert transform module 620. The right branch component 624 also includes a phase and delay relationship with the high frequency component 608 created by the all-pass filter and the delay of the Hilbert transform modules 614 and 620. In some embodiments, the Hilbert transform module 620 uses the left branch component 616 instead of the right branch component 618 to generate the left branch component 622 and the right branch component 624. This results in the right branch component 624 having a phase and delay relationship with the high frequency component 608 created by the all-pass filter (e.g., no delay) of the Hilbert transform 614 and the all-pass filter and delay of the Hilbert transform module 620.

[0108] Figure 8 The diagram illustrates a method of driving a signal with white noise (eg Figure 6 , which shows the output frequency response of the sum 802 (middle) of multiple channels and the difference 804 (side) of multiple channels.

[0109] like Figure 8 As shown in , while this filter does produce the desired perceptual cues in the region of about 11 kHz, it also imparts additional coloration in the middle and sides of the lower frequencies. In some embodiments, this can be accomplished by placing a crossover network (such as Figure 6 The crossover network module 604 shown in FIG. 6 is applied to the input audio for correction so that the HPSM module only processes audio data within the desired frequency range (e.g., high frequency components), or directly removes the pole / zero pairs corresponding to the spectral transformation region.

[0110] Design using first-order non-orthogonal rotation-based decorrelation (FNORD)

[0111] In some embodiments, a first-order non-orthogonal rotation-based decorrelation (FNORD) filter network may be used to achieve a similar perceptual effect. Fig. 9 is a block diagram of a PSM module 900 implemented using a FNORD filter network according to some embodiments. The PSM module 900 may correspond to Figure 1 1, the PSM module 900 provides decorrelation of a monophonic channel into multiple channels and includes an amplitude response module 902, an all-pass filter configuration module 904, and an all-pass filter module 906. The PSM module 900 processes a monophonic input channel x(t) 912 to generate multiple output channels, such as channel y provided to a speaker 910a. a (t) and channel y provided to speaker 910b b (t) (which may correspond to Figure 1 114). Although two output channels are shown, the PSM module 900 can generate any number of output channels (each output channel is referred to as a channel y(t)). The PSM module 900 can be implemented as part of a computing device such as a music player, a speaker, a smart speaker, a smart phone, a wearable device, a tablet computer, a laptop computer, a desktop computer, etc. Fig. 9 The PSM module 900 is illustrated as including an amplitude response module 902 and a filter configuration module 904 in addition to an all-pass filter module 906, but in some embodiments, the PSM module 900 may include an all-pass filter module 906, wherein the amplitude response module 902 and / or the filter configuration module 904 are implemented separately from the PSM module 900.

[0112] The amplitude response module 902 determines a target amplitude response that defines one or more spatial cues to be encoded into the output channel y(t) (e.g., to be encoded into the mid and side components of the output channel y(t)). The target amplitude response is defined by a relationship between amplitude values ​​and frequency values ​​of a channel (e.g., the mid and side components of the channel), such as amplitude as a function of frequency. In some embodiments, the target amplitude response defines one or more spatial cues on the channel, which may include a target broadband attenuation, a target sub-band attenuation, a critical point, a filter characteristic, or a sound field position. The amplitude response module 902 may receive data 914 and a monophonic channel x(t) 912 and use these inputs to determine the target amplitude response. The data 914 may include information such as characteristics of the spatial cues to be encoded, characteristics of a rendering device (e.g., one or more speakers), the intended content of the audio data, or the perceptual capabilities of a listener in context. In some embodiments, the monophonic channel x(t) 912 may correspond to Figure 1The audio input 120 illustrated in FIG. 1 , or a portion of the audio input (eg, a high frequency component of the input audio, such as Figure 6 ). In embodiments where a mono channel x(t) 912 corresponds to a portion of the audio input, the output channel y(t) may be coupled to a channel corresponding to the remainder of the audio input (e.g., coupled to a channel such as Figure 6 The low-frequency components (shown in FIG. 1 ) are combined to generate a combined output channel.

[0113] The target broadband attenuation is a specification for the attenuation across all frequencies. The target subband attenuation is a specification for the amplitude of the frequency range defined by the subband. The target amplitude response may include one or more target subband attenuation values, each for a different subband.

[0114] A critical point is a specification of the curvature of the target amplitude response of the filter, which is described as the frequency value at which the gain of one of the output channels (e.g., the side component of the output channel) is at a predefined value, such as -3dB or -∞dB. The placement of this point may have a global impact on the curvature of the target amplitude response. An example of a critical point corresponds to a frequency at which the target amplitude response is -∞dB. Since the behavior of the target amplitude response is to eliminate signals at frequencies near this point, the critical point is a zero point. Another example of a critical point corresponds to a frequency at which the target amplitude response is -3dB. Since the behavior of the target amplitude responses of the sum channel and the difference channel (e.g., the mid component and the side component of the channel) intersect at this point, the critical point is a crossover point.

[0115] Filter characteristics are parameters that specify how the mid and side components of a channel are filtered. Examples of filter characteristics include high-pass filter characteristics, low-pass characteristics, band-pass characteristics, or band-stop characteristics. The filter characteristics describe the shape of the resulting sum as if it were the result of equalization filtering. Equalization filtering can be described in terms of what frequencies are allowed to pass through the filter or what frequencies are rejected. Thus, a low-pass characteristic allows frequencies below the inflection point to pass and attenuates frequencies above the inflection point. A high-pass characteristic does the opposite, allowing frequencies above the inflection point to pass and attenuating frequencies below the inflection point. A band-pass characteristic allows frequencies in a frequency band around the inflection point to pass and attenuates other frequencies. A band-stop characteristic suppresses frequencies in a frequency band around the inflection point and allows other frequencies to pass.

[0116] The target magnitude response may define more than one spatial cue to be encoded into the output channel y(t). For example, the target magnitude response may define a spatial cue specified by a critical point and filter characteristics of a mid-component or a side component of an all-pass filter. In another example, the target magnitude response may define a spatial cue specified by a target broadband attenuation, a critical point, and filter characteristics. Although discussed as independent specifications, for most regions of the parameterized space, the specifications may be dependent on each other. This result may be due to the system being nonlinear with respect to phase. To address this issue, additional higher-level descriptors of the target magnitude response may be designed that are nonlinear functions of the target magnitude response parameters.

[0117] The filter configuration module 904 determines the properties of the single-input multiple-output all-pass filter based on the target amplitude response received from the amplitude response module 902. In particular, the filter configuration module determines the transfer function of the all-pass filter based on the target amplitude response and determines the coefficients of the all-pass filter based on the transfer function. The all-pass filter is a decorrelation filter that encodes spatial cues described according to the target amplitude response and is applied to the monophonic input channel x(t) to generate the output channel y a (t) and y b (t).

[0118] The all-pass filter can include different configurations and parameters based on spatial cues and / or constraints defined by a target amplitude response. A filter with a target amplitude response that encodes spatial cues can be colorless, for example, preserving (e.g., completely) the spectral content of individual output channels (e.g., left / right output channels). In this way, the filter can be used to encode elevation cues by embedding coloration into the mid / side space in the form of frequency-dependent amplitude cues, while preserving the spectral content of the left and right signals. Because the filter is colorless, monophonic content can be placed at a specific location in the sound field (e.g., specified by a target elevation angle), where the spatial placement of the audio is decoupled from its overall coloration.

[0119] Fig. 10A and Fig. 10B is a block diagram of an example PSM module based on a first-order non-orthogonal rotation-based decorrelation (FNORD) technique in accordance with some embodiments. Fig. 10A shows a detailed view of a PSM module 900 according to some embodiments, and Fig. 10B A more detailed view of the wideband phase rotator 1004 within the all-pass filter module 906 of the PSM module 900 is provided in accordance with some embodiments.

[0120] like Fig. 10A As shown in FIG. 1 , the all-pass filter module 906 receives a mono input audio signal x(t) 912 and a rotation control parameter θbf 1048 and the first-order coefficient β bf 1050. Input audio signal x(t) 912 and rotation control parameter θ bf 1048 is utilized by the broadband phase rotator 1004, which uses the rotation control parameter θ bf 1048 processes the input audio signal 912 to generate a left wideband rotated component 1020 and a right wideband rotated component 1022. The left wideband rotated component 1020 is then provided to a narrowband phase rotator 1024 for further processing, while the right wideband rotated component 1022 is provided as an output channel y of the PSM module 900 according to some embodiments. b (t) (eg, as a right output channel). The narrowband phase rotator 1024 receives the left wideband rotation component 1020 from the wideband phase rotator 1004 and the first-order coefficients β from the filter configuration module 904. bf 1050 to generate a narrowband rotation component 1028, which is then used as the output channel y of the PSM module 900 a (t) (e.g., as the left output channel).

[0121] According to some embodiments, the control data 914 for configuring the amplitude response module 902 may include the critical point f c 1038. Filter characteristics θ bf 1036, and the sound field position Γ 1040. This data is provided to the PSM module 900 via the amplitude response module 902, which determines the critical point (in radians) ω c 1044, filter characteristics θ bf 1042 and secondary term In some embodiments, the amplitude response module 902 modifies one or more parameters of the control data 914 (eg, the critical point f) based on one or more parameters of the input audio signal x(t) 912. c 1038. Filter characteristics θ bf 1036 and / or sound field position Γ1040). In some embodiments, such as Fig. 10A In the embodiment shown in FIG. 1 , the filter characteristic θ bf 1042 is equivalent to the filter characteristic θ bf 1036. These intermediate representations 1042, 1044, and 1046 are provided to the filter configuration module 904, which generates a filter that can include at least first-order coefficients β bf 1050 and filter configuration data of the rotation control parameter 1048. The first order coefficient β bf1050 is provided to the all-pass filter module 906 via the first-order all-pass filter 1026. In some embodiments, the rotation control parameter θ bf 1048 can be equivalent to the filter characteristic θ bf 1036 and 1042, while in other embodiments the parameter may be scaled for convenience. For example, in some embodiments, the filter characteristic is associated with a parameter range (e.g., 0 to 0.5) with a meaningful center point, and the rotation control parameter is scaled relative to the filter characteristic to change the parameter range, e.g., to 0 to 1. In some embodiments, the filter characteristic is scaled linearly (e.g., to maintain increased resolution in the poles compared to the center point), while in other embodiments, a nonlinear mapping may be used (e.g., to increase numerical resolution around the center point). In the following equation, the rotation control parameter θ bf is considered unscaled, but it will be appreciated that the same principles apply when the rotation control parameter θ is scaled. bf 1048 is provided to the all-pass filter module 906 via the broadband phase rotator 1004.

[0122] Fig. 10B An example implementation of the wideband phase rotator 1004 according to some embodiments is described in detail. The wideband phase rotator 1004 receives a mono input audio signal x(t) 912 and a rotation control parameter θ bf 1048. The input audio signal x(t) 912 is first processed by the Hilbert transform module 1006 to generate a left branch component 1008 and a right branch component 1010. According to some embodiments, the Hilbert transform module 1006 may use Figure 7 , but it should be understood that other implementations of the Hilbert transform module 1006 may be used in other embodiments. The left branch component 1008 and the right branch component 1010 are provided to the 2D orthogonal rotation module 1012. The left branch component 1008 is also provided to the output of the broadband phase rotator 1004 as a right broadband rotated component 1022. Since the broadband phase rotator 1004 is configured to rotate the left branch and right branch signals relative to each other, one way to achieve this in some embodiments is to keep the left branch component 1008 constant as the right broadband rotated component 1022, and rotate the left branch component and the right branch component to form the left broadband rotated component 1020.

[0123] According to some embodiments, in addition to the left branch component 1008 and the right branch component 1010, the 2D orthogonal rotation module 1012 may also receive a rotation control parameter θ from the filter configuration module 904. bf 1048, such as Fig. 10A. The 2D orthogonal rotation module 1012 uses the data to generate a left rotation component 1014 and a right rotation component 1016. The projection module 1018 then receives the left rotation component 1014 and the right rotation component 1016, which are combined (eg, added) to form a left broadband rotation component 1020. Fig. 10A As shown in FIG. 1 , the wideband phase rotator 1004 outputs the left wideband rotation component 1020 to the narrowband phase rotator 1024 to generate a narrowband rotation component 1028 as the left output channel y of the PSM module. a (t), and generates the right broadband rotation component 1022 as the right output channel y of the PSM module b (t) (which bypasses the narrowband phase rotator 1024 or passes through it unchanged). In other embodiments, the narrowband rotated component 1028 and the left branch component 1008 (which Fig. 10A and Fig. 10B , used as the right broadband rotation component 1022 in the embodiment shown in FIG. , is instead mapped to the right output channel y b (t) and the left output channel y a (t).

[0124] In some embodiments, the PSM module 900 can be formally described by the following equation (8):

[0125]

[0126] In some embodiments, the single-input multi-output all-pass filter is composed of multiple parts, each of which will be explained in turn. According to some embodiments, these components may include A f , A b and H2.

[0127] According to some embodiments, A f Can correspond to Fig. 10A The narrowband phase rotator 1024 in A f is a first-order all-pass filter, the output of one channel of which is in the form of equation (9):

[0128] y(t)≡-β f x(t)+x(t-1)+β f y(t-1) (9)

[0129] where β f are the coefficients of the filter, ranging from -1 to +1. The second output of the filter can simply pass the input unchanged. Therefore, according to some embodiments, filter A f The implementation can be defined by equation (10):

[0130] A f (x(t),βf )≡[y(t),x(t)] (10)

[0131] A f The transfer function is expressed as the differential phase shift from one output to another The differential phase shift is a function of the radian frequency ω, as defined by equation (11):

[0132]

[0133] The target amplitude response can be obtained by using It is derived by replacing θ in equation (5) or (6), depending on whether the response is placed in the middle (equation (5)) or at the side (equation (6)).

[0134] The frequency fc at which the sum gain αf=3dB can be used as a critical point for tuning is defined by:

[0135]

[0136] and

[0137]

[0138] By normalizing the target amplitude response to 0 dB, this critical point corresponds to parameter fc, which can be the -3 dB point. In equation (8), A f The outputs of are subscripted to acknowledge that, according to some embodiments, only the output of the first channel is used.

[0139] In equation (8), A b is a single-input multiple-output all-pass filter, which can correspond to Fig. 10A The broadband phase rotator 1004 in A b It can be formally defined in equation (14) as:

[0140] A b (x(t), θ)=[(H2(x(t))1cosθ+H2(x(t))2sinθ) H2(x(t))1] (14)

[0141] where H2(x(t)) is the discrete form of the filter, implemented using a pair of orthogonal all-pass filters and defined using a continuous-time prototype according to equation (15):

[0142]

[0143] In some embodiments, the all-pass filter H(x(t)) provides a constraint on a 90 degree phase relationship between the two output signals and a unit amplitude relationship between the input and the two output signals, but does not necessarily guarantee a specific phase relationship between the input (mono) signal and one of the two (stereo) output signals.

[0144] The discrete form of H(x(t)) is denoted H2(x(t)) and is defined by its action on the mono signal x(t). The result is a two-dimensional vector, as defined in equation (16):

[0145]

[0146] According to some embodiments, a discrete single-input multiple-output all-pass filter may correspond to Fig. 10B The Hilbert transform module 1006 in FIG. Figure 7 The Hilbert transform module 700 in . According to some embodiments, in equation (14), θ determines the rotation angle of the first output of Ab relative to the second output.

[0147] Finally, according to some embodiments, the supply to the complete system A in equation (8) can be determined as follows bf These parameters can include β bf and θ bf , which can correspond to Fig. 10A The rotation control parameter θ in bf 1048 and the first-order coefficient β bf 1050. In some embodiments, β bf The central arc frequency ω c It was determined as follows:

[0148]

[0149] where equation (12) can be used according to the desired center frequency f c To calculate ω c .exist Fig. 10A In, ω c Corresponding to the critical point ω c 1044,f c Corresponding to the critical point f c 1038, and the action of equation (17) is performed in part within the filter configuration module 904, producing the first-order coefficients β bf In some embodiments, the quadratic term It can be obtained from equation (18) from θ bf And the Boolean sound field position parameter Γ is derived:

[0150]

[0151] The quadratic term 1046 is given by Fig. 10A The amplitude response module 902 in is provided to the filter configuration module 904.

[0152] In some embodiments, the advanced parameter f c ,θ bf and Γ may be sufficient to intuitively and conveniently tune the system. According to such an embodiment, the center frequency f c Determine the inflection point in Hz where the target amplitude response asymptotically approaches -∞dB. Parameter θ bf Allows control about the inflection point f c The filter characteristics. For 0<θ bf <1 / 4, this characteristic is low-pass, at f c is zero at θ, and the spectral slope in the target amplitude function increases with θ bf The increase in the frequency is conducive to smooth interpolation from low frequency to flat. bf <1 / 2, as θ bf The increase of the characteristic from f c The flat value at θ is zero and smoothly interpolated to the high pass. bf =1 / 4 point, the target amplitude function is pure band-stop, at f c The parameter Γ is a Boolean value that will be determined by f c and θ bf The determined target amplitude function is placed into the mid channel (ie, L+R) or the side channel (ie, LR). Since both outputs of the filter network are subject to all-pass constraints, the effect of Γ is to switch between complementary target amplitude responses.

[0153] In some embodiments, to achieve a magnitude response of a 60 degree vertical cue, the parameter f c =11kHz,θ bf =0.13 and X=1 to configure the above FNORD filter network. Fig.11 A frequency response graph is illustrated showing the output frequency response of a FNORD filter network configured to achieve a magnitude response for a 60 degree vertical cue, in accordance with some embodiments. Fig.11 The output frequency response in the mid component 1110 and the side component 1120 is illustrated, where the FNORD filter network is driven by white noise. In some embodiments, the filter parameters f are selected based on analysis of HRTF-based elevation cues at the desired angle. c ,θ bf and / or Γ.

[0154] In some embodiments, the PSM module 900 uses a frequency domain specification of an all-pass filter. For example, in some cases, more complex spatial cues may be needed, such as those sampled from an anthropometric dataset. Within certain limitations, the above techniques can be used to embed arbitrary cues into the phase differences of an audio stream based on their amplitude frequency domain representation. For example, the filter configuration module 904 can use equations of the form of equations (5) or (6) to vectorize the target amplitude response of the K narrowband attenuation coefficients for the middle or side. To determine the vector transfer function of K phase angles

[0155] The phase angle vector θ generates a finite impulse response filter as defined in equation (19):

[0156]

[0157] Where DFT-1 stands for Inverse Discrete Fourier Transform (IDFT) and Then, a vector of 2(K-1) FIR filter coefficients Bn(θ) can be applied to x(t) as defined in equation (20):

[0158]

[0159] in Indicates the convolution operation.

[0160] To reproduce the effect from the previous example, and achieve a target magnitude response corresponding to a 60 degree altitude cue, the observed HRIR can be sampled and applied to a DFT of length 2(K-1) to produce It can be used to determine the target magnitude response vector using the following operation

[0161]

[0162] in and are operations that return the real and imaginary parts of a complex number, respectively, and all operations are applied component-wise to vectors. This target magnitude response substituted into the middle or side portion can now be applied to one of equations (5) or (6) to determine the vector of K phase angles From this vector, a FIR filter B can be derived. This filter is then substituted into equation (19) to derive a single-input multiple-output all-pass filter.

[0163] While equations (19) and (20) provide an effective means for constraining the target amplitude response, their implementation will typically rely on a relatively high-order FIR filter resulting from an inverse DFT operation. This may not be suitable for resource-limited systems. In this case, a low-order infinite impulse response (IIR) implementation may be used, such as discussed in conjunction with equation (8).

[0164] The all-pass filter module 906 applies the all-pass filter configured by the filter configuration module 904 to the mono channel x(t) to generate the output channel y a (t) and y b (t). The application of the all-pass filter to the channel x(t) can be performed as defined in equations (8), (20), or as Fig. 9 or Fig. 10A The all-pass filter module 906 provides each output channel to a corresponding speaker, such as channel y a (t) is provided to speaker 910a and channel y b (t) is provided to speaker 910b. Fig. 9 It is not shown, but it should be understood that the output channel y a (t) and y b (t) may be transmitted via one or more intermediary components (e.g., component processor module 106, crosstalk processor module 110, and / or L / R to M / S converter module and M / S to L / R converter module 104 and 108, such as Figure 1 ) and is provided to speakers 910a and 910b.

[0165] Super intermediate processing

[0166] In some embodiments, PSM processing may be performed on a target portion of a received audio signal, such as a mid component of the audio signal or a super mid component of the audio signal. Fig.12 1 is a block diagram of an audio processing system 1200 according to one or more embodiments. The system 1200 generates a super-mid component to isolate a target portion (e.g., speech) of an audio signal, and performs PSM processing on the super-mid component to spatially shift the target portion. Some embodiments of the system 1200 have different components than those described herein. Similarly, in some cases, functionality may be allocated between components in a manner different than that described herein.

[0167] System 1200 includes an L / R to M / S converter module 1206 , a quadrature component generator module 1212 , a quadrature component processor module 1214 including the PSM module 102 , and a crosstalk processor module 1224 .

[0168] L / R to M / S converter module 1206 receives left channel 1202 and right channel 1204 and generates mid component 1208 and side component 1210 from channels 1202 and 1204. Discussion of L / R to M / S converter module 1206 M / S converter module 104 may be applicable to L / R to M / S converter module 1206.

[0169] The orthogonal component generator module 1212 processes the middle component 1208 and the side component 1210 to generate at least one of the following: a super middle component M1, a super side component S1, a residual middle component M2, and a residual side component S2. The super middle component M1 is the spectral energy of the middle component 1208 after the spectral energy of the side component 1210 is removed. The super side component S1 is the spectral energy of the side component 1210 after the spectral energy of the middle component 1208 is removed. The residual middle component M2 is the spectral energy of the middle component 1208 after the spectral energy of the super middle component M1 is removed. The residual side component S2 is the spectral energy of the side component 1210 after the spectral energy of the super side component S1 is removed. The system 1200 generates a left channel 1242 and a right output channel 1244 by processing at least one of the super middle component M1, the super side component S1, the residual middle component M2, and the residual side component S2. The orthogonal component generator module 1212 generates the left channel 1242 and the right output channel 1244 with respect to Fig.13A , Fig. 13B and Fig. 13C Further description.

[0170] The quadrature component processor module 1214 processes one or more of the super-mid component M1, the super-side component S1, the residual mid component M2, and / or the residual side component S2, and converts the processed components into a processed left component 1220 and a processed right component 1222. The discussion regarding the component processor module 106 is applicable to the quadrature component processor module 1214, except that the processing is performed on the super-mid component M1, the super-side component S1, the residual mid component M2, and / or the residual side component S2 instead of the mid component and the side component. For example, the processing of the components M1, M2, S1, and S2 may include various types of processing, such as spatial cue processing (e.g., amplitude or delay-based panning, binaural processing, etc.), single-band or multi-band equalization, single-band or multi-band dynamic processing (e.g., compression, expansion, limiting, etc.), single-band or multi-band gain or delay stages, adding audio effects, or other types of processing. In some embodiments, the quadrature component processor module 1214 uses the super middle component M1, the super side component S1, the residual middle component M2 and / or the residual side component S2 to perform sub-band spatial processing and / or crosstalk compensation processing. The quadrature component processor module 1214 may also include an L / R to M / S converter to convert the components M1, S2, S1 and S2 into a processed left component 1220 and a processed right component 1222.

[0171] The quadrature component processor module 1214 also includes a PSM module 102, which can operate on one or more of the super-middle component M1, the super-side component S1, the residual mid-component M2 and / or the residual side component S2. For example, the PSM module 102 can receive the super-middle component M1 as input and generate spatially shifted left and right channels. The super-middle component M1 may include, for example, an isolated portion of an audio signal representing speech, and thus may be selected for HPSM processing. The left channel generated by the PSM module 102 is used to generate a processed left component 1020, and the right channel generated by the PSM module 102 is used to generate a processed right component 1222. About Fig.12 The quadrature component processor module 1214 is further described.

[0172] The crosstalk processor module 1224 receives the processed left component 1220 and the processed right component 1222 and performs crosstalk processing thereon. The crosstalk processor module 1224 outputs a left channel 1242 and a right channel 1244. Discussions regarding the crosstalk processor module 1224 may be applicable to the crosstalk processor module 1224. In some embodiments, crosstalk processing (e.g., simulation or elimination) may be performed before orthogonal component processing, such as before converting the left channel 1202 and the right channel 1204 into a middle component and a side component. The left channel 1242 may be provided to the left speaker 112 and the right channel 1244 may be provided to the right speaker 114.

[0173] Example Quadrature Component Generator

[0174] Figure 13A-13C 1313, 1323, and 1343, respectively, according to one or more embodiments. Orthogonal component generator modules 1313, 1323, and 1343 are examples of orthogonal component generator module 1212. Some embodiments of modules 1313, 1323, and 1343 have different components than those described herein. Similarly, in some cases, functions can be allocated between components in a manner different from that described herein.

[0175] refer to Fig.13A , the orthogonal component generator module 1313 includes a subtraction unit 1305, a subtraction unit 1309, a subtraction unit 1315, and a subtraction unit 1319. As described above, the orthogonal component generator module 1313 receives the middle component 1208 and the side component 1210, and outputs one or more of a super middle component M1, a super side component S1, a residual middle component M2, and a residual side component S2.

[0176] The subtraction unit 1305 removes the spectral energy of the side component 1210 from the spectral energy of the middle component 1208 to generate the super middle component M1. For example, the subtraction unit 1305 subtracts the amplitude of the side component 1210 in the frequency domain from the amplitude of the middle component 1208 in the frequency domain, while retaining the phase, to generate the super middle component M1. The subtraction in the frequency domain can be performed by Fourier transforming the time domain signal to generate a signal in the frequency domain and then subtracting the signal in the frequency domain. In other examples, the subtraction in the frequency domain can be performed in other ways, such as using a wavelet transform instead of a Fourier transform. The subtraction unit 1309 generates a residual middle component M2 by removing the spectral energy of the super middle component M1 from the spectral energy of the middle component 1208. For example, the subtraction unit 1309 subtracts the amplitude of the super middle component M1 in the frequency domain from the amplitude of the middle component 1208 in the frequency domain, while retaining the phase, to generate the residual middle component M2. Although subtracting the side component from the middle component in the time domain results in the original right channel of the signal, the above operation in the frequency domain isolates the portion of the spectral energy of the middle component that is different from the spectral energy of the side component (called M1, or super-middle component) and distinguishes between the portion of the spectral energy of the middle component that is different from the spectral energy of the side component (called M1, or super-middle component) and the portion of the spectral energy of the middle component that is the same as the spectral energy of the side component (called M2, or residual middle component).

[0177] In some embodiments, additional processing may be used when subtracting the spectral energy of the side component 1210 from the spectral energy of the middle component 1006 results in a negative value for the super middle component M1 (e.g., for one or more bins in the frequency domain). In some embodiments, when subtracting the spectral energy of the side component 1210 from the spectral energy of the middle component 1208 results in a negative value, the super middle component M1 is fixed at a value of 0. In some embodiments, the super middle component M1 is wrapped by taking the absolute value of the negative value as the value of the super middle component M1. When subtracting the spectral energy of the side component 1210 from the spectral energy of the middle component 1208 results in a negative value for M1, other types of processing may be used. Similar additional processing, such as fixing at 0, wrapping, or other processing, may be used when the subtraction that generates the super side component S1, the residual side component S2, or the residual middle component M2 results in a negative value. When the subtraction results in a negative value, fixing the super middle component M1 at 0 will provide spectral orthogonality between M1 and the two side components. Similarly, when the subtraction results in a negative value, fixing the super-side component S1 at 0 will provide spectral orthogonality between S1 and the two middle components. By creating orthogonality between the super-middle component and the side component and their appropriate middle / side contrast components (i.e., the side component of the super-middle component, the middle component of the super-side component), the derived residual middle component M2 and residual side component S2 contain spectral energy that is not orthogonal (i.e., common) to their appropriate middle / side contrast components. That is, when a fix at 0 is applied to the super-middle component and the residual middle component is derived using the M1 component, a super-middle component that has no common spectral energy with the side component and a residual middle component that has exactly the same spectral energy as the side component are generated. When the super-side is fixed to 0, the same relationship applies to the super-side and the residual side. When frequency domain processing is applied, a trade-off is usually made in the resolution between frequency and timing information. As the frequency resolution increases (i.e., as the FFT window size and the number of frequency bins increase), the time resolution decreases, and vice versa. The spectral subtraction described above occurs on a per-frequency bin basis, so in some situations, such as when removing sound energy from the super-mid component, it may be preferable to have a larger FFT window size (e.g., 8192 samples, resulting in 4096 frequency bins given a real-valued input signal). Other situations may require a higher time resolution, and therefore a lower overall delay and a lower frequency resolution (e.g., 512 sample FFT window size, resulting in 256 frequency bins given a real-valued input signal). In the latter case, when the low-frequency resolution of the mid and side components are subtracted from each other to derive the super-mid component M1 and the super-side component S1, audible spectral artifacts may be produced because the spectral energy of each frequency bin is a representation of the average energy over too large a frequency range.In this case, taking the absolute value of the difference between the middle and side components when deriving the super middle component M1 or the super side component S1 can help mitigate perceptual artifacts by allowing each frequency bin to diverge from the true orthogonality in the components. In addition to or instead of wrapping around 0, we can apply a coefficient to the subtraction value that scales the value between 0 and 1, thereby providing a method of interpolating between super middle and residual middle / super side components and residual side components at one extreme (i.e., a value of 1) with perfect orthogonality and super middle and side components at the other extreme (i.e., a value of 0) with the same super middle and side components as their corresponding original middle and side components.

[0178] The subtraction unit 1315 removes the spectral energy of the middle component 1208 in the frequency domain from the spectral energy of the side component 1210 in the frequency domain while preserving the phase to generate the super-side component S1. For example, the subtraction unit 1315 subtracts the amplitude of the middle component 1208 in the frequency domain from the amplitude of the side component 1210 in the frequency domain while preserving the phase to generate the super-side component S1. The subtraction unit 1319 removes the spectral energy of the super-side component S1 from the spectral energy of the side component 1210 to generate the residual side component S2. For example, the subtraction unit 1319 subtracts the amplitude of the super-side component S1 in the frequency domain from the amplitude of the side component 1210 in the frequency domain while preserving the phase to generate the residual side component S2.

[0179] exist Fig. 13B , the orthogonal component generator module 1323 is similar to the orthogonal component generator module 1313 in that it receives the middle component 1208 and the side component 1210 and generates a super middle component M1, a residual middle component M2, a super side component S1, and a residual side component S2. The orthogonal component generator module 1323 is different from the orthogonal generator module 1313 in that the super middle component M1 and the super side component S1 are generated in the frequency domain and then converted back to the time domain to generate the residual middle component M2 and the residual side component S2. The orthogonal component generator module 1323 includes a forward FFT unit 1320, a bandpass unit 1322, a subtraction unit 1324, a super intermediate processor 1325, an inverse FFT unit 1326, a time delay unit 1328, a subtraction unit 1330, a forward FFT unit 1332, a bandpass unit 1334, a subtraction unit 1336, a super side processor 1337, an inverse FFT unit 1340, a time delay unit 1342 and a subtraction unit 1344.

[0180] A forward fast Fourier transform (FFT) unit 1320 applies a forward FFT to the intermediate component 1208 to convert the intermediate component 1208 to the frequency domain. The converted intermediate component 1208 in the frequency domain includes an amplitude and a phase. A bandpass unit 1322 applies a bandpass filter to the frequency domain intermediate component 1208, wherein the bandpass filter indicates the frequency in the super intermediate component M1. For example, in order to isolate the typical human vocal range, the bandpass filter can indicate the frequency between 300 and 8000 Hz. In another example, in order to remove the audio content associated with the typical human vocal range, the bandpass filter can retain lower frequencies (e.g., generated by a bass guitar or drum) and higher frequencies (e.g., generated by a cymbal) in the super intermediate component M1. In other embodiments, instead of and / or in addition to the bandpass filter applied by the bandpass unit 1322, the orthogonal component generator module 1323 applies various other filters to the frequency domain intermediate component 1208. In some embodiments, the quadrature component generator module 1323 does not include the bandpass unit 1322 and does not apply any filter to the frequency domain middle component 1208. In the frequency domain, the subtraction unit 1324 subtracts the side component 1210 from the filtered middle component to generate the super middle component M1. In other embodiments, instead of the quadrature component processor module (e.g., Fig.12 The orthogonal component generator module 1323 applies various audio enhancement functions to the frequency domain super-middle component M1 in addition to the subsequent processing performed by the orthogonal component processor module 1214 applied to the super-middle component M1. The super-middle processor 1325 performs processing on the super-middle component M1 in the frequency domain before converting the super-middle component M1 in the frequency domain to the time domain. The processing may include sub-band spatial processing and / or crosstalk compensation processing. In some embodiments, instead of and / or in addition to the processing that can be performed by the orthogonal component processor module 1214, the super-middle processor 1325 performs processing on the super-middle component M1. The inverse FFT unit 1326 applies the inverse FFT to the super-middle component M1 and converts the super-middle component M1 back to the time domain. The super-middle component M1 in the frequency domain includes the amplitude of M1 and the phase of the middle component 1208, and the inverse FFT unit 1326 converts it to the time domain. The time delay unit 1328 applies a time delay to the intermediate component 1208 so that the intermediate component 1208 and the super intermediate component M1 arrive at the subtraction unit 1330 at the same time. The subtraction unit 1330 subtracts the super intermediate component M1 in the time domain from the time delayed intermediate component 1208 in the time domain to generate a residual intermediate component M2. In this example, the spectral energy of the super intermediate component M1 is removed from the spectral energy of the intermediate component 1208 using processing in the time domain.

[0181] The forward FFT unit 1332 applies a forward FFT to the side component 1210, converting the side component 1210 to the frequency domain. The converted side component 1210 in the frequency domain includes an amplitude and a phase. The bandpass unit 1334 applies a bandpass filter to the frequency domain side component 1210. The bandpass filter indicates the frequency in the super side component S1. In other embodiments, in addition to and / or in place of the bandpass filter, the orthogonal component generator module 1323 applies various other filters to the frequency domain side component 1210. In the frequency domain, the subtraction unit 1336 subtracts the intermediate component 1208 from the filtered side component 1210 to generate the super side component S1. In other embodiments, instead of and / or in addition to the subsequent processing applied to the super side component S1 performed by the orthogonal component processor (e.g., the orthogonal component processor module 1214), the orthogonal component generator module 1323 applies various audio enhancements to the frequency domain super side component S1. The super side processor 1337 performs processing on the super side component S1 in the frequency domain before converting the super side component S1 in the frequency domain to the time domain. The processing may include sub-band spatial processing and / or crosstalk compensation processing. In some embodiments, the super side processor 1337 performs processing on the super side component S1 instead of and / or in addition to the processing that can be performed by the orthogonal component processor module 1214. The inverse FFT unit 1340 applies the inverse FFT to the side component S1 in the frequency domain to generate the super side component S1 in the time domain. The super side component S1 in the frequency domain includes the amplitude of S1 and the phase of the side component 1210, which is converted to the time domain by the inverse FFT unit 1326. The time delay unit 1342 time delays the side component 1210 so that the side component 1210 arrives at the subtraction unit 1344 at the same time as the super side component S1. The subtraction unit 1344 then subtracts the super side component S1 in the time domain from the time delayed side component 1210 in the time domain to generate a residual side component S2. In this example, the spectral energy of the super-side component S1 is removed from the spectral energy of the side component 1210 using processing in the time domain.

[0182] In some embodiments, if the processing performed by super middle processor 1325 and super side processor 1337 is performed by orthogonal component processor module 1214, these components may be omitted.

[0183] exist Fig. 13CIn the figure, the orthogonal component generator module 1343 is similar to the orthogonal component generator module 1323 in that it receives the middle component 1208 and the side component 1210 and generates a super middle component M1, a residual middle component M2, a super side component S1, and a residual side component S2, but the difference is that the orthogonal component generator module 1343 generates each component M1, M2, S1 and S2 in the frequency domain and then converts these components to the time domain. The orthogonal component generator module 1343 includes a forward FFT unit 1347, a bandpass unit 1349, a subtraction unit 1351, a super intermediate processor 1352, a subtraction unit 1353, a residual intermediate processor 1354, an inverse FFT unit 1355, an inverse FFT unit 1357, a forward FFT unit 1361, a bandpass unit 1363, a subtraction unit 1365, a super side processor 1366, a subtraction unit 1367, a residual side processor 1368, an inverse FFT unit 1369 and an inverse FFT unit 1371.

[0184] The forward FFT unit 1347 applies a forward FFT to the intermediate component 1208 to convert the intermediate component 1208 to the frequency domain. The converted intermediate component 1208 in the frequency domain includes an amplitude and a phase. The forward FFT unit 1361 applies a forward FFT to the side component 1210 to convert the side component 1210 to the frequency domain. The converted side component 1210 in the frequency domain includes an amplitude and a phase. The bandpass unit 1349 applies a bandpass filter to the frequency domain intermediate component 1208, and the bandpass filter indicates the frequency of the super intermediate component M1. In some embodiments, in addition to and / or instead of the bandpass filter, the orthogonal component generator module 1343 applies various other filters to the frequency domain intermediate component 1208. The subtraction unit 1351 subtracts the frequency domain side component 1210 from the frequency domain intermediate component 1208 to generate the super intermediate component M1 in the frequency domain. The super intermediate processor 1352 performs processing on the super intermediate component M1 in the frequency domain before converting the super intermediate component M1 in the frequency domain to the time domain. In some embodiments, the super intermediate processor 1352 performs sub-band spatial processing and / or crosstalk compensation processing. In some embodiments, the super intermediate processor 1352 performs processing on the super intermediate component M1 instead of and / or in addition to the processing that can be performed by the orthogonal component processor module 1214. The inverse FFT unit 1357 applies the inverse FFT to the super intermediate component M1 and converts the super intermediate component M1 back to the time domain. The super intermediate component M1 in the frequency domain includes the amplitude of M1 and the phase of the intermediate component 1208, which the inverse FFT unit 1357 converts to the time domain. The subtraction unit 1353 subtracts the super intermediate component M1 from the intermediate component 1208 in the frequency domain to generate a residual intermediate component M2. The residual intermediate processor 1354 performs processing on the residual intermediate component M2 in the frequency domain before converting the residual intermediate component M2 in the frequency domain to the time domain. In some embodiments, the residual intermediate processor 1354 performs sub-band spatial processing and / or crosstalk compensation processing on the residual intermediate component M2. In some embodiments, the residual intermediate processor 1354 performs processing on the residual intermediate component M2 instead of and / or in addition to the processing that can be performed by the orthogonal component processor module 1214. The inverse FFT unit 1355 applies an inverse FFT to convert the residual intermediate component M2 to the time domain. The residual intermediate component M2 in the frequency domain includes the amplitude of M2 and the phase of the intermediate component 1208, which the inverse FFT unit 1355 converts to the time domain.

[0185] Bandpass unit 1363 applies a bandpass filter to frequency domain side component 1210. The bandpass filter specifies the frequency in super side component S1. In other embodiments, in addition to and / or in place of the bandpass filter, orthogonal component generator module 1343 applies various other filters to frequency domain side component 1210. In the frequency domain, subtraction unit 1365 subtracts intermediate component 1208 from filtered side component 1210 to generate super side component S1. Super side processor 1366 performs processing on super side component S1 in the frequency domain before converting super side component S1 in the frequency domain to the time domain. In some embodiments, super side processor 1366 performs subband spatial processing and / or crosstalk compensation processing on super side component S1. In some embodiments, super side processor 1366 performs processing on super side component S1 instead of and / or in addition to the processing that can be performed by orthogonal component processor module 1214. Inverse FFT unit 1371 applies inverse FFT to convert super side component S1 back to the time domain. The super side component S1 in the frequency domain includes the amplitude of S1 and the phase of the side component 1210, and the inverse FFT unit 1371 converts it to the time domain. The subtraction unit 1367 subtracts the super side component S1 from the side component 1210 in the frequency domain to generate a residual side component S2. The residual side processor 1368 performs processing on the residual side component S2 in the frequency domain before converting the residual side component S2 in the frequency domain to the time domain. In some embodiments, the residual side processor 1368 performs subband spatial processing and / or crosstalk compensation processing on the residual side component S2. In some embodiments, instead of and / or in addition to the processing that can be performed by the orthogonal component processor module 1214, the residual side processor 1368 performs processing on the residual side component S2. The inverse FFT unit 1369 applies an inverse FFT to the residual side component S2 and converts it to the time domain. The residual side component S2 in the frequency domain includes the amplitude of S2 and the phase of the side component 1210, and the inverse FFT unit 1369 converts it to the time domain.

[0186] In some embodiments, if the processing performed by the super intermediate processor 1352, the super side processor 1366, the residual intermediate processor 1354, or the residual side processor 1368 is performed by the orthogonal component processor module 1214, these components may be omitted.

[0187] Example Quadrature Component Processor

[0188] Fig.14A 1 is a block diagram of a quadrature component processor module 1417 according to one or more embodiments. The quadrature component processor module 1417 is an example of the quadrature component processor module 1412. Some embodiments of the module 1417 have different components than those described herein. Similarly, in some cases, functionality may be allocated between components in a manner different than that described herein.

[0189] The quadrature component processor module 1417 includes a component processor module 1420 , a PSM module 102 , an adding unit 1422 , an M / S to L / R converter module 1424 , an adding unit 1426 , and an adder 1428 .

[0190] The component processor module 1420 performs processing like the component processor module 106, except that the super middle component M1, the super side component S1, the residual middle component M2, and / or the residual side component S2 are used instead of the middle component and the side component. For example, the component processor module 1420 performs sub-band spatial processing and / or crosstalk compensation processing on at least one of the super middle component M1, the residual middle component M2, the super side component S1, and the residual side component S2. As a result of the sub-band spatial processing and / or crosstalk compensation of the component processor module 1420, the orthogonal component processor module 1417 outputs at least one of the processed M1, the processed M2, the processed S1, and the processed S2. In some embodiments, one or more of the components M1, M2, S1, or S2 can bypass the component processor module 1420.

[0191] In some embodiments, the orthogonal component processor module 1417 performs sub-band spatial processing and / or crosstalk compensation processing on at least one of the super middle component M1, the residual middle component M2, the super side component S1, and the residual side component S2 in the frequency domain. The orthogonal component generator module 410 may provide the components M1, M2, S1, or S2 in the frequency domain to the orthogonal component processor module 1417 without performing an inverse FFT. After generating the processed M1, the processed M2, and the processed side component 1442, the orthogonal component processor module 1417 may perform an inverse FFT to convert these components back to the time domain. In some embodiments, the orthogonal component processor module 1417 performs an inverse FFT on the processed M1, the processed M2, the processed S1, and the processed S1, and generates a processed side component 1446 in the time domain.

[0192] Example components of the quadrature component processor module 1417 are Fig.15 and Fig.16. In some embodiments, the quadrature component processor module 1417 performs both sub-band spatial processing and crosstalk compensation processing. The processing performed by the quadrature component processor module 1417 is not limited to sub-band spatial processing or crosstalk compensation processing. Any type of spatial processing using mid / side space, such as by using a super-mid component instead of a mid component or using a super-side component instead of a side component, may be performed by the quadrature component processor module 1417. Some other types of processing may include gain application, amplitude or delay-based panning, binaural processing, reverberation, dynamic range processing (such as compression and limiting), and other linear or nonlinear audio processing techniques and effects ranging from chorus or flanging to machine learning-based vocal or instrumental style transfer, conversion, or resynthesis.

[0193] The PSM module 102 receives the processed M1 and applies the PSM processing to spatially shift the processed M1, producing a left channel 1432 and a right channel 1434. Although the PSM module 102 is shown as being applied to the super middle component M1, the PSM module can be applied to one or more of the components M1, M2, S1, or S2. In some embodiments, the components processed by the PSM module 102 bypass the processing of the component processor module 1420. For example, the PSM module 102 can process the super middle component M1 instead of the processed M1.

[0194] The addition unit 1422 adds the processed S1 to the processed S2 to generate a processed side component 1442. The M / S to L / R converter module 1424 uses the processed M2 and the processed side component 1442 to generate a processed left component 1444 and a processed right component 1446. In some embodiments, the processed left component 1444 is generated based on the sum of the processed M2 and the processed side component 1442, and the processed right component 1446 is generated based on the difference between the processed M2 and the processed side component 1442. Other M / S to L / R types of transforms may be used to generate the processed left component 1444 and the processed right component 1446.

[0195] The summing unit 1426 adds the left channel 1432 from the PSM module 102 to the processed left component 1444 to generate the left channel 1452. The summing unit 1428 adds the right channel 1434 from the PSM module 102 to the processed right component 1446 to generate the right channel 1454. More generally, one or more left channels from the PSM module 102 may be added to the left component from the M / S to L / R converter module 1424 (e.g., generated using the super component / residual component not processed by the PSM module 102) to generate the left channel 1452, and one or more right channels from the PSM module 102 may be added to the right component from the M / S to L / R converter module 1424 (e.g., generated using the super component / residual component not processed by the PSM module 102) to generate the right channel 1454.

[0196] In this way, the quadrature component processor module 1417 applies PSM processing to the super-middle component M1 of the audio signal isolated by the L / R to M / S converter module 1206 and the quadrature component generator module 1212. The PSM enhanced stereo signal (including the left channel 1432 and the right channel 1434) can then be summed with the residual left / right signal (e.g., the processed left component 1444 and the processed right component 1446 generated without the super-middle component). In addition to or in lieu of this example, other methods of isolating components of the input signal used for PSM processing can be used, including audio source separation based on machine learning.

[0197] In some embodiments, the quadrature component processor module 1417 applies PSM to process the middle component M of the audio signal, rather than the super middle component M1. Fig. 14B FIG. 14 is a block diagram of a quadrature component processor module 1419 according to one or more embodiments. In some embodiments, Fig. 14B The quadrature component processor module 1419 may be implemented similar to Fig.12, but without the orthogonal component generator module 1212, such that the orthogonal component processor module receives mid component and side component signals (e.g., mid component 1208 and side component 1210) instead of super-mid component, super-side component, residual mid component, and residual side component. In some embodiments, the orthogonal component processor module 1419 includes a component processor module similar to the component processor module 106 to generate a processed mid component and a processed side component based on the received mid component and side component (not shown). The PSM module 102 receives the mid component M (or processed mid component) and applies PSM processing to spatially shift the received mid signal to generate a PSM processed left channel 1432 and a PSM processed right channel 1434 of the mid signal, which are combined with the side component S (or processed side component) by the M / S to L / R converter module 1424 to generate a left channel 1452 and a right channel 1454. For example, as Fig. 14B As shown in FIG. 1 , the M / S to L / R converter module 1424 uses an addition unit 1460 to generate a left channel 1452 as a sum of a PSM-processed left channel 1432 and a side component S, and uses a subtraction unit 1462 to generate a right channel 1454 as a difference between a PSM-processed right channel 1434 and the side component S. In other words, the M / S to L / R converter module 1424 is used to mix a side signal (which is located in a subspace defined by a left component (an inverse component of a right component) on a left-right basis) into a PSM-processed stereo signal in a left-right space by combining the side signal with the left channel and the inverse of the side signal with the right channel.

[0198] Example Subband Spatial Processor

[0199] Fig.15 1 is a block diagram of a subband spatial processor module 1510 according to one or more embodiments. The subband spatial processor module 1510 is an example of a component of the component processor module 106 or 1420. The subband spatial processor module 1510 includes a middle EQ filter 1504(1), a middle EQ filter 1504(2), a middle EQ filter 1504(3), a middle EQ filter 1504(4), a side EQ filter 1506(1), a side EQ filter 1506(2), a side EQ filter 1506(3), and a side EQ filter 1506(4). Some embodiments of the subband spatial processor module 1510 have different components than those described herein. Similarly, in some cases, functionality may be allocated between components in a manner different than that described herein.

[0200] The subband spatial processor module 1510 receives the non-spatial component Ym and the spatial component Ys and gain adjusts the subband of one or more of these components to provide spatial enhancement. When the subband spatial processor module 1510 is part of the component processor module 1420, the non-spatial component Ym can be a super middle component M1 or a residual middle component M2. The spatial component Ys can be a super side component S1 or a residual side component S2. When the subband spatial processor module 1510 is part of the component processor module 106, the non-spatial component Ym can be a middle component 126 and the spatial component Ys can be a side component 128.

[0201] The subband spatial processor module 1510 receives the non-spatial component Ym and applies the intermediate EQ filters 1504(1) to 1504(4) to different subbands of Ym to generate enhanced non-spatial components Em. The subband spatial processor module 1510 also receives the spatial component Ys and applies the side EQ filters 1506(1) to 1506(4) to different subbands of Ys to generate enhanced spatial components Es. The subband filters may include various combinations of peak filters, notch filters, low-pass filters, high-pass filters, low-shelf filters, high-shelf filters, band-pass filters, band-stop filters and / or all-pass filters. The subband filters may also apply gains to each subband. More particularly, the subband spatial processor module 1510 includes a subband filter for each of the n frequency subbands of the non-spatial component Ym and a subband filter for each of the n subbands of the spatial component Ys. For example, for n=4 subbands, the subband spatial processor module 1510 includes a series of subband filters for the non-spatial component Ym, including an intermediate equalization (EQ) filter 1504(1) for subband (1), an intermediate EQ filter 1504(2) for subband (2), an intermediate EQ filter 1504(3) for subband (3), and an intermediate EQ filter 1504(4) for subband (4). Each intermediate EQ filter 1504 applies a filter to a frequency subband portion of the non-spatial component Ym to generate an enhanced non-spatial component Em.

[0202] The subband spatial processor module 1510 also includes a series of subband filters for the frequency subbands of the spatial component Ys, including a side equalization (EQ) filter 1506(1) for subband (1), a side EQ filter 1506(2) for subband (2), a side EQ filter 1506(3) for subband (3), and a side EQ filter 1506(4) for subband (4). Each side EQ filter 1506 applies a filter to a frequency subband portion of the spatial component Ys to generate an enhanced spatial component Es.

[0203] Each of the n frequency subbands of the non-spatial component Ym and the spatial component Ys may correspond to a frequency range. For example, frequency subband (1) may correspond to 0 to 300 Hz, frequency subband (2) may correspond to 300 to 510 Hz, frequency subband (3) may correspond to 510 to 2700 Hz, and frequency subband (4) may correspond to 2700 Hz to the Nyquist frequency. In some embodiments, each of the n frequency subbands is a merged set of critical bands. A corpus of audio samples from a variety of music genres may be used to determine the critical bands. The long-term average energy ratio of the middle component to the side component on the 24 Bark level critical bands is determined based on the samples. Continuous frequency bands with similar long-term average ratios are then grouped together to form a set of critical bands. The range of the frequency subbands and the number of frequency subbands may be adjustable.

[0204] In some embodiments, the subband spatial processor module 1510 processes the residual middle component M2 into a non-spatial component Ym and uses one of the side component, the super-side component S1 or the residual side component S2 as the spatial component Ys.

[0205] In some embodiments, subband spatial processor module 1510 processes one or more of super middle component M1, super side component S1, residual middle component M2 and residual side component S2. The filter of the subband applied to each component in these components can be different. Super middle component M1 and residual middle component M2 can be processed as discussed for non-spatial component Ym. Super side component S1 and residual side component S2 can be processed as discussed for spatial component Ys.

[0206] Example Crosstalk Compensation Processor

[0207] Fig.16 1 is a block diagram of a crosstalk compensation processor module 1610 according to one or more embodiments. The crosstalk compensation processor module 1610 is an example of a component of the component processor module 106 or 1420. Some embodiments of the crosstalk compensation processor module 1610 have different components than those described herein. Similarly, in some cases, functionality may be allocated between components in a manner different than that described herein.

[0208] The crosstalk compensation processor module 1610 includes a middle component processor 1620 and a side component processor 1630. The crosstalk compensation processor module 1610 receives the non-spatial component Ym and the spatial component Ys and applies a filter to one or more of these components to compensate for spectral defects caused by (e.g., subsequent or previous) crosstalk processing. When the crosstalk compensation processor module 1610 is part of the component processor module 1420, the non-spatial component Ym can be a super middle component M1 or a residual middle component M2. The spatial component Ys can be a super side component S1 or a residual side component S2. When the crosstalk compensation processor module 1610 is part of the component processor module 106, the non-spatial component Ym can be the middle component 126 and the spatial component Ys can be the side component 128.

[0209] The crosstalk compensation processor module 1610 receives the non-spatial component Ym, and the intermediate component processor 1620 applies a set of filters to generate an enhanced non-spatial crosstalk compensation component Zm. The crosstalk compensation processor module 1610 also receives the spatial subband component Ys and applies a set of filters in the side component processor 1630 to generate an enhanced spatial subband component Es. The intermediate component processor 1620 includes a plurality of filters 1640, such as m intermediate filters 1640 (a), 1640 (b) to 1640 (m). Here, each of the m intermediate filters 1640 processes one of the m frequency bands of the non-spatial component Xm. The intermediate component processor 1620 generates an intermediate crosstalk compensation channel Zm by processing the non-spatial component Xm accordingly. In some embodiments, the intermediate filter 1640 is configured using a frequency response diagram of the non-spatial Xm and crosstalk processing by simulation. In addition, by analyzing the frequency response diagram, any spectral defects that appear as artifacts of the crosstalk processing exceeding a predetermined threshold (e.g., 10 dB), such as peaks or valleys in the frequency response diagram, can be estimated. These artifacts are primarily caused by the addition of the delayed and possibly inverted contralateral signal to its corresponding ipsilateral signal in the crosstalk processing, effectively introducing a comb filter-like frequency response into the final rendered result. An intermediate crosstalk compensation channel Zm can be generated by the intermediate component processor 1620 to compensate for estimated peaks or valleys, where each of the m frequency bands corresponds to a peak or valley. Specifically, based on the specific delays, filter frequencies, and gains applied in the crosstalk processing, the peaks and valleys in the frequency response are shifted up and down, resulting in variable amplification and / or attenuation of energy in specific regions of the spectrum. Each intermediate filter 1640 can be configured to adjust for one or more of the peaks and valleys.

[0210] The side component processor 1630 includes a plurality of filters 1650, such as m side filters 1650 (a), 1650 (b) to 1650 (m). The side component processor 1630 generates a side crosstalk compensation channel Zs by processing the spatial component Xs. In some embodiments, a frequency response diagram of a space X with crosstalk processing can be obtained by simulation. By analyzing the frequency response diagram, any spectrum defects exceeding a predetermined threshold (e.g., 10 dB) that appear as artifacts of the crosstalk processing, such as peaks or valleys in the frequency response diagram, can be estimated. The side crosstalk compensation channel Zs can be generated by the side component processor 1630 to compensate for the estimated peak or valley. Specifically, based on the specific delay, filter frequency, and gain applied in the crosstalk processing, the peaks and valleys in the frequency response are shifted up and down, resulting in variable amplification and / or attenuation of energy in a specific region of the spectrum. Each side filter 1650 can be configured to adjust for one or more of the peaks and valleys. In some embodiments, the intermediate component processor 1620 and the side component processor 1630 may include different numbers of filters.

[0211] In some embodiments, the middle filter 1640 and the side filter 1650 may include a biquad filter having a transfer function defined by equation (7). One method of implementing such a filter is a direct form I topology, as defined by equation (22):

[0212]

[0213] Where X is the input vector and Y is the output vector. Other topologies can be used, depending on their maximum word length and saturation behavior. The biquad can then be used to implement a second-order filter with real-valued input and output. To design a discrete-time filter, a continuous-time filter is first designed and then transformed to discrete time via a bilinear transform. In addition, frequency warping can be used to compensate for the resulting shift in center frequency and bandwidth.

[0214] For example, the peak filter may have an S-plane transfer function defined by equation (23):

[0215]

[0216] Where s is a complex variable, A is the amplitude of the peak, and Q is the filter "quality", and the digital filter coefficients are defined by the following equation (24):

[0217] b0=1+αA

[0218] b1=-2*cos(ω0)

[0219] b2=1-αA

[0220]

[0221] a1=-2cos(ω0)

[0222] where ω0 is the center frequency of the filter in radians, and Furthermore, the filter quality Q can be defined by equation (25):

[0223]

[0224] where is the bandwidth, and f c Is the center frequency. Intermediate filter 1640 is shown as a series connection, and side filter 1650 is shown as a series connection. In some embodiments, intermediate filter 1640 is applied in parallel with intermediate component Xm, and side filter is applied in parallel with side component Xs. In some embodiments, crosstalk compensation processor module 1610 processes each of super intermediate component M1, super side component S1, residual intermediate component M2 and residual side component S2. The filter applied to each component can be different.

[0225] Example Crosstalk Processor

[0226] Fig.17 1 is a block diagram of a crosstalk simulation processor module 1700 according to one or more embodiments. The crosstalk simulation processor module 1700 is an example of the crosstalk processor module 110 or the crosstalk processor module 1224. Some embodiments of the crosstalk simulation processor module 1700 have different components than those described herein. Similarly, in some cases, functions may be allocated between components in a manner different from that described herein.

[0227] The crosstalk simulation processor module 1700 generates the contralateral sound component for output to stereo headphones, thereby providing a speaker-like listening experience on the headphones. L It can be the processed left component 134 / 1220 and the right input channel X R It can be the processed right component 136 / 1222.

[0228] The crosstalk simulation processor module 1700 includes a left head shadow low pass filter 1702, a left head shadow high pass filter 1724, a left crosstalk delay 1704, and a left head shadow gain 1710 to process the left input channel X L The crosstalk simulation processor module 1700 also includes a right head shadow low pass filter 1706, a right head shadow high pass filter 1726, a right crosstalk delay 1708, and a right head shadow gain 1712 to process the right input channel X R The crosstalk simulation processor module 1500 further includes an adding unit 1714 and an adding unit 1716 .

[0229] The left head shadow low pass filter 1702 and the left head shadow high pass filter 1724 are input to the left channel X. L A modulation is applied that models the frequency response of the signal after passing through the listener's head. The output of the left head shadow high pass filter 1724 is provided to the left crosstalk delay 1704 to which a time delay is applied. The time delay represents the transaural distance experienced by the contralateral sound component relative to the ipsilateral sound component. The left head shadow gain 1710 applies a gain to the output of the left crosstalk delay 1704 to generate the right left analog channel W. L .

[0230] Similarly, for the right input channel X R , right head shadow low pass filter 1706 and right head shadow high pass filter 1726 input channel X to the right R A modulation is applied which models the frequency response of the listener's head. The output of the right head shadow high pass filter 1726 is provided to the right crosstalk delay 1708 which applies a time delay. The right head shadow gain 1712 applies a gain to the output of the right crosstalk delay 1708 to generate the right crosstalk analog channel W R .

[0231] The application of the head shadow low pass filter, head shadow high pass filter, crosstalk delay, and head shadow gain to each of the left and right channels may be performed in a different order.

[0232] The adding unit 1714 adds the right crosstalk analog channel W R and left input channel X L Add together to generate the left output channel O L The adding unit 1716 adds the left crosstalk analog channel W L With the right input channel X R Add together to generate the left output channel O R .

[0233] Fig.18 1 is a block diagram of a crosstalk cancellation processor module 1800 according to one or more embodiments. The crosstalk cancellation processor module 1800 is an example of the crosstalk processor module 110 or the crosstalk processor module 1224. Some embodiments of the cancellation processor module 1800 have different components than those described herein. Similarly, in some cases, functions may be allocated between components in a manner different from that described herein.

[0234] The crosstalk cancellation processor module 1800 receives the left input channel X L and right input channel X R , and for channel X L , X RCrosstalk cancellation is performed to generate the left output channel O L and right output channel O R . Left input channel X L It can be the processed left component 134 / 1220 and the right input channel X R It can be the processed right component 136 / 1222.

[0235] The crosstalk cancellation processor module 1800 includes an in-band and out-of-band divider 1810, inverters 1820 and 1822, opposite-side estimators 1830 and 1840, combiners 1850 and 1852, and an in-band and out-of-band frequency band combiner 1860. These components operate together to divide the input channel T L , T R It is divided into in-band components and out-of-band components, and crosstalk cancellation is performed on the in-band components to generate output channel O L , O R .

[0236] By dividing the input audio signal T into different frequency band components and by performing crosstalk cancellation on selective components (e.g., in-band components), crosstalk cancellation can be performed for specific frequency bands while avoiding degradation in other frequency bands. If crosstalk cancellation is performed without dividing the input audio signal T into different frequency bands, the audio signal after such crosstalk cancellation may exhibit significant attenuation or amplification in non-spatial and spatial components at low frequencies (e.g., below 350 Hz), higher frequencies (e.g., above 12000 Hz), or both. By selectively performing crosstalk cancellation in-band (e.g., between 250 Hz and 14000 Hz) (where most of the influential spatial cues are located), a balanced overall energy can be maintained across the spectrum in the mix, especially in the non-spatial components.

[0237] The in-band and out-of-band divider 1810 divides the input channel T L , T R In-band channels T L,In , T R,In and out-of-band channel T L,Out 、T R,Out Specifically, the in-band and out-band divider 1810 divides the left enhancement compensation channel T L Divided into left in-band channel T L,In and the left out-of-band channel T L,Out Similarly, the in-band and out-band divider 1810 divides the right enhancement compensation channel T R Divided into right in-band channel T R,In and right out-of-band channel T R,OutEach in-band channel may cover a portion of a respective input channel corresponding to a frequency range comprising, for example, 250 Hz to 14 kHz. The frequency band range may be adjustable, for example, depending on loudspeaker parameters.

[0238] The inverter 1820 and the contralateral estimator 1830 operate together to generate the left contralateral cancellation component S L , to compensate for the left in-band channel T L,In Similarly, the inverter 1822 and the opposite side estimator 1840 operate together to generate the right opposite side cancellation component S R , to compensate for the right in-band channel T R,In resulting in a contralateral sound component.

[0239] In one approach, inverter 1820 receives the in-band channel T L,In and invert the received in-band channel T L,In The polarity of the channel is reversed to generate an in-band channel T L,In '. The contralateral estimator 1830 receives the inverted in-band channel T L,In ', and extract the inverted in-band channel T by filtering L,In ' corresponds to the opposite sound component. Because the filtering is done on the in-band channel T L,In ' is executed, so the part extracted by the opposite side estimator 1830 becomes the in-band channel T L,In Therefore, the portion extracted by the contralateral estimator 1830 becomes the left contralateral cancellation component S L , which can be added to the control in-band channel T R,In To reduce the in-band channel T L,In In some embodiments, the inverter 1820 and the contralateral estimator 1830 are implemented in a different order.

[0240] The inverter 1822 and the opposite side estimator 1840 are related to the in-band channel T R,In A similar operation is performed to generate the right contralateral cancellation component SR. Therefore, for the sake of brevity, its detailed description is omitted herein.

[0241] In one example implementation, the contralateral estimator 1830 includes a filter 1832, an amplifier 1834, and a delay unit 1836. The filter 1832 receives an inverted input channel T L,In ', and extract the inverted in-band channel T by filtering L,In The portion of the ' corresponding to the opposite sound component. Examples of filter implementations are notch or shelving filters with a center frequency selected between 5000 and 10000 Hz and a Q selected between 0.5 and 1.0. Gain dB (G dB) can be derived from equation (26):

[0242] G dB = -3.0 - log 1.333 (D) (26)

[0243] Where D is the delay amount of the delay unit 1836 in samples, for example, at a sampling rate of 48 kHz. Another implementation is a low-pass filter with a corner frequency selected between 5000 and 10000 Hz and a Q selected between 0.5 and 1.0. In addition, the amplifier 1834 amplifies the extracted portion by a corresponding gain factor G L,In , and the delay unit 1836 delays the amplified output from the amplifier 1834 according to the delay function D to generate the left contralateral cancellation component S L The contralateral estimator 1840 includes a filter 1842, an amplifier 1844, and a delay unit 1846. The delay unit 1846 is used to delay the inverted in-band channel T R,In 'Perform similar operations to generate the right contralateral cancellation component S R In one example, the contralateral estimators 1830, 1840 generate the left and right contralateral cancellation components S according to the following equations: L , S R :

[0244] S L =D[G L, In *F[T L, In ']] (27)

[0245] S R =D[G R, In *F[T R, In ']] (28)

[0246] Where F[] is the filter function and D[] is the delay function.

[0247] The configuration of crosstalk cancellation can be determined by speaker parameters. In one example, the filter center frequency, delay, amplifier gain, and filter gain can be determined based on the angle between the two speakers relative to the listener. In some embodiments, values ​​between the speaker angles are used to interpolate other values.

[0248] Combiner 1850 combines the right contralateral cancellation component SR into the left in-band channel T L,In To generate the left in-band crosstalk channel U L , and combiner 1852 combines the left contralateral cancellation component SL into the right in-band channel T R,In To generate the right in-band crosstalk channel UR The in-band and out-of-band combiner 1860 combines the left in-band crosstalk channel U L With out-of-band channel T L,Out Combined to generate the left output channel O L , and the right in-band crosstalk channel U R With out-of-band channel T R,Out Combined to generate the right output channel O R .

[0249] Therefore, the left output channel O L Includes in-band channel T R,In The right contralateral cancellation component SR corresponding to the anti-phase of the part attributed to the contralateral sound in the right output channel O R Includes in-band channel T L,In The left contralateral cancellation component SL is corresponding to the anti-phase of the part due to the contralateral sound in the right output channel O. R The wavefront of the sound component on the same side output by the right speaker reaching the right ear can be cancelled according to the left output channel O L The wavefront of the opposite sound component output by the left speaker. Similarly, according to the left output channel O L The wavefront of the sound component on the same side output by the left speaker reaching the left ear can be cancelled according to the right output channel O R The wavefront of the contralateral sound component output by the right speaker. Therefore, the contralateral sound component can be reduced to enhance spatial detectability.

[0250] Example PSM Process

[0251] Fig.19 19 is a flow chart of a process 1900 for PSM processing according to one or more embodiments. The process 1900 may include fewer or more steps, and the steps may be performed in a different order. In some embodiments, the PSM processing may be performed using a Hilbert Transform Perceptual Sound Field Modification (HPSM) module.

[0252] The audio processing system (e.g., the PSM module 102 of the audio processing system 100 or 1200) separates 1905 the input channel into a low frequency component and a high frequency component. The crossover frequency defining the boundary between the low frequency component and the high frequency component may be adjustable to ensure that the frequencies of interest to the PSM processing are included in the high frequency component. In some embodiments, the audio processing system applies a gain to the low frequency component and / or the high frequency component.

[0253] An input channel may be a specific portion of an audio signal that is extracted for PSM processing. In some embodiments, an input channel is a mid component or a side component of a (e.g., stereo or multi-channel) audio signal. In some embodiments, an input channel is a super-mid component, a super-side component, a residual mid component, or a residual side component of an audio signal. In some embodiments, an input channel is associated with a sound source, such as a voice or an instrument, which will be combined with other sounds into an audio mix.

[0254] The audio processing system applies 1910 a first Hilbert transform to the high frequency component to generate a first left branch component and a first right branch component, the first left branch component being 90 degrees out of phase with respect to the first right branch component.

[0255] The audio processing system applies 1915 a second Hilbert transform to the first right branch component to generate a second left branch component and a second right branch component, the first left branch component being 90 degrees out of phase with respect to the first right branch component.

[0256] In some embodiments, the audio processing system applies a delay and / or gain to the first left branch component. The audio processing system may apply a delay and / or gain to the second right branch component. These gains and delays may be used to manipulate the perceived results of the PSM processing.

[0257] The audio processing system combines the first left branch component with the low frequency component to generate a left channel 1920. The audio processing system combines the second right branch component with the low frequency component to generate a right channel 1925. The left channel may be provided to a left speaker and the right channel may be provided to a right speaker.

[0258] Fig. 20 2 is a flow chart of another process 2000 for PSM processing using a first-order non-orthogonal rotation-based decorrelation (FNORD) filter network according to some embodiments. Fig. 20 The processes shown in can be performed by components of an audio system (e.g., system 100, 202, or 1200). In other embodiments, other entities can perform Fig. 20 Embodiments may include different and / or additional steps, or perform the steps in a different order.

[0259] The audio system determines 2005 a target amplitude response that defines one or more spatial cues to be encoded into a monophonic audio signal to generate a plurality of resulting channels, wherein the one or more spatial cues are associated with one or more frequency-dependent amplitude cues encoded into the middle / side space of the resulting channels that do not change the overall coloration of the resulting channels. The one or more spatial cues may include at least one elevation cue associated with a target elevation angle. Each elevation cue may correspond to one or more frequency-dependent amplitude cues to be encoded into the middle / side space of the audio signal, such as a target amplitude function corresponding to a narrow region of infinite attenuation at one or more specific frequencies. On the other hand, since the left / right cues of elevation are typically symmetrical in coloration, the left / right signals may be constrained to be colorless. In some embodiments, the spatial cues may be based on sampled HRTFs.

[0260] In some embodiments, the target amplitude response may also define one or more parameterized spatial cues, which may include target broadband attenuation, target subband attenuation, critical points, filter characteristics, and / or the sound field position into which the cues are to be embedded. The critical point may be an inflection point at 3 dB. The filter characteristics may include one of a high-pass filter characteristic, a low-pass characteristic, a bandpass characteristic, or a band-stop characteristic. The sound field position may include a middle or side channel, or, in the case where the number of output channels is greater than two, other subspaces within the output space, such as subspaces determined via paired and / or hierarchical summations and / or differences. One or more spatial cues may be determined based on the characteristics of the presentation device (e.g., the frequency response of the speaker, the position of the speaker), the expected content of the audio data, the listener's perceptual ability in the context, or the minimum quality expectations of the audio presentation system involved. For example, if the speaker cannot adequately reproduce frequencies below 200 Hz, spatial cues embedded in this range should be avoided. Similarly, if the expected audio content is speech, the audio system may select a target amplitude response that affects only the frequency that the ear is most sensitive to, which is within the expected bandwidth of the speech. If the listener will be getting auditory cues from other sources in the context (such as another speaker array in the location), the audio system can determine a target amplitude response that is complementary to those simultaneous cues.

[0261] The audio system determines the transfer function of the 2010 single-input multiple-output all-pass filter based on the target amplitude response. The transfer function defines the relative rotation of the phase angle of the output channels. For each output, the transfer function describes the effect of the filter network on its input as a function of phase angle rotation as a function of frequency.

[0262] The audio system determines the coefficients of the 2015 all-pass filter based on the transfer function. These coefficients will be selected and applied to the input audio stream in a manner that best suits the constraint and / or clue type. Some examples of coefficient sets are defined in equations (12), (13), (17) and (19). In some embodiments, determining the coefficients of the all-pass filter based on the transfer function includes using an inverse discrete Fourier transform (IDFT). In this case, the coefficient set can be determined as defined in equation (19). In some embodiments, determining the coefficients of the all-pass filter based on the transfer function includes using a phase vocoder. In this case, the coefficient set can be determined as defined in equation (19), except that these will be applied to the frequency domain before resynthesizing the time domain data. In some embodiments, the coefficients include at least a rotation control parameter and a first-order coefficient, which is determined based on received critical point parameters, filter characteristic parameters and sound field position parameters.

[0263] The audio system processes 2020 the mono channel with coefficients of the all-pass filter to generate a plurality of channels. For example, in some embodiments, the all-pass filter module receives the mono audio channel and performs a wideband phase rotation on the mono audio channel based on the rotation control parameter to generate a plurality of wideband rotated component channels (e.g., left and right wideband rotated components), and performs a narrowband phase rotation on at least one of the plurality of wideband rotated component channels based on the first-order coefficients to determine a narrowband rotated component channel, which, together with one or more remaining channels of the wideband rotated component channels, forms a plurality of channels output by the audio system.

[0264] In some embodiments, if the system operates in the time domain using an IIR implementation, as in equation (8), the coefficients can scale the appropriate feedback and feedforward delays. If a FIR implementation is used, as in equation (19), only the feedforward delay can be used. If the coefficients are determined and applied in the spectral domain, they can be applied to the spectral data as complex multiplications before resynthesis. The audio system can provide multiple output channels to a presentation device, such as a user device connected to the audio system via a network.

[0265] The example PSM processing flows described above all utilize all-pass filter networks to encode spatial cues by perceptually placing mono content at specific locations in the sound field (e.g., locations associated with target elevation angles.) Because the all-pass filter networks described herein are colorless, these filters allow the user to decouple the spatial placement of audio from its overall coloration.

[0266] Orthogonal component space processing

[0267] Fig.2121 is a flow chart of a process 2100 for spatial processing using at least one of a super-mid component, a residual mid component, a super-side component, or a residual side component according to one or more embodiments. The spatial processing may include gain application, amplitude or delay-based panning, binaural processing, reverberation, dynamic range processing (such as compression and limiting), linear or non-linear audio processing techniques and effects, chorus effects, flanger effects, machine learning-based vocal or instrumental style transfer, conversion or resynthesis methods, etc. The process is performed to provide spatially enhanced audio to a user's device. The process may include fewer or more steps, and the steps may be performed in a different order.

[0268] An audio processing system (e.g., audio processing system 1200) receives 2110 an input audio signal (e.g., left channel 1202 and right input channel 1204). In some embodiments, the input audio signal may be a multi-channel audio signal including a plurality of left and right channel pairs. Each left and right channel pair may be processed as discussed herein for the left input channel and the right input channel.

[0269] The audio processing system generates 2120 a non-spatial mid component (e.g., mid component 1208) and a spatial side component (e.g., side component 1210) from an input audio signal. In some embodiments, an L / R to M / S converter (e.g., L / R to M / S converter module 1206) performs conversion of the input audio signal into the mid component and the side component.

[0270] The audio processing system generates 2130 at least one of a super-middle component (e.g., super-middle component M1), a super-side component (e.g., super-side component S1), a residual middle component (e.g., residual middle component M2), and a residual side component (e.g., residual side component S2). The audio processing system may generate at least one and / or all of the components listed above. The super-middle component includes the spectral energy of the side component removed from the spectral energy of the middle component. The residual middle component includes the spectral energy of the super-middle component removed from the spectral energy of the middle component. The super-side component includes the spectral energy of the middle component removed from the spectral energy of the side component. The residual side component includes the spectral energy of the super-side component removed from the spectral energy of the side component. The processing used to generate M1, M2, S1, or S2 may be performed in the frequency domain or in the time domain.

[0271] The audio processing system filters at least one of the super-mid component, the residual mid component, the super-side component, and the residual side component 2140 to enhance the audio signal. The filtering may include HPSM processing, in which a series of Hilbert transforms are applied to high frequency components of the residual mid component, the super-side component, the residual side component, or the super-mid component. In one example, the super-mid component receives HPSM processing, while one or more of the residual mid component, the super-side component, or the residual side component receives other types of filtering.

[0272] The filtering may include PSM processing, where the HRTF data is sampled by anthropometric sampling as discussed above in conjunction with equation (20) or by Fig. 10A and Fig. 10B The parameter specification of the spatial cues discussed in more detail uncolor encodes the spatial cues.In one example, the super mid component receives PSM processing while one or more of the residual mid component, the super side component, or the residual side component receives no filtering or other type of filtering.

[0273] The filtering may include other types of filtering, such as spatial cue processing. Spatial cue processing may include adjusting the frequency-dependent amplitude or frequency-dependent delay of the super-mid component, residual mid component, super-side component, or residual side component. Some examples of spatial cue processing include amplitude- or delay-based panning or binaural processing.

[0274] Filtering may include dynamic range processing, such as compression or limiting. For example, when a threshold level for compression is exceeded, the super middle component, residual middle component, super side component, or residual side component may be compressed according to a compression ratio. In another example, when a threshold level for limiting is exceeded, the super middle component, residual middle component, super side component, or residual side component may be limited to a maximum level.

[0275] The filtering may include machine learning based changes to the super-mid component, the residual mid component, the super-side component, or the residual side component. Some examples include machine learning based vocal or instrumental style transfer, conversion, or resynthesis.

[0276] The filtering of the super-mid component, residual mid component, super-side component or residual side component may include gain application, reverberation, and other linear or non-linear audio processing techniques and effects ranging from chorus and / or flanging or other types of processing. In some embodiments, the filtering may include filtering for sub-band spatial processing and crosstalk compensation, as described below in conjunction with Fig. 22 discussed in more detail.

[0277] Filtering can be performed in the frequency domain or the time domain. In some embodiments, the mid-component and the side component are converted from the time domain to the frequency domain, supercomponents and / or residual components are generated in the frequency domain, filtering is performed in the frequency domain, and the filtered components are converted to the time domain. In other embodiments, supercomponents and / or residual components are converted to the time domain, and filtering is performed on these components in the time domain.

[0278] The audio processing system generates 2150 a left output channel (e.g., left output channel 1242) and a right output channel (e.g., right output channel 1244) using one or more filtered super components / residual components. For example, a conversion from M / S to L / R may be performed using a mid component or a side component generated based on at least one of a filtered super mid component, a filtered residual mid component, a filtered super side component, or a filtered residual side component. In another example, a filtered super mid component or a filtered residual mid component may be used as a mid component for an M / S to L / R conversion, or a filtered super side component or a residual side component may be used as a side component for an M / S to L / R conversion.

[0279] Orthogonal component sub-band space and crosstalk processing

[0280] Fig. 222200 is a flowchart of a process for performing sub-band spatial processing and crosstalk processing compensation using at least one of a super-mid component, a residual mid component, a super-side component, or a residual side component according to one or more embodiments. Crosstalk processing may include crosstalk cancellation or crosstalk simulation. Sub-band spatial processing may be performed to provide audio content with enhanced spatial detectability, such as by creating a perception that sound is directed to the listener from a large area rather than from a specific point in space corresponding to the location of the speaker (e.g., sound field enhancement), thereby bringing a more immersive listening experience to the listener. Crosstalk simulation may be used for audio output to headphones to simulate a speaker experience with contralateral crosstalk. Crosstalk cancellation may be used for audio output to speakers to remove the effects of crosstalk interference. Crosstalk compensation is used to compensate for spectral defects caused by crosstalk cancellation or crosstalk simulation. The process may include fewer or more steps, and the steps may be performed in different orders. Super-mid components / side components and residual mid components / side components may be manipulated in different ways to achieve different purposes. For example, in the case of crosstalk compensation, target subband filtering may be applied only to the super-mid component M1 (where most of the vocal dialogue energy in much movie content occurs) in an effort to remove spectral artifacts produced by the crosstalk processing only in that component. In the case of sound field enhancement with or without crosstalk processing, target subband gains may be applied to the residual mid component M2 and the residual side component S2. For example, the residual mid component M2 may be attenuated and the residual side component S2 may be inversely amplified to increase the distance between these components from a gain perspective (and, if done tastefully, to increase spatial detectability) without producing a drastic overall change in perceived loudness in the final L / R signal, while also avoiding attenuation in the super-mid component M1 (e.g., the portion of the signal that typically contains most of the sound energy).

[0281] The audio processing system receives 2210 an input audio signal, the input audio signal comprising a left channel and a right channel. In some embodiments, the input audio signal may be a multi-channel audio signal comprising a plurality of left and right channel pairs. Each left and right channel pair may be processed as discussed herein for a left input channel and a right input channel.

[0282] The audio processing system applies 2220 crosstalk processing to the received input audio signal. The crosstalk processing includes at least one of crosstalk simulation and crosstalk cancellation.

[0283] In steps 2230 to 2260, the audio processing system uses one or more of the super mid component, the super side component, the residual mid component, or the residual side component to perform subband spatial processing and crosstalk compensation for crosstalk processing. In some embodiments, crosstalk processing may be performed after the processing in steps 2230 to 2260.

[0284] The audio processing system generates 2230 a mid component and a side component from the (eg, crosstalk processed) audio signal.

[0285] The audio processing system generates 2240 at least one of a super mid component, a residual mid component, a super side component, and a residual side component. The audio processing system may generate at least one and / or all of the components listed above.

[0286] The audio processing system filters 2250 a subband of at least one of the super-mid component, the residual mid component, the super-side component, and the residual side component to apply subband spatial processing to the audio signal. Each subband may include a frequency range, such as may be defined by a critical band set. In some embodiments, the subband spatial processing also includes time delaying a subband of at least one of the super-mid component, the residual mid component, the super-side component, and the residual side component. In some embodiments, filtering includes applying HPSM processing.

[0287] The audio processing system filters 2260 at least one of the super-mid component, the residual mid component, the super-side component, and the residual side component to compensate for spectral defects from the crosstalk processing of the input audio signal. The spectral defects may include peaks or valleys in a frequency response graph of the super-mid component, the residual mid component, the super-side component, or the residual side component that exceed a predetermined threshold (e.g., 10 dB) that appear as artifacts of the crosstalk processing. The spectral defects may be estimated spectral defects.

[0288] In some embodiments, the filtering of spectral quadrature components for sub-band spatial processing in step 2250 and the crosstalk compensation in step 2260 may be integrated into a single filtering operation for each spectral quadrature component selected for filtering.

[0289] In some embodiments, filters for super-mid component / side component / residual mid component / side component for sub-band spatial processing or crosstalk compensation may be performed in conjunction with filtering for other purposes, such as gain application, amplitude or delay-based panning, binaural processing, reverberation, dynamic range processing (such as compression and limiting), linear or non-linear audio processing techniques and effects (from chorus and / or flanging to machine learning-based vocal or instrumental style transfer, conversion or resynthesis methods), or other types of processing using any of the super-mid component, residual mid component, super-side component, and residual side component.

[0290] Filtering can be performed in the frequency domain or the time domain. In some embodiments, the mid-component and the side component are converted from the time domain to the frequency domain, supercomponents and / or residual components are generated in the frequency domain, filtering is performed in the frequency domain, and the filtered components are converted to the time domain. In other embodiments, supercomponents and / or residual components are converted to the time domain, and filtering is performed on these components in the time domain.

[0291] The audio processing system generates 2270 left and right output channels from the filtered super-mid component. In some embodiments, the left and right output channels are additionally based on at least one of the filtered residual mid component, the filtered super-side component, and the filtered residual side component.

[0292] Sample Computer

[0293] Fig.23 2300 is a block diagram of a computer 2300 according to some embodiments. The computer 2300 is an example of a computing device that includes circuitry that implements an audio system such as the audio system 100, 202, or 1200. Illustrated is at least one processor 2302 coupled to a chipset 2304. The chipset 2304 includes a memory controller hub 2320 and an input / output (I / O) controller hub 2322. The memory 2306 and the graphics adapter 2312 are coupled to the memory controller hub 2320, and the display device 2318 is coupled to the graphics adapter 2312. The storage device 2308, the keyboard 2310, the pointing device 2314, and the network adapter 2316 are coupled to the I / O controller hub 2322. The computer 2300 may include various types of input or output devices. Other embodiments of the computer 2300 have different architectures. For example, in some embodiments, the memory 2306 is directly coupled to the processor 2302.

[0294] The storage device 2308 includes one or more non-transitory computer-readable storage media, such as a hard drive, a compact disk read-only memory (CD-ROM), a DVD, or a solid-state memory device. The memory 2306 stores program code (including one or more instructions) and data used by the processor 2302. The program code may correspond to the reference Figures 1 to 3 Processing aspects of the description.

[0295] Pointing device 2314 is used in conjunction with keyboard 2310 to enter data into computer system 2300. Graphics adapter 2312 displays images and other information on display device 2318. In some embodiments, display device 2318 includes touch screen functionality for receiving user input and selections. Network adapter 2316 couples computer system 2300 to a network. Some embodiments of computer 2300 have a computer system that is compatible with the Internet. Fig.23 Different and / or other components than those shown in .

[0296] The circuit may include one or more processors executing program code stored in a non-transitory computer readable medium, which program code, when executed by the one or more processors, configures the one or more processors to implement an audio system or a module of an audio system. Other examples of circuits implementing an audio system or a module of an audio system may include an integrated circuit, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other types of computer circuits.

[0297] Additional Considerations

[0298] Example benefits and advantages of the disclosed configuration include dynamic audio enhancements due to an enhanced audio system that adapts to the device and the associated audio rendering system, and other relevant information provided by the device OS, such as use case information (e.g., indicating that the audio signal is used for music playback rather than gaming). The enhanced audio system can be integrated into the device (e.g., using a software development kit) or stored on a remote server for on-demand access. In this way, the device does not need to devote storage or processing resources to maintain an audio enhancement system specific to its audio rendering system or audio rendering configuration. In some embodiments, the enhanced audio system enables different levels of querying of the rendering system information, so that effective audio enhancements can be applied on different levels of available device-specific rendering information.

[0299] Throughout this specification, multiple instances can realize the components, operations or structures described as single instances. Although the individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations can be performed simultaneously, and the operations are not required to be performed in the illustrated order. The structure and functionality presented as separate components in the example configuration can be implemented as a combined structure or component. Similarly, the structure and functionality presented as a single component can be implemented as separate components. These and other changes, modifications, additions and improvements fall within the scope of this paper theme.

[0300] Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. A module may constitute a software module (e.g., code embodied on a machine-readable medium or in a transmission signal) or a hardware module. A hardware module is a tangible unit that is capable of performing certain operations and that may be configured or arranged in a certain manner. In an example embodiment, one or more computer systems (e.g., independent client or server computer systems) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

[0301] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. In some example embodiments, the modules mentioned herein may include processor-implemented modules.

[0302] Similarly, the methods described herein may be implemented at least in part by a processor. For example, at least some operations of a method may be performed by one or more processors or a hardware module implemented by a processor. The performance of certain operations may be distributed between one or more processors, not only residing in a single machine, but also deployed on multiple machines. In some example embodiments, one or more processors may be located in a single location (e.g., in a home environment, in an office environment, or as a server farm), while in other embodiments, the processor may be distributed in multiple locations.

[0303] Unless otherwise specifically stated, discussions herein using terms such as "processing," "computing," "calculating," "determining," "presenting," "displaying," and the like may refer to the actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0304] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment. The phrase "in one embodiment" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0305] Some embodiments may be described using the expressions "coupled" and "connected" and their derivatives. It should be understood that these terms are not intended to be synonymous with each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited to this context.

[0306] As used herein, the terms "comprises," "includes," "has," or any other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or device that includes a series of elements need not be limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or device. In addition, unless expressly stated to the contrary, "or" refers to an inclusive or rather than an exclusive. For example, a condition A or B is satisfied by any of the following conditions: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0307] In addition, "a" or "an" is used to describe the elements and components of the embodiments herein. This is done only for convenience and to give a general sense of the invention. This description should be read to include one or at least one, and the singular also includes the plural, unless otherwise apparent.

[0308] Some parts of this specification sheet describe embodiments according to algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are usually used by technical personnel in the field of data processing to effectively convey their work essence to other technical personnel in the field. Although these operations are described functionally, computationally or logically, they should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without loss of generality, it is sometimes proved to be very convenient to refer to these operational arrangements as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0309] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product that includes a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0310] Embodiments may also relate to apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in this specification may include a single processor or may be an architecture that employs a multi-processor design to increase computing power.

[0311] Embodiments may also relate to products produced by the computing processes described herein. Such products may include information produced by the computing processes, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.

[0312] After reading this disclosure, those skilled in the art will appreciate additional alternative structural and functional designs for systems and processes for audio content decorrelation through the principles disclosed herein. Therefore, although specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations that are obvious to those skilled in the art may be made to the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

[0313] Finally, the language used in this specification is selected primarily for readability and instructional purposes, and it is not selected to describe or limit the patent rights. Therefore, the scope of the patent rights is not limited by this detailed description, but by any claims issued in an application based on this. Therefore, the disclosure of the embodiments is intended to illustrate, not to limit, the scope of the patent rights set forth in the appended claims.

Claims

1. A system comprising: one or more processors; as well as A non-transitory computer readable medium comprising stored program code, which when executed by the one or more processors configures the one or more processors to: Separate audio channels into low-frequency and high-frequency components; applying a first Hilbert transform to the high frequency component to generate a first left branch component and a first right branch component, the first left branch component being 90 degrees out of phase with respect to the first right branch component; applying a second Hilbert transform to the first right branch component to generate a second left branch component and a second right branch component, the second left branch component being 90 degrees out of phase with respect to the second right branch component; combining the first left branch component with the low frequency component to generate a left channel; as well as The second right branch component is combined with the low frequency component to generate a right channel.

2. The system of claim 1, wherein the program code further configures the one or more processors to: apply a first gain to the low frequency component and apply a second gain to the high frequency component, the first gain and the second gain being different.

3. The system of claim 1, wherein the program code further configures the one or more processors to: apply a first delay to the first left branch component and apply a second delay to the second right branch component, the first delay and the second delay being different.

4. The system of claim 1, wherein the program code further configures the one or more processors to: apply a first gain to the first left branch component, and apply a second gain to the second right branch component, the first gain and the second gain being different.

5. The system of claim 1, wherein: The program code configuring the one or more processors to apply the first Hilbert transform to the high frequency component configures the one or more processors to: applying a first series of all-pass filters to the high frequency component to generate the first left branch component; as well as Applying a first delay and a second series of all-pass filters to the high frequency component to generate the first right branch component; and The program code configuring the one or more processors to apply the second Hilbert transform to the first right branch component configures the one or more processors to: applying a third series of all-pass filters to the first right branch component to generate the second left branch component; and A second delay and a fourth series of all-pass filters are applied to the first right branch component to generate the second right branch component.

6. The system of claim 1 , wherein the program code further configures the one or more processors to generate the audio channel by: generating a mid component and a side component from a left input channel and a right input channel of an audio signal; and A super mid component is generated, the super mid component including the spectral energy of the side component removed from the spectral energy of the mid component.

7. The system of claim 1, wherein the program code further configures the one or more processors to: generating a mid component and a side component from the left channel and the right channel; applying a filter to the mid component and the side component; and A left output channel and a right output channel are generated from the filtered mid component and the filtered side component.

8. The system of claim 1, wherein the program code further configures the one or more processors to generate the audio channels by combining channels of a multi-channel audio signal.

9. The system of claim 1, wherein the program code further configures the one or more processors to generate the audio channel by isolating a portion of an audio signal.

10. The system of claim 1, wherein the high frequency component comprises audio for speech.

11. A non-transitory computer readable medium comprising stored program code, which when executed by one or more processors configures the one or more processors to: Separate audio channels into low-frequency and high-frequency components; applying a first Hilbert transform to the high frequency component to generate a first left branch component and a first right branch component, the first left branch component being 90 degrees out of phase with respect to the first right branch component; applying a second Hilbert transform to the first right branch component to generate a second left branch component and a second right branch component, the second left branch component being 90 degrees out of phase with respect to the second right branch component; combining the first left branch component with the low frequency component to generate a left channel; as well as The second right branch component is combined with the low frequency component to generate a right channel.

12. The computer-readable medium of claim 11, wherein the program code further configures the one or more processors to apply a first gain to the low frequency component and a second gain to the high frequency component, the first gain and the second gain being different.

13. The computer-readable medium of claim 11, wherein the program code further configures the one or more processors to apply a first delay to the first left branch component and a second delay to the second right branch component, the first delay and the second delay being different.

14. The computer-readable medium of claim 11, wherein the program code further configures the one or more processors to apply a first gain to the first left branch component and a second gain to the second right branch component, the first gain and the second gain being different.

15. The computer-readable medium of claim 11, wherein: The program code configuring the one or more processors to apply the first Hilbert transform to the high frequency component configures the one or more processors to: applying a first series of all-pass filters to the high frequency component to generate the first left branch component; as well as Applying a first delay and a second series of all-pass filters to the high frequency component to generate the first right branch component; and The program code configuring the one or more processors to apply the second Hilbert transform to the first right branch component configures the one or more processors to: applying a third series of all-pass filters to the first right branch component to generate the second left branch component; and A second delay and a fourth series of all-pass filters are applied to the first right branch component to generate the second right branch component.

16. The computer-readable medium of claim 11, wherein the program code further configures the one or more processors to generate the audio channel by: generating a mid component and a side component from a left input channel and a right input channel of an audio signal; and A super mid component is generated, the super mid component including the spectral energy of the side component removed from the spectral energy of the mid component.

17. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to: generating a mid component and a side component from the left channel and the right channel; applying a filter to the mid component and the side component; as well as A left output channel and a right output channel are generated from the filtered mid component and the filtered side component.

18. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to generate the audio channels by combining channels of a multi-channel audio signal.

19. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to generate the audio channel by isolating a portion of an audio signal.

20. The computer-readable medium of claim 11, wherein the high frequency components include audio for speech.

21. A method comprising, by one or more processors: Separate audio channels into low-frequency and high-frequency components; applying a first Hilbert transform to the high frequency component to generate a first left branch component and a first right branch component, the first left branch component being 90 degrees out of phase with respect to the first right branch component; applying a second Hilbert transform to the first right branch component to generate a second left branch component and a second right branch component, the second left branch component being 90 degrees out of phase with respect to the second right branch component; combining the first left branch component with the low frequency component to generate a left channel; as well as The second right branch component is combined with the low frequency component to generate a right channel.

22. The method of claim 21, further comprising applying, by the one or more processors, a first gain to the low frequency component and a second gain to the high frequency component, the first gain and the second gain being different.

23. The method of claim 21, further comprising applying, by the one or more processors, a first delay to the first left branch component and a second delay to the second right branch component, the first delay and the second delay being different.

24. The method of claim 21, further comprising applying, by the one or more processors, a first gain to the first left branch component and a second gain to the second right branch component, the first gain and the second gain being different.

25. The method of claim 21, wherein: Applying the first Hilbert transform to the high frequency component comprises: applying the first series of all-pass filters to the high frequency component to generate a first left branch component; and Applying a first delay and a second series of all-pass filters to the high frequency component to generate a first right branch component; and Applying the second Hilbert transform to the first right branch component comprises: applying a third series of all-pass filters to the first right branch component to generate the second left branch component; and A second delay and a fourth series of all-pass filters are applied to the first right branch component to generate the second right branch component.

26. The method of claim 21, further comprising generating, by the one or more processors, the audio channels by: generating a mid component and a side component from a left input channel and a right input channel of an audio signal; A super mid component is generated, the super mid component including the spectral energy of the side component removed from the spectral energy of the mid component.

27. The method of claim 21, further comprising, by the one or more processors: generating a mid component and a side component from the left channel and the right channel; applying a filter to the mid component and the side component; and A left output channel and a right output channel are generated from the filtered mid component and the filtered side component.

28. The method of claim 21, further comprising, by the one or more processors: generating the audio channels by combining channels of a multi-channel audio signal.

29. The method of claim 21, further comprising, by the one or more processors: generating the audio channel by isolating a portion of an audio signal.

30. The method of claim 31, wherein the high frequency component comprises audio for speech.

Images