Colorless generation of elevation perceptual cues using all-pass filter networks
By encoding spatial suggestions into a monaural signal using a single-input, multiple-output all-pass filter, the perception of spatial cues in audio content is enhanced, improving immersion and clarity without requiring multiple amplifiers or speakers.
Patent Information
- Application Number
- JP2024166285
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-01
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-07-07
AI Technical Summary
Existing audio encoding technologies fail to effectively convey spatial cues in audio content, limiting the perception of space and immersion in sound fields.
Encoding spatial suggestions along a sagittal plane into a monaural signal using a single-input, multiple-output all-pass filter to generate multiple channels, applying frequency-dependent phase shifts and amplitude responses to create a sense of space.
Enhances the perception of spatial cues in audio, improving immersion and clarity by altering the perceptual spatial quality of audio channels, reducing the need for multiple amplifiers and speakers.
Smart Images

Figure 2024169670000001_ABST
Abstract
Description
[Technical field]
[0001] FIELD This disclosure relates generally to audio processing, and more specifically to encoding spatial cues into audio content. [Background technology]
[0002] Audio content may be encoded to include spatial characteristics of a sound field, allowing a user to perceive a sense of space in the sound field. For example, audio from a particular source (e.g., a voice or an instrument) may be mixed with the audio content in a manner that creates a sense of space associated with the audio, such as the perception that the audio is coming to the user from a particular direction of arrival, or is located in a particular type of location (e.g., a small room, a large auditorium, etc.). Summary of the Invention
[0003] Some embodiments include a method for encoding spatial implications along a sagital plane into a monaural signal to generate multiple resultant channels, the method including determining, by a processing circuit, a target amplitude response for a mid or side component of the resulting multiple channels based on the spatial implications associated with a frequency-dependent phase shift, converting the target amplitude response for either the mid or side component into a transfer function for a single-input multiple-output allpass filter, and processing the monaural signal using the allpass filter, the allpass filter being configured based on the transfer function.
[0004] Some embodiments include a system for generating multiple channels from a mono channel, where the multiple channels are encoded with one or more spatial implications. The system includes one or more computing devices configured to determine a target amplitude response for a mid or side component of the multiple channels based on the spatial implications associated with a frequency-dependent phase shift. The one or more computers are further configured to convert the target amplitude response for either the mid or side component to a transfer function for a single-input multiple-output all-pass filter and process the mono signal using the all-pass filter, where the all-pass filter is configured based on the transfer function.
[0005] Some embodiments include a non-transitory computer-readable medium including instructions stored thereon for generating multiple channels from a mono channel, where the multiple channels are encoded with one or more spatial implications, which instructions, when executed by the at least one processor, configure the at least one processor to: determine a target amplitude response for a mid or side component of the resulting multiple channels based on the spatial implications associated with a frequency dependent phase shift, transform the target amplitude response for the mid or side component to a transfer function of a single-input multiple-output all-pass filter, and process the mono signal using the all-pass filter, where the all-pass filter is configured based on the transfer function.
[0006] Some embodiments relate to spatially shifting a portion of audio content (e.g., vocalizations) using a series of Hilbert Transforms. Some embodiments include one or more processors and a non-transitory computer-readable medium. The computer-readable medium includes stored program code that, when executed by the one or more processors, configures the one or more processors to: separate an audio channel into low and high frequency components; apply a first Hilbert Transform to the high frequency components to generate a first left leg component and a first right leg component, where the first left leg component is 90 degrees out of phase with the first right leg component; apply a second Hilbert Transform to the first right leg component to generate a second left leg component and a second right leg component, where the second left leg component is 90 degrees out of phase with the second right leg component; combine the first left leg component with the low frequency components to generate a left channel; and combine the second right leg component with the low frequency components to generate a right channel.
[0007] Some embodiments include a non-transitory computer readable medium including stored program code that, when executed by one or more processors, configures the one or more processors to: separate an audio channel into low and high frequency components, apply a first Hilbert transform to the high frequency components to generate a first left leg component and a first right leg component, where the first left leg component is 90 degrees out of phase with the first right leg component, apply a second Hilbert transform to the first right leg component to generate a second left leg component and a second right leg component, where the second left leg component is 90 degrees out of phase with the second right leg component, combine the first left leg component with the low frequency components to generate a left channel, and combine the second right leg component with the low frequency components to generate a right channel.
[0008] Some embodiments include a method executed by one or more processors that includes separating an audio channel into low and high frequency components, applying a first Hilbert transform to the high frequency components to generate a first left leg component and a first right leg component, the first left leg component being 90 degrees out of phase with the first right leg component, applying a second Hilbert transform to the first right leg component to generate a second left leg component and a second right leg component, the second left leg component being 90 degrees out of phase with the second right leg component, combining the first left leg component with the low frequency components to generate a left channel, and combining the second right leg component with the low frequency components to generate a right channel. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram illustrating an audio processing system according to some embodiments. [Diagram 2] FIG. 1 is a block diagram illustrating a computing system environment according to some embodiments. [Diagram 3] 1 illustrates a graph showing sampled HRTFs measured at an elevation angle of 60 degrees in accordance with some embodiments. [Figure 4] FIG. 1 illustrates a graph showing an example of a perceptual cue characterized by a target amplitude function corresponding to a narrow region of infinite attenuation at 11 kHz, according to some embodiments. [Diagram 5] 1 illustrates a frequency plot produced by driving a second-order all-pass filter section having coefficients shown in Table 1 with white noise in accordance with some embodiments. [Figure 6] FIG. 2 is a block diagram illustrating a PSM module implemented using a Hilbert transform in accordance with one or more embodiments. [Figure 7] 2 is a block diagram illustrating a Hilbert transformer module according to one or more embodiments. [Figure 8] FIG. 7 illustrates a frequency plot produced by driving the HPSM module in FIG. 6 with white noise, showing the output frequency response of the sum of multiple channels (mid) and the difference of multiple channels (side), in accordance with some embodiments. [Figure 9] FIG. 2 is a block diagram illustrating a PSM module implemented using an FNORD filter network according to some embodiments. [Figure 10A] FIG. 9 is a detailed block diagram illustrating a PSM module 900 according to some embodiments. [Figure 10B] FIG. 2 is a block diagram illustrating a Broadband Phase Rotator implemented within an all-pass filter module of a PSM module according to some embodiments. [Figure 11] FIG. 13 illustrates a frequency response graph showing the output frequency response of an FNORD filter network configured to achieve an amplitude response for a 60 degree vertical cue in accordance with some embodiments. [Figure 12] FIG. 10 is a block diagram illustrating an audio processing system 1000 according to one or more embodiments. [Figure 13A] FIG. 2 is a block diagram illustrating an orthogonal component generator according to one or more embodiments. [Figure 13B] FIG. 2 is a block diagram illustrating a quadrature component generator according to one or more embodiments. [Figure 13C] FIG. 2 is a block diagram illustrating a quadrature component generator according to one or more embodiments. [Figure 14A] FIG. 4 is a block diagram illustrating a quadrature component processor module according to one or more embodiments. [Figure 14B] 1 shows a block diagram illustrating a quadrature component processor module according to one or more embodiments. [Figure 15]FIG. 2 is a block diagram illustrating a subband spatial processor module according to one or more embodiments. [Figure 16] FIG. 2 is a block diagram illustrating a crosstalk compensation processor module according to one or more embodiments. [Figure 17] FIG. 2 is a block diagram illustrating a crosstalk simulation processor module according to one or more embodiments. [Figure 18] FIG. 2 is a block diagram illustrating a crosstalk cancellation processor module according to one or more embodiments. [Figure 19] 1 is a flowchart illustrating a process for PSM processing using a Hilbert Transform Perceptual Soundstage Modification (HPSM) module in accordance with one or more embodiments. [Figure 20] 1 is a flowchart illustrating another process for PSM processing using a First Order Non-Orthogonal Rotation-Based Decorrelation (FNORD) filter network in accordance with some embodiments. [Figure 21] 1 is a flowchart illustrating a process for spatial processing using at least one of a hyper mid component, a residual mid component, a hyper side component, or a residual side component in accordance with one or more embodiments. [Figure 22]FIG. 10 is a flowchart illustrating a process for subband spatial processing and compensation for crosstalk processing using at least one of a hyper-mid component, a residual mid component, a hyper-side component, or a residual side component in accordance with one or more embodiments. [Diagram 23] FIG. 1 is a block diagram illustrating a computer according to some embodiments.
[0010] The figures depict various embodiments for purposes of illustration only, and those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be used without departing from the principles described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] [Detailed Description] The drawings (FIG.) and the following description relate to preferred embodiments for purposes of illustration only. It should be noted from the following description that alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be used without departing from the principles of the claimed subject matter.
[0012] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It should be noted that, wherever possible, similar or similar reference numbers may be used in the figures and may indicate similar or similar functionality. The figures depict embodiments of the disclosed system (or method) for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be used without departing from the principles described herein.
[0013] Encoding spatial perceptual cues into a mono audio source may be desirable in a variety of applications involving the presentation of multiple simultaneous streams of audible content. Examples of such applications include: Conferencing use case: The addition of spatial cues applied to one or more distant talkers can improve overall voice intelligibility and contribute to the listener's overall sense of immersion. Video and music playback / streaming use cases: One or more audio channels, or signal components of one or more audio channels, can be enhanced by adding spatial cues to improve the intelligibility or sense of space of sounds or other elements of a mix. Collaborative Viewing Entertainment Use Case: Where the streams are separate content channels, such as one or more remote talkers and entertainment program material, that need to be mixed to form an immersive experience, applying spatial perception cues to one or more elements can enhance the sense of perceptual distinction between the elements of the mix and increase the listener's perceptual bandwidth.
[0014] Embodiments relate to audio systems that modify the perceived spatial quality (e.g., sound stage and overall position relative to the head of a target listener) of one or more audio channels. In some embodiments, modifying the perceived spatial quality of an audio channel can be used to separate the coloration of a particular source from its perceived position in space and / or to reduce the required number of amplifiers and speakers needed to encode such effects.
[0015] The audio signal processing performed by the audio system is referred to as Perceptual Sound Stage Modification (PSM) processing. The perceptual result of PSM processing is referred to herein as spatial shifting. The psychoacoustic effect is typically experienced by the user as a general shifting of the sound source above, around, or towards the head, perceptually distinguishing the sound source from other parts of the audio content. This psychoacoustic effect results from the phase and time relationship between the left and right channels, as accentuated by a network of all-pass filters and delays. In some embodiments, this network of filters and delays may be implemented as one or more second-order allpass sections, such as a series of Hilbert transforms, or using a first-order non-orthogonal rotation-based decorrelation (FNORD) filter network, each of which will be described in more detail below. The perceptual result of PSM processing may vary depending on various listening configurations (e.g., headphones or speakers, etc.). For some content and algorithm configurations, the result may be the impression that the perceived signal is spread (e.g., diffuse) around the listener's head. For mono input signals (e.g., non-spatial audio signals), the diffusion effect of PSM processing can be used for mono-to-stereo upmixing.
[0016] In some embodiments, the audio system may separate a target portion of an audio signal from a remaining portion of the audio signal, apply various configurations of PSM processing to perceptually shift the target portion, and mix the processing result back with the remaining portion (e.g., unprocessed or differently processed). Such a system may be perceived as clarifying, elevating, or otherwise differentiating the target portion within the overall audio mix. In some embodiments, PSM processing is used to perceptually shift a portion of an audio signal, including a sung voice or a spoken voice. By convention, the voice in a television, movie, or music audio stream is often located in the center of the sound stage and is therefore part of the mid component (also called the non-spatial or correlated component) of a stereo or multi-channel audio signal. Thus, PSM processing may be applied to the mid component of the audio signal, or to the hyper-mid component, which includes the spectral energy of the side components (also called the spatial or non-correlated components) that are removed from the spectral energy of the mid component.
[0017] PSM processing may be combined with other types of processing. For example, an audio system may apply processing to the shifted portion of the audio signal to perceptually transform the shifted portion and distinguish it from other components in the mix. These types of additional processing may include one or more of single-band or multi-band equalization, single-band or multi-band dynamics processing (e.g., limiting, compression, expansion, etc.), single-band or multi-band gain or delay, crosstalk processing (e.g., crosstalk cancellation processing and / or crosstalk simulation processing, etc.), or compensation for crosstalk processing. In some embodiments, PSM processing may be performed together with mid / side processing, such as subband spatial processing, in which subbands of mid and side components of the audio signal generated via PSM processing are gain adjusted to enhance the spatial sensation of the sound field.
[0018] Separation of audio channels for PSM processing may be accomplished in a variety of ways. In some embodiments, PSM processing may be performed on spectrally orthogonal audio components, such as the hyper-mid components of an audio signal. In other embodiments, PSM processing is performed on an audio channel associated with a sound source (e.g., vocalization, etc.), and the processed channel is then mixed with other audio content (e.g., background music, etc.).
[0019] The following description focuses primarily on upmixing a mono signal to stereo (i.e., two output channels) because the majority of audio presentation equipment is stereo, but it will be understood that the techniques described can be easily generalized to include a greater number of channels. Stereo embodiments may be described in terms of mid / side processing, where the phase difference between the left and right channels results in complementary regions of amplification and attenuation in the mid / side space.
[0020] [Example of a voice processing system] 1 is a block diagram illustrating an audio processing system 100 according to one or more embodiments. System 100 uses PSM processing to spatially shift audio signals and apply other types of spatial (e.g., mid / side) processing. Some embodiments of system 100 have different components than those described herein. Similarly, in some cases, functionality may be distributed among components in a different manner than that described herein.
[0021] System 100 includes a PSM module 102, an L / RM / S converter module 104, a component processor module 106, an M / SL / R converter module 108, and a crosstalk processor module 108. PSM module 102 receives input audio 120 and generates spatially shifted left and right channels 122 and 124. Operation of PSM 102, according to various embodiments, is described in more detail below in conjunction with Figures 6-11.
[0022] The L / RM / S converter module 104 receives a left channel 122 and a right channel 124 and generates a mid component 126 (e.g., a non-spatial component) and a side component 128 (e.g., a spatial component) from the channels 122 and 124. In some embodiments, the mid component 126 is generated based on the addition of the left channel 122 and the right channel 122, and the side component 128 is generated based on the difference of the left channel 122 and the right channel 124. In some embodiments, the transformation of a point in L / R space to a point in M / S space may be expressed according to equation (1) as follows:
[0023]
number
[0024] Meanwhile, the inverse transformation can be expressed according to equation (2) as follows:
[0025]
number
[0026] It is understood that in other embodiments, other L / RM / S type transforms may be used to generate the mid and side components 126 and 128. In some embodiments, the transforms shown in equations (1) and (2) may be used instead of a true orthonormal form in which both the forward and inverse transforms are scaled by √2 due to reduced computational complexity. For ease of explanation, regardless of the specific transform used, the convention of transforming the coordinates of row vectors by multiplication on the right hand side and the notation of the transformed coordinates having the basis as a label on them will be used as shown in equation (3) below. That is,
[0027]
number
[0028] The component processor module 106 processes the mid component 126 to generate a processed mid component 130 and processes the side component 128 to generate a processed side component 314. The processing on each of the components 126 and 128 may include various types of filter extraction, such as spatially sensitive processing (e.g., amplitude or delay based panning, binaural processing, etc.), single or multi-band equalization, single or multi-band dynamics processing (e.g., compression, expansion, limiting, etc.), single or multi-band gain or delay stages, addition of audio effects, or other types of processing. In some embodiments, the component processor module 106 uses the mid component 126 and the side component 128 to perform sub-band spatial processing and / or crosstalk compensation processing. Sub-band spatial processing is processing performed on frequency sub-bands of the mid and side components to spatially enhance the audio signal. Crosstalk compensation processing is processing that adjusts for spectral artifacts caused by crosstalk processing, such as crosstalk correction for speakers or crosstalk simulation for headphones. The various components that may be included in the component processor module 106 are further described with reference to Figures 12A-13.
[0029] The M / SL / R converter module 108 receives the processed mid component 130 and the processed side component 132 and generates a processed left component 134 and a processed right component 136. In some embodiments, the M / SL / R converter module 108 converts the processed mid component 130 and the side component 132 based on the inverse of the conversion performed by the L / RM / S converter module 104, e.g., the processed left component 134 is generated based on the addition of the processed mid component 130 and the processed side component 132, and the processed right component 136 is generated based on the difference of the processed mid component 130 and the processed side component 132. Other M / SL / R conversion types may be used to generate the processed left component 134 and the processed right component 136.
[0030] The crosstalk processor module 110 receives the processed left component 134 and the processed right component 136 and performs crosstalk processing. Crosstalk processing includes, for example, crosstalk simulation or crosstalk cancellation. Crosstalk simulation is processing performed on an audio signal (e.g., output through headphones) to simulate the effect of loudspeakers. Crosstalk cancellation is processing performed on an audio signal (e.g., output through speakers) to reduce crosstalk caused by loudspeakers. The crosstalk processor module 110 outputs a left channel 138 and a right output channel 140. In some embodiments, crosstalk processing (e.g., simulation or cancellation, etc.) may be performed before component processing, such as before converting the left channel 122 and the right channel 124 into mid and side components. Various components that may be included in the crosstalk processor module 110 are further described with reference to FIGS. 15 and 16.
[0031] In some embodiments, the PSM module 100 is incorporated into the component processor module 106. The L / RM / S converter module 104 receives a left channel and a right channel, which may represent an (e.g., stereo) input to the audio processing system 100. The L / RM / S converter module 104 uses the left and right input channels to generate a mid component and a side component. The PSM module 100 of the component processor module 106 processes the mid component and / or the side component as input to generate the left and right channels, as discussed herein for the input audio 102. The component processor module 106 may also perform other types of processing on the mid and side components, and the M / SL / R converter module 108 generates the left and right channels from the processed mid and side components. The left channel generated by the HPSM module 100 is combined with the left channel generated by the M / SL / R converter module 108 to generate the processed left component. The right channel produced by the PSM module 100 is combined with the right channel produced by the M / SL / R converter module 108 to produce a processed right component.
[0032] System 100 provides a left channel 138 to a left speaker 112 and a right channel 140 to a right speaker 114. Speakers 112 and 114 may be components of a smartphone, tablet, smart speaker, laptop, desktop, exercise machine, etc. Speakers 112 and 114 may be part of an appliance that includes system 100 or may be separate from system 100 such that they are connected to system 100 via a network. The network may include wired and / or wireless connections. The network may include a local area network, a wide area network (including, for example, the Internet), or a combination thereof.
[0033] 2 is a block diagram illustrating a computing system environment 200 according to some embodiments. Computing system 200 may include audio system 202, which may include one or more computing devices (e.g., servers, etc.) connected to user equipment 210a and 210b via network 208. Audio system 202 provides audio content to user equipment 210a and 210b (also individually referred to as user equipment 210) via network 208. Network 208 facilitates communication between system 202 and user equipment 210. Network 106 may include various types of networks, including the Internet.
[0034] The audio system 202 includes one or more processors 204 and a computer-readable medium 206. The one or more processors 204 execute program modules that cause the one or more processors 204 to perform functions, such as generating multiple output channels from a mono channel. The processor 204 may include a central processing unit (CPU), a graphics processing unit (GPU), a controller, a state machine, other types of processing circuitry, or a combination of one or more of these. The processor 204 may further include local memory for storing program modules, operating system data, among other things.
[0035] The computer-readable medium 206 is a non-transitory storage medium that stores program code for the PSM module 102, the component processor module 106, the crosstalk processor module 110, the L / R converter module 104 and the M / S converter module 108, and the channel summation module 212. The PSM module 102 generates multiple output channels from the mono channel, which may be further processed using the component processor module 106, the crosstalk processor module 110, and / or the L / R converter module 104 and the M / S converter module 108. The system 202 provides the multiple output channels to a user equipment 210a, which includes multiple speakers 214 for rendering each of the output channels.
[0036] The channel summation module 212 generates a mono output channel by adding together multiple output channels generated by the PSM module 102 and / or other modules. The system 202 provides the mono output channel to the user equipment 210b, which includes a single speaker 216 for rendering the mono output channel. In some embodiments, the channel summation module 212 is located in the user equipment 210b. The audio system 202 provides multiple output channels to the user equipment 210b, which converts the multiple channels to a mono output channel for the speaker 216. The user equipment 210 presents audio content to a user. The user equipment 210 can be a user's computing device, such as a music player, a smart speaker, a smartphone, a wearable device, a tablet, a laptop, or a desktop.
[0037] [Mid / Side space coloration] In some embodiments, spatial suggestions are encoded into the audio signal by creating coloration effects in the mid / side spaces while avoiding coloration effects in the left / right spaces. In some embodiments, this is accomplished by applying all-pass filters with specifically selected characteristics to the left / right spaces to produce the target coloration in the mid / side spaces. For example, in a two-channel system, the relationship between the left / right phase angle and the mid / side gain may be expressed using the following equation (4):
[0038]
number
[0039] Where:
[0040]
number
[0041] is a two-dimensional row vector composed of the mid and side target gain factors, each in decibels, at a particular frequency ω,
[0042]
number
[0043] is the objective function for the phase relationship between the left and right channels.
[0044]
number
[0045] Solving equation (4) for σ gives the frequency-dependent phase differential required to apply to the left / right spaces according to equations (5) and (6) below:
[0046]
number
[0047] Note that if the constraint that the system is colorless in left and right space is applied, then only the transfer functions of the mid or side components can be specified. Therefore, the system of equations (5) and (6) becomes over-determined, and only one of the above equations can be solved without breaking the required symmetry. In some embodiments, control over either the mid or side components is obtained by choosing a particular equation. If the constraint that the system is colorless in left and right space were removed, further degrees of freedom could be achieved. In systems with more than two channels, various techniques could be used instead of mid and side, such as pair-wise transformations or hierarchical sum and difference transformations.
[0048] [Example implementation of an all-pass filter for encoding elevation cues] In some embodiments, spatial perceptual cues can be encoded into audio signals by embedding frequency-dependent amplitude cues (i.e., coloration) into mid / side spaces while constraining the left and right signals to be colorless. For example, elevation cues (such as spatial perceptual cues that lie along the sagittal plane) can be encoded using this framework because left and right cues with respect to elevation are theoretically symmetric in coloration.
[0049] In some embodiments, a notable feature in the Head-Related Transfer Function (HRTF)-based elevation suggestion is a notch that starts at about 8 kHz and rises monotonically as a function of elevation angle to approximately 16 kHz, which is used to derive the appropriate coloration of the mid channel for encoding the elevation angle. Using this mid-encoded cue, a corresponding frequency-dependent phase shift can be derived, which can be further used to derive a function implemented via a filter network (e.g., PSM module 100, etc.) such as the one described below. In some embodiments, the HRTF-based elevation suggestion can be characterized as a notch that starts at about 8 kHz and rises monotonically as a function of elevation angle to approximately 12 kHz.
[0050] For ease of explanation, the following example filter framework is described in relation to encoding the same perceptual suggestion according to some embodiments where the target elevation angle is 60 degrees (e.g., spatially shifting the audio content 60 degrees above horizontal). However, it is understood that in other embodiments, similar techniques are used to encode perceptual suggestions having different elevation angles. FIG. 3 illustrates a graph showing the HRTF of a sample extraction measured at an elevation angle of 60 degrees according to some embodiments. FIG. 4 illustrates a graph showing an example of a perceptual suggestion characterized by a target amplitude function corresponding to a narrow region of infinite attenuation at approximately 11 kHz according to some embodiments. Such suggestions may be used to create an elevation angle perception in most people across a wide variety of presentation scenarios. The graph in FIG. 4 illustrates a simplified HRTF of a sample extraction, but it is understood that more complex suggestions may also be derived based on the framework described herein.
[0051] [Design using second-order allpass section] In some embodiments, the PSM module 100 is implemented using two independent cascades of second-order all-pass filters and delay elements to achieve the desired phase shift in left / right space to encode perceptual suggestions as described above in connection with FIG. 4. In some embodiments, the second-order sections are implemented as biquad sections whose coefficients are applied to feedback and feedforward taps of up to two delayed samples. As discussed herein, a convention is used to name the feedback coefficients A1 and A2 of one sample and two samples, respectively, and the feedforward coefficients B0, B1, and B2 of zero sample, one sample, and two samples, respectively.
[0052] In some embodiments, the PSM module 100 is implemented using a second-order all-pass filter configured to perform pole and zero cancellation, allowing the magnitude component of the transfer function to remain flat while the phase response is modified. By using all-pass filter sections for both channels in the left and right space, a specific phase shift can be guaranteed across the spectrum. This has the added benefit of allowing for a given phase offset between the left and right sides, resulting in an increased sense of spaciousness in addition to the desired nulls in the mid / side space.
[0053] Table 1 below illustrates an example set of biquad coefficients that may be used in the framework of a second-order all-pass filter with an additional two-sample delay on the right channel, according to some embodiments. The biquad coefficients illustrated in Table 1 may be designed for a sampling rate of 44.1 kHz, but may be used for systems having other sampling rates (e.g., 48 kHz, etc.) as well.
[0054] [Table 1]
[0055] A network of filters with coefficients shown in Table 1 may produce appropriate phase responses in left / right space resulting in a significant null / amplification in mid / side space at 11 kHz. Figure 5 illustrates a frequency plot produced by driving a second-order all-pass filter section with coefficients shown in Table 1 with white noise, according to some embodiments, showing the output frequency response of the sum 502 of multiple channels (mid) and the difference 504 of multiple channels (side).
[0056] In some embodiments, the PSM module 100 implemented using a second-order all-pass filter section may be further extended using a crossover network to eliminate processing for frequency regions that do not require it. The use of a crossover network may increase the flexibility of the embodiment by allowing further processing on perceptually significant cues to eliminate unnecessary auditory data.
[0057] In some embodiments, the PSM module 100 implemented using a second-order all-pass filter section may be implemented using a network of serially chained Hilbert Transforms, as will be described in more detail below.
[0058] [Example of the Hilbert Transform Perceptual Sound Stage Modification (HPSM) module] 6 is a block diagram illustrating a PSM module implemented using the Hilbert transform, according to one or more embodiments. The PSM module 600, also referred to as a Hilbert transform based perceptual sound stage modification (HPSM) module, applies a network of serially concatenated Hilbert transforms to an input audio 602 (which may correspond to the input audio 120 shown in FIG. 1) to perceptually shift the input audio 602.
[0059] Module 600 includes a crossover network module 604, a gain unit 610, a gain unit 612, a Hilbert transformer module 614, a Hilbert transformer module 620, a delay unit 626, a gain unit 628, a delay unit 630, a gain unit 630, a gain unit 632, a summing unit 634, and a summing unit 636. Some embodiments of module 600 have different components than those described herein. Similarly, in some cases, functionality may be distributed among components in a different manner than described herein.
[0060] The crossover network module 604 receives the input audio 602 and generates a low frequency component 606 and a high frequency component 608. The low frequency component includes a subband of the input audio 602 that has a lower frequency than the subband of the high frequency component 608. In some embodiments, the low frequency component 606 includes a first portion of the input audio that includes low frequencies, and the high frequency component 608 includes a remaining portion of the input audio that includes high frequencies.
[0061] As described in more detail below, the high frequency components 608 are processed using a series of Hilbert Transforms, while the low frequency components 606 bypass the series of Hilbert Transforms, after which the low frequency components and the processed high frequency components 608 are recombined. The crossover frequency between the frequency components 606 and the high frequency components 608 may be adjustable. For example, more frequencies may be included in the high frequency components 608 to increase the perceived strength of the spatial shift by the HPSM module 600, while more frequencies may be included in the low frequency components 606 to reduce the perceived strength of the shift. In another example, the crossover frequency is set such that frequencies for a sound source of interest (e.g., speech, etc.) are included in the high frequency components 608.
[0062] The input audio 602 may include a mono channel or may be a mixdown of a stereo signal or other multi-channel signal (e.g., surround sound, ambisonics, etc.). In some embodiments, the input audio 602 is audio content associated with a sound source that is to be incorporated into an audio mix. For example, the input audio 602 may be a vocalization that is processed by the module 600 and the results of the processing are combined with other audio content (e.g., background music, etc.) to generate the audio mix.
[0063] Gain unit 610 applies a gain to the low frequency components 606, and gain unit 612 applies a gain to the high frequency components 608. Gain units 610 and 612 may be used to mutually adjust the overall levels of the low frequency components 606 and the high frequency components 608. In some embodiments, gain unit 610 or gain unit 612 may be omitted from module 600.
[0064] The Hilbert transformer modules 614 and 620 apply a series of Hilbert transforms to the high frequency components 608. The Hilbert transformer module 614 applies a Hilbert transform to the high frequency components 608 to generate left leg components 616 and right leg components 618. The left leg components 616 and right leg components 618 are audio components that are 90 degrees out of phase with respect to each other. In some embodiments, the left leg components 616 and right leg components 618 are out of phase with respect to each other at an angle other than 90 degrees, such as, for example, between 20 degrees and 160 degrees.
[0065] The Hilbert transformer module 620 applies a Hilbert transform to the right leg component 618 generated by the Hilbert transformer module 614 to generate the left leg component 122 and the right leg component 624. The left leg component 622 and the right leg component 624 are audio components that are 90 degrees out of phase with respect to each other. In some embodiments, the Hilbert transformer module 620 generates the right leg component 624 without generating the left leg component 122. In some embodiments, the left leg component 622 and the right leg component 624 are out of phase with each other at an angle other than 90 degrees, such as, for example, between 20 degrees and 160 degrees.
[0066] In some embodiments, each of the Hilbert transformer modules 614 and 620 is implemented in the time domain and includes cascaded all-pass filters and delays, as described in more detail below in connection with Figure 7. In other embodiments, the Hilbert transformer modules 614 and 620 are implemented in the frequency domain.
[0067] The delay unit 626, the gain unit 628, the delay unit 630, and the gain unit 632 provide adjustment controls to manipulate the perceptual results of the processing by the module 600. The delay unit 626 applies a time delay to the left leg component 616 generated by the Hilbert transformer module 614. The gain unit 628 applies a gain to the left leg component 616. In some embodiments, the delay unit 626 or the gain unit 628 may be omitted from the module 600.
[0068] The delay unit 630 applies a time delay to the right leg component 624 generated by the Hilbert transformer module 620. The gain unit 632 applies a gain to the right leg component 624. In some embodiments, the delay unit 630 or the gain unit 632 may be omitted from the module 600.
[0069] A summing unit 634 combines the low frequency component 606 with a left leg component 616 to generate a left channel 642. The left leg component 616 is an output from the first Hilbert transformer module 614 in the series. The left leg component 616 may include a delay applied by a delay unit 626 and a gain applied by a gain unit 628.
[0070] A summing unit 636 combines the low frequency component 606 with a right leg component 624 to generate a right channel 644. The right leg component 624 is an output from a second Hilbert transformer module 620 in the series. The right leg component 624 may include a delay applied by a delay unit 626 and a gain applied by a gain unit 628.
[0071] 7 is a block diagram illustrating a Hilbert transformer module 700 according to one or more embodiments. The Hilbert transformer module 700 is an example of the Hilbert transformer module 614 or the Hilbert transformer module 620. The Hilbert transformer module 700 receives an input component 702 and uses the input component 702 to generate a left leg component 712 and a right leg component 724. Some embodiments of the Hilbert transformer module 700 have different components than those described herein. Similarly, in some cases, functionality may be distributed among components in a different manner than described herein.
[0072] The Hilbert transformer module 700 includes an all-pass filter cascade module 740 for generating a left leg component 712, a delay unit 714, and an all-pass filter cascade module 742 for generating a right leg component 724. The all-pass filter cascade module 714 includes a series of all-pass filters 704, 706, 708, and 710. The delay unit 714 applies a time delay to the input component 702. The all-pass filter cascade module 742 includes a series of all-pass filters 716, 718, 720, and 722. Each of the all-pass filters 704-710 and 716-722 passes frequencies equal in gain while varying the phase relationship between different frequencies. In some embodiments, each all-pass filter 704-710 and 716-722 is a biquad filter as defined by equation (7).
[0073]
number
[0074] where z is a complex variable and a0, a1, a2, b0, b1, and b2 are digital filter coefficients. Different biquad filters may contain different coefficients to apply different phase changes.
[0075] The all-pass filter cascade modules 740 and 742 may each include a different number of all-pass filters. The Hilbert transformer module 700 is an eighth order filter with eight all-pass filters, four for each of the left leg component 712 and the right leg component 724. In other embodiments, the Hilbert transformer module 700 is an eighth order filter (e.g., four all-pass filters for each of the all-pass filter cascade modules 740 and 742) or a sixth order filter (e.g., three all-pass filters for each of the all-pass filter cascade modules 740 and 742).
[0076] As discussed above in connection with FIG. 6, the module 600 includes a series of Hilbert transformer modules 614 and 620. Using a Hilbert transformer module 700 for each of the Hilbert transformer modules 614 and 620, the left leg component 616 is generated by one all-pass filter cascade module 740 applied to the high frequency component 608. The right leg component 624 is generated by two passes through the Hilbert transformer module 700 by two delay units 714 and two all-pass filter cascade modules 742. In some embodiments, the Hilbert transformer modules 614 and 620 may be different. For example, the Hilbert transformer modules 614 and 620 may include filters of different orders, such as an 8th order filter for one of the Hilbert transformer modules and a 6th order filter for another one of the Hilbert transformer modules.
[0077] Using the Hilbert transformer module 700 for the Hilbert transformer modules 614 and 620, the right leg component 624 includes a phase and delay relationship to the right leg component 618, which is created by the all-pass filter and delay of the Hilbert transformer module 620. The right leg component 624 also includes a phase and delay relationship to the high frequency component 608, which is created by the all-pass filter and delay in the Hilbert transformer modules 614 and 620. In some embodiments, the Hilbert transformer module 620 uses the left leg component 616 rather than the right leg component 618 to generate the left leg component 622 and the right leg component 624. This results in the right leg component 624 having a phase and delay relationship to the high frequency component 608, which is created by the all-pass filter (e.g., and no delay) of the Hilbert transformer 614 and the delay and all-pass filter of the Hilbert transformer module 620.
[0078] FIG. 8 illustrates a frequency plot generated by driving an HPSM module (as described in FIG. 6) with white noise in accordance with some embodiments, showing the output frequency response of the sum 802 of multiple channels (mid) and the difference 804 of multiple channels (side).
[0079] As shown in Figure 8, this filter does produce the desired perceptual suggestion in the region around 11 kHz, while also imparting additional mid and side coloration at lower frequencies. In some embodiments, this can be corrected by applying a crossover network (such as the crossover network module 604 shown in Figure 6) to the input audio, so that the HPSM module processes only audio data within the desired frequency range (e.g., high frequency content), or by directly removing the pole / zero pairs that correspond to that region of the spectral transform.
[0080] [Design using First Order Non-orthogonal Rotation-Based Decorrelation (FNORD)] In some embodiments, a similar perceptual effect may be achieved using a first-order non-orthogonal rotation-based decorrelation (FNORD) filter network. FIG. 9 is a block diagram illustrating a PSM module 900 implemented using an FNORD filter network, according to some embodiments. The PSM module 900 may correspond to the PSM module 102 illustrated in FIG. 1, but provides for decorrelation of a mono channel into multiple channels, and includes a magnitude response module 902, an all-pass filter configuration module 904, and an all-pass filter module 906. The PSM module 900 processes a mono input channel x(t) 912 to produce a channel y(t) that is provided to a speaker 910a. a (t), and channel y (t) provided to speaker 910b (which may correspond to left speaker 112 and right speaker 114 illustrated in FIG. 1 ). b9 illustrates the PSM module 900 as including a magnitude response module 902 and a filter configuration module 904 in addition to an all-pass filter module 906, in some embodiments, the PSM module 900 may include an all-pass filter module 906 with the magnitude response module 902 and / or the filter configuration module 904 implemented separately from the PSM module 900.
[0081] The amplitude response module 902 determines a target amplitude response that defines one or more spatial implications to be encoded into the output channel y(t) (e.g., the mid and side components of the output channel y(t)). The target amplitude response is defined by a relationship between the amplitude and frequency values of the channel (e.g., the mid and side components of the channel), such as amplitude as a function of frequency. In some embodiments, the target amplitude response defines one of the implications or spatial implications on the channel, which may include a target wideband attenuation, a target subband attenuation, a critical point, a filter characteristic, or a sound stage position. The amplitude response module 902 may receive data 914 and the mono channel x(t) 912 and use these inputs to determine the target amplitude response. The data 914 may include information such as characteristics of the spatial implications to be encoded, characteristics of the presentation device (e.g., one or more speakers), the expected content of the audio data, or the perceptual capabilities of the listener in the scene. In some embodiments, mono channel x(t) 912 may correspond to the audio input 120 illustrated in Figure 1, or a portion of the audio input (e.g., a high frequency component of the input audio, such as the high frequency component 608 of the input audio 602 illustrated in Figure 6). In embodiments in which mono channel x(t) 912 corresponds to a portion of the audio input, output channel y(t) may be combined with a channel corresponding to the remaining portion of the audio input (e.g., having a low frequency component, as illustrated in Figure 1) to generate a combined output channel.
[0082] The target broadband attenuation is a specification of the attenuation across all frequencies. The target subband attenuation is a specification of the amplitude for a frequency range defined by a subband. The target amplitude response may include one or more target subband attenuation values, each for a different subband.
[0083] A critical point is a specification of the curvature of the target magnitude response of a filter, described as the frequency value at which the gain for one of the output channels (e.g., the side component of an output channel) is a predefined value, such as -3 dB or -∞ dB. The placement of this point can have an overall effect on the curvature of the target magnitude response. One example of a critical point corresponds to the frequency at which the target magnitude response is -∞ dB. This critical point is a null point, since the behavior of the target magnitude response is to null signals at frequencies close to this point. Another example of a critical point corresponds to the frequency at which the target magnitude response is -3 dB. This critical point is a crossover point, since the behavior of the target magnitude response for the sum channel and the difference channel (e.g., the mid and side components of the channel) cross at this point.
[0084] Filter characteristics are parameters that specify how to filter out the mid and side components of a channel. Examples of filter characteristics include high-pass, low-pass, band-pass, or band-reject characteristics. Filter characteristics describe the shape of the resulting sum as if it were the result of equalization filtering. Equalization filtering can be described in terms of which frequencies can pass the filter or which frequencies are rejected. Thus, a low-pass characteristic allows frequencies below an inflection point to pass and attenuates frequencies above the inflection point. A high-pass characteristic does the opposite by passing frequencies above the inflection point and attenuating frequencies below the inflection point. A band-pass characteristic allows frequencies in a band around the inflection point to pass and attenuates other frequencies. A band-reject characteristic rejects frequencies in a band around the inflection point and allows other frequencies to pass.
[0085] The target magnitude response may define multiple spatial implications to be coded into the output channel y(t). For example, the target magnitude response may specify spatial implications specified by critical points and filter characteristics of the mid or side components of an all-pass filter. In another example, the target magnitude response may specify spatial implications specified by target broadband attenuation, critical points, and filter characteristics. Although described as being independent specifications, the specifications may be interdependent for most regions of the parameter space. This result may be due to the system being nonlinear in phase. To address this, additional higher level descriptors of the target magnitude response may be devised that are nonlinear functions of the target magnitude response parameters.
[0086] The filter configuration module 904 determines the characteristics of a single-input multiple-output all-pass filter based on the target magnitude response received from the magnitude response module 902. In particular, the filter configuration module determines a transfer function of the all-pass filter based on the target magnitude response and determines coefficients of the all-pass filter based on the transfer function. The all-pass filter is a decorrelation filter that encodes the spatial implications described in terms of the target magnitude response and is applied to the mono input channel x(t) to generate the output channel y a (t) and y b Generate (t).
[0087] The all-pass filter may include various configurations and parameters based on the spatial implication and / or constraints defined by the target amplitude response. The filter with the target amplitude response of the encoded spatial implication may be colorless, for example, preserving the spectral content (e.g., overall) of the individual output channels (e.g., left / right output channels). The filter is then used to encode the elevation implication by embedding coloration in the mid / side space in the form of a frequency-dependent amplitude implication while preserving the spectral content of the left and right signals. Because the filter is colorless, the monaural content can be placed at a specific location in the sound stage (e.g., as specified by the target elevation angle), and the spatial placement of the sound is decoupled from its overall coloration.
[0088] 10A and 10B are block diagrams illustrating an example of a PSM module based on a first-order non-orthogonal rotation-based decorrelation (FNORD) technique, according to some embodiments. FIG 10A shows a detailed view of a PSM module 900, according to some embodiments, while FIG 10B provides a more detailed view of a wideband phase rotator 1004 within an all-pass filter module 906 of the PSM module 900, according to some embodiments.
[0089] As in FIG. 10A, the all-pass filter module 906 receives a mono input audio signal x(t) 912, a rotation control parameter θ bf 1048, and the first coefficient β bf 1050 receives the input speech signal x(t) 912 and the rotation control parameter θ bf 1048 is utilized by a broadband phase rotator 1004, which has a rotation control parameter θ bf1048 to generate a left wideband rotated component 1020 and a right wideband rotated component 1022. The left wideband rotated component 1020 is then provided to a narrow-band phase rotator 1024 for further processing, while the right wideband rotated component 1022 is provided to an output channel y b The narrowband phase rotator 1024 receives the left wideband rotation component 1020 from the wideband phase rotator 1004 and outputs the first-order coefficient β bf 1050 to generate a narrowband rotated component 1028, which is fed to an output channel y (e.g., the left output channel) of the PSM module 900. a (t) is provided.
[0090] According to some embodiments, the control data 914 for configuring the magnitude response module 902 includes a critical point f c 1038, filter characteristic θ bf 1036, and the sound stage position Γ 1040. This data is provided to the PSM module 900 via the magnitude response module 902, which determines the critical points coc 1044 (in radians), the filter characteristic θ bf 1042, and a secondary term φ 1046. In some embodiments, the magnitude response module 902 determines a parameter of the control data 914 (e.g., a critical point f c 1038, filter characteristic θ bf 1036, and / or the sound stage position Γ 1040. In some embodiments, such as that shown in FIG. bf 1042 is a filter characteristic θ bf1036. These mid representations 1042, 1044, and 1046 are provided to the filter construction module 904, which then constructs at least the primary coefficients β bf 1050 and rotation control parameters 1048. bf 1050 is provided to the all-pass filter module 906 via a first order all-pass filter 1026. In some embodiments, the rotation control parameter θ bf 1048 is the filter characteristic θ bf 1036 and 1042, while in other embodiments this parameter may be scaled for convenience. For example, in some embodiments the filter characteristic is associated with a parameter range (e.g., 0 to 0.5, etc.) that has a meaningful center point, and the rotation control parameter is scaled relative to the filter characteristic to change the parameter range, e.g., to 0 to 1. In some embodiments the filter characteristic is scaled linearly (e.g., to maintain increased resolution at the extremes compared to the center point), while in other embodiments a non-linear mapping may be used (e.g., to increase numerical resolution relative to the center point). In the following equation, the rotation control parameter θ bf is treated as unscaled, it being understood that the same principles may be applied if the rotation control parameters are scaled. bf 1048 is provided to the all-pass filter module 906 via a wideband phase rotator 1004 .
[0091] 10B details an example implementation of the wideband phase rotator 1004, according to some embodiments. The wideband phase rotator 1004 receives the mono input audio signal x(t) 912 and a rotation control parameter θ bf1048. The input speech signal x(t) 912 is first processed by a Hilbert transformer module 1006 to generate a left leg component 1008 and a right leg component 1010. The Hilbert transformer module 1006 may be implemented using the configuration shown in FIG. 7, although it will be understood that other implementations of the Hilbert transformer module 1006 may be used in other embodiments. The left leg component 1008 and the right leg component 1010 are provided to a 2D orthogonal rotation module 1012. The left leg component 1008 is also provided to the output of the wideband phase rotator 1004 as a right wideband rotation component 1022. Because the wideband phase rotator 1004 is configured to rotate the left and right leg signals relative to each other, one way to achieve this in some embodiments is to hold the left leg component 1008 constant as the right wideband rotation component 1022 and rotate the left and right leg components to form the left wideband rotation component 1020.
[0092] In addition to the left leg component 1008 and the right leg component 1010, the 2D orthogonal rotation module 1012 receives a rotation control parameter θ from the filter configuration module 904 according to some embodiments, as shown in FIG. bf 1048. The 2D orthogonal rotation module 1012 uses this data to generate a left rotated component 1014 and a right rotated component 1016. The projection module 1018 then receives the left rotated component 1014 and the right rotated component 1016, which are combined (e.g., added) to form a left wideband rotated component 1020. As shown in FIG. 10A, the wideband phase rotator 1004 rotates the left output channel y a The narrowband rotation component 1028 is fed to the right output channel y b10A and 10B , the narrowband phase rotator 1028 outputs the left wideband rotation component 1020 to the narrowband phase rotator 1024 to generate the right wideband rotation component 1022 as (t) (which either bypasses the narrowband phase rotator 1024 or passes through it unchanged). In other embodiments, the narrowband rotation component 1028 and the left leg component 1008 (which in the embodiment shown in FIGS. 10A and 10B serves as the right wideband rotation component 1022) are instead fed to the right output channel y b (t) and the left output channel y a (t) are each mapped to
[0093] In some embodiments, the PSM module 900 may be formally described by the following equation (8):
[0094]
number
[0095] In some embodiments, this single-input multiple-output all-pass filter is composed of several parts, each of which will be described in turn. According to some embodiments, these components include A f , A b , and H2.
[0096] According to some embodiments, A f may correspond to narrowband phase rotator 1024 in FIG. f is a first-order all-pass filter with one channel output assuming the form of equation (9).
[0097]
number
[0098] Here, β f are the coefficients of the filter, which range from −1 to +1. The second output of the filter may simply pass the input through unchanged. Therefore, according to some embodiments, filter A fThe implementation of may be defined by equation (10):
[0099]
number
[0100] A f The transfer function of is the differential phase shift from one output to the other.
[0101]
number
[0102] This differential phase shift is a function of the center frequency (radian frequency) ω, as defined by equation (11).
[0103]
number
[0104] where the target magnitude response is determined by changing θ in either (5) or (6) depending on whether the response should be placed in the mid (5) or side (6).
[0105]
number
[0106] As a critical point for adjustment, the frequency f at which the total gain αf=3 dB can be used c is defined by the following formula:
[0107]
number
[0108] and
[0109]
number
[0110] By normalizing the target amplitude response to 0 dB, this critical point is determined by the parameter f c This may be the -3 dB point. f The outputs of are subscripted to identify only the output of the first channel that is used, according to some embodiments.
[0111] In equation (8), A b A is a single-input multiple-output all-pass filter, which may correspond to the wideband phase rotator 1004 in FIG. b can be formally defined as in equation (14).
[0112]
number
[0113] where H2(x(t)) is the discrete form of the filter, implemented using a pair of orthogonal all-pass filters, defined using a continuous-time prototype according to equation (15), i.e.
[0114]
number
[0115] In some embodiments, an all-pass filter
[0116]
number
[0117] constrains a 90 degree phase relationship between the two output signals, and a unity magnitude relationship between the input signal and both output signals, but does not necessarily guarantee a particular phase relationship between the input (mono) signal and either of the two (stereo) output signals.
[0118]
number
[0119] The discrete form of
[0120]
number
[0121] and is defined by its operation on the mono signal x(t). The result is a two-dimensional vector, as defined by equation (16):
[0122]
number
[0123] Discrete single-input multiple-output all-pass filters
[0124]
number
[0125] corresponds to the Hilbert transformer module 1006 in Figure 10B, and may also correspond to the Hilbert transformer module 700 in Figure 7, according to some embodiments. In equation (14), θ determines the rotation angle of the first output relative to the second output of Ab, according to some embodiments.
[0126] Finally, the parameter A supplied to the complete system in Eq. (8) bfAccording to some embodiments, the rotation control parameter θ bf 1048 and the first coefficient β bf β bf and θ bf In some embodiments, β bf is the center radian frequency ω c can be determined from
[0127]
number
[0128] Here, ω c is calculated using equation (12) to find the desired center frequency f c In FIG. 10A, ω c is the critical point ω c Corresponding to 1044, f c is the critical point f c 1038, the operations of equation (17) are partially implemented within the filter construction module 904, resulting in the first-order coefficients β bf In some embodiments, the quadratic term φ 1046 is calculated via equation (18) as bf and the Boolean sound stage position parameter Γ, i.e.
[0129]
number
[0130] This quadratic term φ 1046 is provided to the filter configuration module 904 by the magnitude response module 902 in FIG. 10A.
[0131] In some embodiments, the high-level parameter f c , θ bf , and Γ may be sufficient to intuitively and conveniently tune this system. According to such an embodiment, the center frequency fc determines the inflection point in Hz where the target magnitude response asymptotically approaches -∞ dB. The parameter θ bf Therefore, the inflection point f c 0<θ bf For the case where the characteristic is low pass with a null at fc, the spectral slope of the target amplitude function is θ bf As θ increases, it smoothly interpolates from favorable low frequencies to flat. bf If <1 / 2, then θ bf As increases, the characteristic is c It smoothly interpolates from flat to high pass with a null at θ bf At the point f = 1 / 4, the target amplitude function is purely band-reject, f c The parameter Γ is null at f c and θ bf is a Boolean value that places the target magnitude function, determined by, in either the mid channel (i.e., L+R) or the side channel (i.e., LR). Due to the global constraints on both outputs to the filter network, the effect of Γ is to switch between complementary target magnitude responses.
[0132] In some embodiments, to achieve an amplitude response for a 60 degree vertical angle, the FNORD filter network described above is configured with a parameter f c =11kHz, θ bf = 0.13, and Γ = 1. FIG. 11 illustrates a frequency response graph showing the output frequency response of an FNORD filter network configured to achieve an amplitude response for a 60 degree vertical angle, according to some embodiments. FIG. 11 illustrates the output frequency response at a mid component 1110 and a side component 1120, where the FNORD filter network is driven by white noise. In some embodiments, the filter parameters f c , θ bf , and / or Γ are selected based on an analysis of HRTF-based elevation angle suggestions at the desired angle.
[0133] In some embodiments, the PSM module 900 uses a frequency domain specification for the all-pass filters. For example, in some cases, more complex spatial cues may be required, e.g., sampled from anthropometric datasets. Within certain limitations, the techniques described above may be used to embed arbitrary cues into the phase differences of the audio stream based on a magnitude frequency domain representation of the cues. For example, the filter configuration module 904 may derive K phase angles from a vectorized target magnitude response of K narrow-band attenuation coefficients in the mid or side using an equation in the form of equation (5) or (6).
[0134]
number
[0135] A vectorized transfer function of
[0136]
number
[0137] The phase angle vector θ generates a Finite Impulse Response filter defined by equation (19):
[0138]
number
[0139] Here, the DFT -1 is the inverse discrete Fourier transform (idft) and
[0140]
number
[0141] Then, a vector of 2(K-1) IFR filter coefficients Bn(θ) is applied to x(t) as defined by equation (20), i.e.
[0142]
number
[0143] Here,
[0144]
number
[0145] represents a convolution operation.
[0146] To replicate the effect from the previous example and achieve a target amplitude response corresponding to a 60 degree height cue, the observed HRIR
[0147]
number
[0148] is sampled and applied to a DFT of length 2(K-1), resulting in
[0149]
number
[0150] , which can be used to generate the target magnitude response vector
[0151]
number
[0152] That is,
[0153]
number
[0154] Where:
[0155]
number
[0156] and
[0157]
number
[0158] are operations that return the real and imaginary components of a complex number, respectively, and all operations are applied vector component wise. This target magnitude response, inserted into either the mid or side, is applied to one of equations (5) or (6) to return a vector of K phase angles,
[0159]
number
[0160] can be determined, from which an FIR filter B can be derived. This filter is then inserted into equation (19) to derive a single-input multiple-output all-pass filter.
[0161] Although equations (19) and (20) provide an effective means for constraining the target magnitude response, their implementation often relies on relatively high-order FR filters resulting from an inverse DFT operation, which may not be suitable for resource-constrained systems. In such cases, a lower-order infinite impulse response (IIR) implementation may be used, as described in connection with equation (8).
[0162] An all-pass filter module 906, as configured by the filter configuration module 904, applies an all-pass filter to the mono channel x(t) to generate an output channel y a (t) and y b The application of the all-pass filter to the channel x(t) may be performed as defined by equations (8), (20), or as depicted in FIG. 9 or FIG. 10A. The all-pass filter module 906 applies the all-pass filter to the channel y(t). a (t) to speaker 910a, channel y b Each output channel is provided to a respective speaker, such as speaker 910b, and so on. Although not shown in FIG. 9, output channel y a (t) and y b It will be understood that (t) may be provided to speakers 910a and 910b via one or more intervening components (e.g., as shown in FIG. 1, component processor module 106, crosstalk processor module 110, and / or L / RM / S converter module 104 and M / SL / R converter module 108, etc.).
[0163] [Hyper Mid Processing] In some embodiments, PSM processing may be performed on a target portion of a received audio signal, such as a mid component of the audio signal or a hyper-mid component of the audio signal. FIG. 12 is a block diagram illustrating an audio processing system 1200 according to one or more embodiments. The system 1200 generates a hyper-mid component to isolate a target portion of the audio signal (e.g., voicing) and performs PSM processing on the hyper-mid component to spatially shift the target portion. Some embodiments of the system 1200 have different components than those described herein. Similarly, in some cases, functionality may be distributed among components in a different manner than that described herein.
[0164] The system 1200 includes an L / RM / S converter module 1206 , a quadrature component generator module 1212 , a quadrature component processor module 1214 that includes the PSM module 102 , and a crosstalk processor module 1224 .
[0165] The L / RM / S converter module 1206 receives the left channel 1202 and the right channel 1204 and generates a mid component 1208 and a side component 1210 from the channels 1202 and 1204. The description regarding the L / RM / S converter module 104 may be applicable to the L / RM / S converter module 1206.
[0166] The quadrature component generator module 1212 processes the mid component 1208 and the side component 1210 to generate at least one of a hyper mid component M1, a hyper side component S1, a residual mid component M2, and a residual side component S2. The hyper mid component M1 is the spectral energy of the mid component 1208 with the spectral energy of the side component 1210 removed. The hyper side component S1 is the spectral energy of the mid component 1208 with the spectral energy of the side component 1210 removed. The residual mid component M2 is the spectral energy of the hyper mid component M1 with the spectral energy of the mid component 1208 removed. The mid component M2 is the spectral energy of the hyper mid component M1 with the spectral energy of the mid component 1208 removed. The residual side component S2 is the spectral energy of the hyper mid component 1210 with the spectral energy of the hyper side component S1 removed. The system 1200 generates a left channel 1242 and a right output channel 1244 by processing at least one of the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and the residual side component S2. The quadrature component generator module 1212 is further described with reference to Figures 13A, 13B, and 13C.
[0167] The quadrature component processor module 1214 processes one or more of the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and / or the residual side component S2, converting the processed components into a processed left component 1220 and a processed right component 1222. The description of the component processor module 106 may be applicable to the quadrature component processor module 1214, except that the processing is performed on the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and / or the residual side, rather than on the mid and side components. For example, the processing on the components M1, M2, S1, and S2 may include various types of spatially sensitive processing (e.g., amplitude or delay based panning, binaural processing, etc.), single or multi-band equalization, single or multi-band dynamics processing (e.g., compression, expansion, limiting, etc.), single or multi-band gain or delay stages, addition of audio effects, or other types of processing. In some embodiments, the quadrature component processor module 1214 performs sub-band spatial processing and / or crosstalk compensation processing using the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and / or the residual side component S2. The quadrature component processor module 1214 may further include an L / RM / S converter for converting the components M1, S2, S1, and S2 into a processed left component 1220 and a processed right component 1222.
[0168] The quadrature component processor module 1214 further includes a PSM module 102 that may operate on one or more of the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and / or the residual side component S2. For example, the PSM module 102 may receive the hyper-mid component M1 as an input and generate spatially shifted left and right channels. The hyper-mid component M1 may include, for example, an isolated portion of the audio signal representing voicing and therefore may be selected for HPSM processing. The left channel generated by the PSM module 102 is used to generate the processed left component 1020 and the right channel generated by the PSM module 102 is used to generate the processed right component 1222. The quadrature component processor module 1214 is further described with reference to FIG. 12.
[0169] The crosstalk processor module 1224 receives the processed left component 1220 and the processed right component 1222 and performs crosstalk processing thereon. The crosstalk processor module 1224 outputs a left channel 1242 and a right channel 1244. The description of the crosstalk processor module 1224 may be applicable to the crosstalk processor module 1224. In some embodiments, crosstalk processing (e.g., simulation or cancellation, etc.) may be performed before quadrature component processing, such as before conversion of the left channel 1202 and the right channel 1204 into mid and side components. The left channel 1242 may be provided to the left speaker 112 and the right channel 1244 may be provided to the right speaker 114.
[0170] [Example of quadrature component generator] 13A-13C are block diagrams illustrating orthogonal component generator modules 1313, 1323, and 1343, respectively, in accordance with one or more embodiments. Orthogonal component generator modules 1313, 1323, and 1343 are examples of orthogonal component generator module 1212. Some embodiments of modules 1313, 1323, and 1343 have different components than described herein. Similarly, in some cases, functionality may be distributed among components in a different manner than described herein.
[0171] 13A, the quadrature component generator module 1313 includes a subtraction unit 1305, a subtraction unit 1309, a subtraction unit 1315, and a subtraction unit 1319. As described above, the quadrature component generator module 1313 receives the mid component 1208 and the side component 1210, and outputs one or more of a hyper-mid component M1, a hyper-side component S1, a residual mid component M2, and a residual side component S2.
[0172] The subtraction unit 1305 removes the spectral energy of the side component 1210 from the spectral energy of the mid component 1208 to generate the hyper-mid component M1. For example, the subtraction unit 1305 subtracts the amplitude of the side component 1210 in the frequency domain from the amplitude of the mid component 1208 in the frequency domain, leaving only the phase, to generate the hyper-mid component M1. The subtraction in the frequency domain may be performed using a Fourier transform on the time domain signal to generate a signal in the frequency domain, and then subtracting the signal in the frequency domain. In another example, the subtraction in the frequency domain may be performed in other ways, such as using a wavelet transform instead of a Fourier transform. The subtraction unit 1309 removes the spectral energy of the hyper-mid component M1 from the spectral energy of the mid component 1208 to generate the residual mid component M2. For example, subtraction unit 1309 subtracts the amplitude of hyper-mid component M1 in the frequency domain from the mid component 1208 amplitude in the frequency domain, while retaining only the phase, to generate a residual mid component M2. Whereas subtracting side from mid in the time domain results in the right channel of the original signal, the above operation in the frequency domain separates and distinguishes between the portion of the spectral energy of the mid component (called M1 or hyper-mid) that is different from the spectral energy of the mid component and the portion of the spectral energy of the side component (called M2 or residual mid) that is the same as the spectral energy of the side component.
[0173] In some embodiments, if the result of subtracting the spectral energy of the side component 1210 from the spectral energy of the mid component 1006 results in a negative value for the hyper-mid component M1 (e.g., for one or more bins in the frequency domain), additional processing may be used. In some embodiments, if the result of subtracting the spectral energy of the side component 1210 from the spectral energy of the mid component 1208 results in a negative value, the hyper-mid component M1 is clamped to a zero value. In some embodiments, the hyper-mid component M1 is wrapped back to its lowest value by taking the absolute value of the negative value as the value of the hyper-mid component M1. Other types of processing may be used if the result of subtracting the spectral energy of the side component 1210 from the spectral energy of the mid component 1208 results in a negative value for M1. Similar additional processing may be used if the result of the subtraction to generate the hyper-side component S1, the residual side component S2, or the residual mid component M2 is negative, such as clamping at zero, wrapping around to a lowest value, or other processing. Fixing the hypermid component M1 to 0 provides spectral orthogonality between M1 and both side components when the subtraction results in a negative value. Similarly, fixing the hyperside component S1 to 0 provides spectral orthogonality between S1 and both mid components when the subtraction results in a negative value. By creating orthogonality between the hypermid and side components and their corresponding appropriate mid / side components (i.e., side components to hypermid, mid components to hyperside, etc.), the residual mid and side components M2 and S2 introduced contain spectral energy that is not orthogonal to (i.e., in common with) their corresponding appropriate mid / side components. That is, when applying a 0 fix to the hypermid and using the M1 component to derive the residual mid, a hypermid component that has no spectral energy in common with the side component and a residual mid component that has completely common spectral energy with the side component are generated.The same relationship applies to the hyperside and residual side when the hyperside is fixed to 0. When applying frequency domain processing, there is typically a trade-off in resolution between frequency and time information. As the frequency resolution increases (i.e., as the FFT window size and number of frequency bins increase), the time resolution decreases, and vice versa. Since the above-mentioned spectral subtraction occurs on a frequency bin basis, in certain situations, such as when removing voicing energy from the hypermid component, it may be more preferable to have a large FFT window size (e.g., 8192 samples, given a real-valued input signal, resulting in 4096 frequency bins). In other situations, higher time resolution may be required, and therefore lower latency and lower frequency resolution overall (e.g., 512 sample FFT window size, given a real-valued input signal, resulting in 256 frequency bins) are needed. In the latter case, the low frequency resolution of mid and side, when subtracted from each other to derive the hypermid and hyperside components M1 and S1, may generate audible spectral artifacts because the spectral energy of each frequency bin is an average representation of the energy over too wide a frequency range. In this case, taking the absolute value of the difference between mid and side when deriving hypermid M1 or hyperside S1 can contribute to reducing the perceptual artifacts by tolerating deviations from true orthogonality in the components per frequency bin. In addition to or instead of wrapping around zero, a factor may be applied to the subtrahend value to scale it between 0 and 1, which may provide a way to interpolate between perfect orthogonality of the hyper and residual mid / side components at one extreme (i.e., when the value is 1) and hypermid M1 and hyperside S1 identical to their corresponding original mid and side components at the other extreme (i.e., when the value is 0).
[0174] The subtraction unit 1315 removes the spectral energy of the mid component 1208 in the frequency domain from the spectral energy of the side component 1210 in the frequency domain while retaining only the phase, to generate the hyper side component S1. For example, the subtraction unit 1315 subtracts the amplitude of the mid component 1208 in the frequency domain from the amplitude of the side component 1210 in the frequency domain while retaining only the phase, to generate the hyper side component S1. The subtraction unit 1319 removes the spectral energy of the hyper side component S1 from the spectral energy of the side component 1210 to generate the residual side component S2. For example, the subtraction unit 1319 subtracts the amplitude of the hyper side component S1 in the frequency domain from the amplitude of the side component 1210 in the frequency domain while retaining the phase, to generate the residual side component S2.
[0175] 5B, the orthogonal component generator module 1323 is similar to the orthogonal component generator module 1313 in that the orthogonal component generator module 1323 receives the mid component 1006 and the side component 1210 and generates a hyper-mid component M1, a residual mid component M2, a hyper-side component S1, and a residual side component S2. The orthogonal component generator module 1323 differs from the orthogonal generator module 1313 by generating the hyper-mid component M1 and the hyper-side component S1 in the frequency domain and then transforming these components back to the time domain to generate the residual mid component M2 and the residual side component S2. The orthogonal component generator module 1323 includes a forward FFT unit 1320, a band pass unit 1322, a subtraction unit 1324, a hypermid processor 1325, an inverse FFT unit 1326, a time delay unit 1328, a subtraction unit 1330, a forward FFT unit 1332, a band pass unit 1334, a subtraction unit 1336, a hyperside processor 1337, an inverse FFT unit 1340, a time delay unit 1342, and a subtraction unit 1344.
[0176] A forward fast Fourier transform (FFT) unit 1320 applies a forward FFT to the mid component 1208, converting it to the frequency domain. The converted mid component 1208 includes amplitude and phase. A bandpass unit 1322 applies a bandpass filter to the frequency domain mid component 1208, where the bandpass filter specifies frequencies in the hyper-mid component M1. For example, to isolate a typical human vocal range, the bandpass filter may specify frequencies between 300 Hz and 8000 Hz. In another example, to remove audio content associated with a typical human vocal range, the bandpass filter may preserve low frequencies (e.g., generated by a bass guitar or drums) and high frequencies (e.g., generated by cymbals) in the hyper-mid component M1. In other embodiments, the quadrature component generator module 1323 applies various other filters to the frequency domain mid component 1208 in addition to and / or instead of the bandpass filter applied by the bandpass unit 1322. In some embodiments, the quadrature component generator module 1323 does not include a bandpass unit 1322 and does not apply any filter to the frequency domain mid component 1208. In the frequency domain, a subtraction unit 1324 subtracts the side component 1210 from the filtered mid component to generate the hyper-mid component M1. In other embodiments, in addition to and / or instead of subsequent processing applied to the hyper-mid component M1, such as that performed by a quadrature component processor module (such as the quadrature component processor module in FIG. 12), the quadrature component generator module 1323 applies various audio enhancements to the frequency domain hyper-mid component M1. The hyper-mid processor 1325 performs processing on the hyper-mid component M1 in the frequency domain before converting to the time domain. This processing may include sub-band spatial processing and / or crosstalk compensation processing.In some embodiments, the hyper-mid processor 1325 performs processing on the hyper-mid component M1 instead of and / or in addition to the processing that may be performed by the quadrature component processor module 1214. The inverse FFT unit 1326 applies an inverse FFT to the hyper-mid component M1 and transforms the hyper-mid component M1 back to the time domain. The hyper-mid component M1 in the frequency domain includes the amplitude of M1 and the phase of the mid component 1208, which the inverse FFT unit 1326 transforms to the time domain. The time delay unit 1328 applies a time delay to the mid component 1208 such that the mid component 1208 and the hyper-mid component M1 arrive at the subtraction unit 1330 at the same time. The subtraction unit 1330 subtracts the hyper-mid component M1 in the time domain from the time-delayed mid component 1208 in the time domain to generate a residual mid component M2. In this example, the spectral energy of the hyper-mid component M1 is removed from the spectral energy of the mid component 1208 using processing in the time domain.
[0177] The forward FFT unit 1332 applies a forward FFT to the side component 1210, transforming the side component 1210 into the frequency domain. The frequency domain transformed side component 1210 includes amplitude and phase. The band pass unit 1334 applies a band pass filter to the frequency domain side component 1210. The band pass filter specifies the frequency in the hyper side component S1. In other embodiments, the quadrature component generator module 1323 applies various other filters to the frequency domain side component 1210 in addition to and / or instead of the band pass filter. In the frequency domain, the subtraction unit 1336 subtracts the mid component 1208 from the filtered out side component 1210 to generate the hyper side component S1. In other embodiments, the quadrature component generator module 1323 applies various audio enhancements to the frequency domain hyper side component S1 in addition to and / or instead of subsequent processing applied to the hyper side component S1, such as performed by a quadrature component processor (e.g., quadrature component processor module 1214, etc.). The hyperside processor 1337 performs processing on the hyperside component S1 in the frequency domain prior to its transformation to the time domain. This processing may include subband spatial processing and / or crosstalk compensation processing. In some embodiments, the hyperside processor 1337 performs processing on the hyperside component S1 instead of and / or in addition to the processing that may be performed by the quadrature component processor module 1214. The inverse FFT unit 1340 applies an inverse FFT to the hyperside component S1 in the frequency domain to generate the hyperside component S1 in the time domain. The hyperside component S1 in the frequency domain includes the amplitude S1 and phase of the side component 1210, and the inverse FFT unit 1326 transforms the side component 1210 to the time domain. The time delay unit 1342 time delays the side component 1210 such that the side component 1210 arrives at the subtraction unit 1344 at the same time as the hyperside component S1. Subsequently, a subtraction unit 1344 subtracts the hyper side component in the time domain S1 from the time-delayed side component in the time domain 1210 to generate a residual side component S2.In this example, the spectral energy of the hyper side component S1 is removed from the spectral energy of the side component 1210 using processing in the time domain.
[0178] In some embodiments, the hyper-mid processor 1325 and the hyper-side processor 1337 may be omitted if the processing performed by these components is performed by the orthogonal component processor module 1214.
[0179] In FIG. 13C, the orthogonal component generator module 1343 is similar to the orthogonal component generator module 1323 in that the orthogonal component generator module 1343 receives the mid component 1208 and the side component 1210 and generates a hyper-mid component M1, a residual mid component M2, a hyper-side component S1, and a residual side component S2, except that the orthogonal component generator module 1343 generates components M1, M2, S1, and S2 in the frequency domain and then converts these components to the time domain. The orthogonal component generator module 1343 includes a forward FFT unit 1347, a band pass unit 1349, a subtraction unit 1351, a hyper mid processor 1352, a subtraction unit 1353, a residual mid processor 1354, an inverse FFT unit 1355, an inverse FFT unit 1355, an inverse FFT unit 1357, a forward FFT unit 1361, a band pass unit 1363, a subtraction unit 1365, a hyper side processor 1366, a subtraction unit 1367, a residual side processor 1368, an inverse FFT unit 1369, and an inverse FFT unit 1371.
[0180] The forward FFT unit 1347 applies a forward FFT to the mid component 1208 to transform the mid component 1208 to the frequency domain. The mid component 1208 transformed to the frequency domain includes an amplitude and a phase. The forward FFT unit 1361 applies a forward FFT to the side component 1210 to transform the side component 1210 to the frequency domain. The side component 1210 transformed to the frequency domain includes an amplitude and a phase. The bandpass unit 1349 applies a bandpass filter to the frequency domain mid component 1208, which bandpass filter specifies the frequency of the hyper-mid component M1. In some embodiments, the quadrature component generator module 1343 applies various other filters to the frequency domain mid component 1208 in addition to and / or instead of the bandpass filter. The subtraction unit 1351 subtracts the frequency domain side component 1210 from the frequency domain mid component 1208 to generate the hyper-mid component M1 in the frequency domain. The hyper-mid processor 1352 performs processing of the hyper-mid component M1 in the frequency domain before transforming the hyper-mid component M1 to the time domain. In some embodiments, the hyper-mid processor 1352 performs sub-band spatial processing and / or crosstalk compensation processing. In some embodiments, the hyper-mid processor 1352 performs processing on the hyper-mid component M1 instead of and / or in addition to processing that may be performed by the quadrature component processor module 1214. The inverse FFT unit 1357 applies an inverse FFT to the hyper-mid component M1 and transforms it back to the time domain. The hyper-mid component M1 in the frequency domain includes the amplitude M1 and phase of the mid component 1208, which the inverse FFT unit 1357 transforms to the time domain. The subtraction unit 1353 subtracts the hyper-mid component M1 from the mid component 1208 in the frequency domain to generate a residual mid component M2. The residual mid processor 1354 performs processing on the residual mid component M2 in the frequency domain before transforming the residual mid component M2 to the time domain. In some embodiments, the residual mid processor 1354 performs subband spatial processing and / or crosstalk compensation processing on the residual mid component M2.In some embodiments, the residual mid processor 1354 performs processing on the residual mid component M2 instead of and / or in addition to the processing that may be performed by the quadrature component processor module 1214. The inverse FFT unit 1355 applies an inverse FFT to transform the residual mid component M2 into the time domain. The residual mid component M2 in the frequency domain includes the amplitude M2 and phase of the mid component 1208, which the inverse FFT unit 1355 transforms into the time domain.
[0181] The bandpass unit 1363 applies a bandpass filter to the frequency domain side component 1210. The bandpass filter specifies the frequencies in the hyperside component S1. In other embodiments, the quadrature component generator module 1343 applies various other filters to the frequency domain side component 1210 in addition to and / or instead of the bandpass filter. In the frequency domain, the subtraction unit 1365 subtracts the mid component 1208 from the filtered side component 1210 to generate the hyperside component S1. The hyperside processor 1366 performs processing on the hyperside component S1 in the frequency domain prior to conversion to the time domain. In some embodiments, the hyperside processor 1366 performs sub-band spatial processing and / or crosstalk compensation processing on the hyperside component S1. In some embodiments, the hyperside processor 1366 performs processing on the hyperside component S1 instead of and / or in addition to processing that may be performed by the quadrature component processor module 1214. The inverse FFT unit 1371 applies an inverse FFT to transform the hyper side component S1 back to the time domain. The hyper side component S1 in the frequency domain includes the amplitude S1 and phase of the side component 1210, which the inverse FFT unit 1371 transforms to the time domain. The subtraction unit 1367 subtracts the hyper side component S1 from the side component 1210 in the frequency domain to generate a residual side component S2. The residual side processor 1368 performs processing of the residual side component S2 in the frequency domain prior to transformation to the time domain. In some embodiments, the residual side processor 1368 performs subband spatial processing and / or crosstalk compensation processing on the residual side component S2. In some embodiments, the residual side processor 1368 performs processing on the residual side component S2 instead of and / or in addition to processing that may be performed by the quadrature component processor module 1214. An inverse FFT unit 1369 applies an inverse FFT to the residual side component S2, transforming it into the time domain.The residual side component S2 in the frequency domain contains the amplitude S2 and phase of the side component 1210, which the inverse FFT unit 1369 transforms to the time domain.
[0182] In some embodiments, the hyper-mid processor 1352, the hyper-side processor 1366, the residual mid processor 1354, or the residual side processor 1368 may be omitted if the processing performed by these components is performed by the orthogonal component processor module 1214.
[0183] [Example of orthogonal component processor] 14A is a block diagram illustrating an orthogonal component processor module 1417 according to one or more embodiments. The orthogonal component processor module 1417 is an example of the orthogonal component processor module 1412. Some embodiments of the module 1417 have different components than described herein. Similarly, in some cases, functionality may be distributed among components in a different manner than described herein.
[0184] The quadrature component processor module 1417 includes a component processor module 1420 , a PSM module 102 , a summing unit 1422 , an M / SL / R converter module 1424 , a summing unit 1426 , and a summation 1428 .
[0185] The component processor module 1420 performs similar processing as the component processor module 106, except that it uses the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and / or the residual side component S2 instead of the mid component and the side component. For example, the component processor module 1420 performs sub-band spatial processing and / or crosstalk compensation processing on at least one of the hyper-mid component M1, the residual mid component M2, the hyper-side component S1, and the residual side component S2. As a result of the sub-band spatial processing and / or crosstalk compensation by the component processor module 1420, the quadrature component processor module 1417 outputs at least one of the processed M1, the processed M2, the processed S1, and the processed S2. In some embodiments, one or more of the components M1, M2, S1, or S2 may bypass the component processor module 1420.
[0186] In some embodiments, the orthogonal component processor module 1417 performs subband spatial processing and / or crosstalk compensation processing on at least one of the hyper mid component M1, the residual mid component M2, the hyper side component S1, and the residual side component S2 in the frequency domain. The orthogonal component generator module 410 may provide the components M1, M2, S1, or S2 in the frequency domain to the orthogonal component processor module 1417 without performing an inverse FFT. After generating the processed M1, processed M2, and processed side components 1442, the orthogonal component processor module 1417 may perform an inverse FFT to transform these components back to the time domain. In some embodiments, the orthogonal component processor module 1417 performs an inverse FFT on the processed M1, processed M2, processed S1, and processed S1 to generate the processed side components 1446 in the time domain.
[0187] Exemplary components of the quadrature component processor module 1417 are shown in Figures 15 and 16. In some embodiments, the quadrature component processor module 1417 performs both sub-band spatial processing and crosstalk compensation processing. The processing performed by the quadrature component processor module 1417 is not limited to sub-band spatial processing or crosstalk compensation processing. Any type of spatial processing using mid-space / side-space can be performed by the quadrature component processor module 1417, such as by using hyper-mid components instead of mid components or hyper-side components instead of side components. Other types of processing may include application of gain, amplitude or delay-based panning, binaural processing, reverberation, dynamic range processing such as compression and limiting, and other linear or nonlinear audio processing techniques and effects ranging from chorus or flanging to machine learning-based approaches to vocal or instrument style transfer, transformation, or resynthesis, etc.
[0188] The PSM module 102 receives the processed M1 and applies PSM processing to spatially shift the processed M1, resulting in a left channel 1432 and a right channel 1434. Although the PSM module 102 is shown as being applied to the hyper-mid component M1, the PSM module may be applied to one or more of the components M1, M2, S1, or S2. In some embodiments, the components processed by the PSM module 102 bypass processing by the component processor module 1420. For example, the PSM module 102 may process the hyper-mid component M1 rather than the processed M1.
[0189] The summing unit 1422 adds the processed S1 to the processed S2 to generate the processed side component 1442. The M / SL / R transformer module 1424 uses the processed M2 and the processed side component 1442 to generate a processed left component 1444 and a processed right component 1446. In some embodiments, the processed left side component 1444 is generated based on the addition of the processed M2 and the processed side component 1442, and the processed right component 1446 is generated based on the difference of the processed M2 and the processed side component 1442. Other M / SL / R type transforms may be used to generate the processed left component 1444 and the processed right component 1446.
[0190] A summing unit 1426 adds a left channel 1432 from the PSM module 102 to the processed left component 1444 to generate a left channel 1452. A summing unit 1428 adds a right channel 1434 from the PSM module 102 to the processed right component 1446 to generate a right channel 1454. More generally, one or more left channels from the PSM module 102 may be added to a left component (e.g., generated using a hyper / residual component not processed by the PSM module 102) from the M / SL / R converter module 1424 to generate the left channel 1452, and one or more right channels from the PSM module 102 may be added to a right component (e.g., generated using a hyper / residual component not processed by the PSM module 102) from the M / SL / R converter module 1424 to generate the right channel 1454.
[0191] As such, the quadrature component processor module 1417 applies PSM processing to the hyper-mid component M1 of the audio signal as separated by the L / RM / S converter module 1206 and the quadrature component generator module 1212. The PSM-enhanced stereo signal (including the left channel 1432 and the right channel 1434) can then be added to the residual left / right signals (e.g., the processed left component 1444 and the processed right component 1446, which are generated without the hyper-mid components). Other approaches for separating the components of the input signal used for PSM processing may be used in addition to or instead of this example, including machine learning-based source separation.
[0192] In some embodiments, the quadrature component processor module 1417 applies PSM processing to the mid component M of the audio signal instead of the hyper-mid component ML. FIG. 14B illustrates a block diagram showing a quadrature component processor module 1419 according to one or more embodiments. In some embodiments, the quadrature component processor module 1419 in FIG. 14B may be implemented as part of an audio processing system similar to the system 1200 shown in FIG. 12, but without the quadrature component generator module 1212, such that the quadrature component processor module 1419 receives mid and side component signals (e.g., mid component 1208 and side component 1210, etc.) instead of the hyper-mid component, hyper-side component, residual mid component, and residual side component. In some embodiments, the quadrature component processor module 1410 includes a component processor module similar to the component processor module 106 to generate processed mid and processed side components from the received mid and side components (not shown). The PSM module 102 receives the mid component M (or processed mid) and applies PSM processing to spatially shift the received mid signal to generate a PSM processed left channel 1432 and a PSM processed right channel 1434 in the mid signal, which are combined with the side component S (or processed side) by the M / SL / R converter module 1424 to generate the left channel 1452 and the right channel 1454. For example, as shown in FIG. 14B, the M / SL / R converter module 1424 generates the left channel 1452 as a sum of the PSM processed left channel 1432 and the side component S using a summation unit 1460, and generates the right channel 1452 as a sum of the PSM processed right channel 1434 and the side component S using a subtraction unit 1462. In other words, the M / SL / R converter module 1424 serves to mix the side signal (which, in a left-right context, is the inverse of the right component and resides in the subspace defined by the left component) into a PSM-processed stereo signal in left-right space by combining the signal for the left channel and the signal inverse for the right channel.
[0193] [Example of a subband spatial processor] 15 is a block diagram illustrating a subband spatial processor module 1510 according to one or more embodiments. The subband spatial processor module 1510 is an example of a component of a component processor module 106 or 1520. The subband spatial processor module 1510 includes a mid EQ filter 1504(1), a mid EQ filter 1504(2), a mid EQ filter 1504(3), a mid EQ filter 1504(4), a side EQ filter 1506(1), a side EQ filter 1506(2), a side EQ filter 1506(3), and a side EQ filter 1506(4). Some embodiments of the subband spatial processor module 1510 have different components than those described herein. Similarly, in some cases, functionality may be distributed among the components in a different manner than that described herein.
[0194] The subband spatial processor module 1510 receives the non-spatial components Ym and the spatial components Ys and gain adjusts one or more subbands of these components to provide spatial enhancement. If the subband spatial processor module 1510 is part of the component processor module 1420, the non-spatial components Ym may be the hyper-mid components M1 or the residual mid components M2. The spatial components Ys may be the hyper-side components S1 or the residual side components S2. If the subband spatial processor module 1510 is part of the component processor module 106, the non-spatial components Ym may be the mid components 126 and the spatial components Ys may be the side components 128.
[0195] The subband spatial processor module 1510 receives the non-spatial component Ym and applies mid EQ filters 1504(1) through 1504(4) to different subbands of Ym to generate emphasized non-spatial components Em. The subband spatial processor module 1510 also receives the spatial component Ys and applies side EQ filters 1506(1) through 1506(4) to different subbands of Ys to generate emphasized spatial components Es. The subband filters may include various combinations of peaking filters, notch filters, low-pass filters, high-pass filters, low-emphasis filters, high-emphasis filters, band-pass filters, band-stop filters, and / or all-pass filters. The subband filters may also apply gains to the respective subbands. More specifically, the subband spatial processor module 1510 includes a subband filter for each of the n frequency subbands in the non-spatial component Ym and a filter subband for each of the n subbands in the spatial component Ys. For n=4 subbands, for example, the subband spatial processor module 1510 includes a series of subband filters for the non-spatial component Ym, including a mid equalization (EQ) filter 1504(1) for subband(1), a mid equalization (EQ) filter 1504(2) for subband(2), a mid EQ filter 1504(3) for subband(3), and a mid EQ filter 1504(4) for subband(4). Each mid EQ filter 1504 applies a filter extraction to a frequency subband portion of the non-spatial component Ym to generate an emphasized non-spatial component Em.
[0196] The subband spatial processor module 1510 further includes a set of subband filters for the frequency subbands of the spatial component Ys, including a side equalization (EQ) filter 1506(1) for subband(1), a side EQ filter 1506(2) for subband(2), a side EQ filter 1506(3) for subband(3), and a side EQ filter 1506(4) for subband(4). Each side EQ filter 1506 applies a filter extraction to a frequency subband portion of the spatial component Ys to produce an enhanced spatial component Es.
[0197] Each of the n frequency subbands in the non-spatial component Ym and the spatial component Ys may correspond to a frequency range. For example, frequency subband (1) may correspond to 0-300 Hz, frequency subband (2) may correspond to 300-510 Hz, frequency subband (3) may correspond to 510-2700 Hz, and frequency subband (4) may correspond to 2700 Hz to the Nyquist frequency. In some embodiments, each of the n frequency subbands is a set of unified critical bands. The critical bands may be determined using a corpus of audio samples from a wide variety of music genres. The long term average energy ratio of the mid to side components across the 24 Bark scale critical bands is determined from the samples. Contiguous frequency bands with similar long term average ratios are then grouped together to form a set of critical bands. The range of the frequency subbands and the number of frequency subbands may be adjustable.
[0198] In some embodiments, the subband spatial processor module 1510 processes the residual mid component M2 as the non-spatial component Ym, and uses one of the side component, the hyper-side component S1, or the residual side component S2 as the spatial component Ys.
[0199] In some embodiments, the subband spatial processor module 1510 processes one or more of the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and the residual side component S2. The filters applied to the subbands of each of these components may be different. The hyper-mid component M1 and the residual mid component M2 may each be processed as described for the non-spatial component Ym. The hyper-side component S1 and the residual side component S2 may each be processed as described for the spatial component Ys.
[0200] [Example of crosstalk compensation processor] 16 is a block diagram illustrating a crosstalk compensation processor module 1610 according to one or more embodiments. The crosstalk compensation processor module 1610 is an example of a component of a component processor module 106 or 1420. Some embodiments of the crosstalk compensation processor module 1610 have different components than those described herein. Similarly, in some cases, functionality may be distributed among the component components in a different manner than described herein.
[0201] The crosstalk compensation processor module 1610 includes a mid component processor 1620 and a side component processor 1630. The crosstalk compensation processor module 1610 receives the non-spatial component Ym and the spatial component Ys and applies filter extraction to one or more of these components to compensate for spectral defects caused by (e.g., post or previous) crosstalk processing. If the crosstalk compensation processor module 1610 is part of the component processor module 1420, the non-spatial component Ym may be the hyper mid component M1 or the residual mid component M2. The spatial component Ys may be the hyper side component S1 or the residual side component S2. If the crosstalk compensation processor module 1610 is part of the component processor module 106, the non-spatial component Ym may be the mid component 126 and the spatial component Ys may be the side component 128.
[0202] The crosstalk compensation processor module 1610 receives the non-spatial component Ym, and the mid component processor 1620 applies a set of filters to generate an enhanced non-spatial crosstalk compensation component Zm. The crosstalk compensation processor module 1610 also receives the spatial subband component Ys, and applies a set of filters in the side component processor 1630 to generate an enhanced spatial subband component Es. The mid component processor 1620 includes a number of filters 1640, such as m mid filters 1640(a), 1640(b) through 1640(m), where each of the m mid filters 1640 filters the non-spatial component Xm. m Thus, the mid component processor 1620 processes one of the m frequency bands in the non-spatial component X m In some embodiments, the mid filter 1640 generates the mid crosstalk compensation channel Zm by processing the non-spatial components X mThe frequency response plot is constructed using the frequency response plot of the crosstalk signal. Furthermore, by analyzing the frequency response plot, any spectral imperfections, such as peaks or troughs in the frequency response plot that exceed a certain threshold (e.g., 10 dB), that arise as artifacts of the crosstalk processing can be estimated. These artifacts are mainly caused by delayed and possibly inverted contralateral signals being added to the corresponding ipsilateral signals in the crosstalk processing, thereby effectively introducing a comb filter-like frequency response to the final rendering result. A mid crosstalk compensation channel Zm can be generated by the mid component processor 1620 to compensate for the estimated peaks or troughs, with each of the m frequency bands corresponding to a peak or trough. Specifically, based on the specific delays, filter extraction frequencies, and gains applied in the crosstalk processing, the peaks or troughs will shift up or down in the frequency response, causing variable amplification and / or variable attenuation of energy in specific regions of the spectrum. Each of the mid-filters 1640 may be configured to adjust one or more of the peaks and troughs.
[0203] The side component processor 1630 includes a number of filters 1650, such as m side filters 1650(a), 1650(b) through 1650(m). The side component processor 1630 generates a side crosstalk compensation channel Zs by processing the spatial component Xs. In some embodiments, a frequency response plot of the spatial component X with crosstalk processing can be obtained through simulation. By analyzing the frequency response plot, any spectral defects, such as peaks or troughs in the frequency response plot that exceed a certain threshold (e.g., 10 dB), that occur as artifacts of the crosstalk processing can be estimated. The side crosstalk compensation channel Zs can be generated by the side component processor 1630 to compensate for the estimated peaks or troughs. Specifically, based on the specific delays, filter extraction frequencies, and gains applied to the crosstalk processing, the peaks or troughs shift up or down in the frequency response, causing variable amplification and / or variable attenuation of energy in a specific region of the spectrum. Each of the side filters 1650 can be configured to adjust one or more peaks or troughs. In some embodiments, the mid component processor 1620 and the side component processor 1630 may include different numbers of filters.
[0204] In some embodiments, the mid-filter 1640 and the side filter 1650 may include biquad filters having a transfer function defined by equation (7). One way to implement such a filter is with a direct form I topology as defined by equation (22), i.e.,
[0205]
number
[0206] where X is the input vector and Y is the output. Other topologies can be used depending on their maximum word length and saturation operation. The biquad can then be used to implement a second-order filter with real-valued inputs and outputs. To design a discrete-time filter, a continuous-time filter is designed and then transformed to discrete time by a bilinear transform. Furthermore, the resulting shifts in center frequency and bandwidth can be compensated for using frequency warping.
[0207] For example, the peaking filter may have an S-plane transfer function defined by equation (23):
[0208]
number
[0209] where s is a complex variable, A is the peak amplitude, Q is the filter “quality”, and the digital filter coefficients are defined by the following equation (24):
[0210]
number
[0211] where ω is the filter center frequency in radians,
[0212]
number
[0213] Furthermore, the filter quality Q can be defined by equation (25), i.e.,
[0214]
number
[0215] where Δf is the bandwidth and f c is the center frequency. The mid filter 1640 is shown as being in series and the side filter 1650 is shown as being in series. In some embodiments, the mid filter 1640 is a filter that filters the mid component X m and the side filter is applied in parallel to the side component Xs.
[0216] In some embodiments, the crosstalk compensation processor module 1610 processes each of the hyper-mid component M1, the hyper-side component S1, the residual mid component M2, and the residual side component S2. The filters applied to each of these components may be different.
[0217] [Example of a crosstalk processor] 17 is a block diagram illustrating a crosstalk simulation processor module 1700 according to one or more embodiments. The crosstalk simulation processor module 1700 is an example of the crosstalk simulation processor module 110 or the crosstalk processor module 1224. Some embodiments of the crosstalk simulation processor module 1700 have different components than those described herein. Similarly, in some cases, functionality may be distributed among the components in a different manner than described herein.
[0218] The crosstalk simulation processor module 1700 generates contralateral sound components for output to stereo headphones, thereby providing a loudspeaker-like listening experience on the headphones. L may be the processed left component 134 / 1220 and the right input channel X R may be the processed right component 136 / 1222.
[0219] The crosstalk simulation processor module 1700 is L , the left head shadow low pass filter 1702, the left head shadow high pass filter 1724, the left crosstalk delay 1704, and the left head shadow gain 1710. The crosstalk simulation processor module 1700 includes a left head shadow low pass filter 1702, a left head shadow high pass filter 1724, a left crosstalk delay 1704, and a left head shadow gain 1710 to process the right input channel X R The crosstalk simulation processor module 1500 further includes a right head shadow low pass filter 1706, a right head shadow high pass filter 1726, a right crosstalk delay 1708, and a right head shadow gain 1712 to process . The crosstalk simulation processor module 1500 further includes a summing unit 1714 and a summing unit 1716.
[0220] The left head shadow low-pass filter 1702 and the left head shadow high-pass filter 1724 model the frequency response of the signal after it passes through the listener's head, which is the left input channel X L The output of the left head shadow high pass filter 1724 is fed to the left crosstalk delay 1704, which applies a time delay that represents the transaural distance passed by the contralateral sound component relative to the ipsilateral sound component. The left head shadow gain 1710 applies a gain to the output of the left crosstalk delay 1704 to modulate the right-left simulation channel W L Generate.
[0221] Similarly, the right input channel X R For right head shadow low pass filter 1706 and right head shadow high pass filter 1726 apply modulation to the right input channel XR that models the frequency response in the listener's head. The output of right head shadow high pass filter 1726 is fed to right crosstalk delay 1708, which applies a time delay. Right head shadow gain 1712 applies gain to the output of right crosstalk delay 1708 to generate the right crosstalk simulation channel W. RGenerate.
[0222] The application of the head shadow low pass filter, head shadow high pass filter, crosstalk delay, and head shadow gain to each of the left and right channels may be performed in different orders.
[0223] The summing unit 1714 outputs the right crosstalk simulation channel W R and left input channel X L to produce the left output channel O L The summing unit 1716 produces the left crosstalk simulation channel W L and right input channel X R to produce the left output channel O R Generate.
[0224] 18 is a block diagram illustrating a crosstalk cancellation processor module 1800 according to one or more embodiments. The crosstalk cancellation processor module 1800 is an example of the crosstalk cancellation processor module 110 or the crosstalk processor module 1224. Some embodiments of the cancellation processor module 1800 have different components than those described herein. Similarly, in some cases, functionality may be distributed among the components in a different manner than described herein.
[0225] The crosstalk cancellation processor module 1800 is L and right input channel X R Receive channel X L , X R Perform crosstalk cancellation on the left output channel O L and the right output channel O R Generates the left input channel X L may be the processed left component 134 / 1220 and the right input channel X R may be the processed right component 136 / 1222.
[0226] The crosstalk cancellation processor module 1800 includes an in-band / out-of-band splitter 1810, inverters 1820 and 1822, contralateral estimators 1830 and 1840, combiners 1850 and 1852, and an in-band / out-of-band combiner 1860. These components work in conjunction to combine the input channels T L , T R into in-band and out-of-band components and perform crosstalk cancellation on the in-band component to generate the output channel O L , O R Generate.
[0227] By splitting the input audio signal T into different frequency band components and performing crosstalk cancellation on selective components (e.g., in-band components), crosstalk cancellation can be performed on specific frequency bands while avoiding degradations in other frequency bands. If crosstalk cancellation is performed without splitting the input audio signal T into different frequency bands, the audio signal after such crosstalk cancellation may show significant attenuation or amplification of non-spatial and spatial components at low frequencies (e.g., below 350 Hz), high frequencies (e.g., above 12000 Hz), or both. By selectively performing crosstalk cancellation on in-band (e.g., between 250 Hz and 14000 Hz), where most of the influential spatial implications are, the overall energy in the mix can be kept balanced across the spectrum, especially in the non-spatial components.
[0228] The in-band / out-of-band splitter 1810 divides the input channel T L , T R In-band channel T L,In , T R,In , and the out-of-band channel T L,Out , T R,Out In particular, the in-band / out-of-band splitter 1810 splits the left enhancement compensation channel T L Left in-band channel T L,In and the left out-of-band channel T L,OutSimilarly, the in-band / out-of-band splitter 1810 splits the right enhancement compensation channel T R Right in-band channel T R,In , right out-of-band channel T R,Out Each in-band channel may encompass a portion of a respective input channel corresponding to a frequency range including, for example, 250 Hz to 14 kHz. The range of the frequency band may be adjustable, for example according to speaker parameters.
[0229] The inverter 1820 and the contralateral estimator 1830 work together to generate the left contralateral cancellation component S L to generate the left in-band channel T L,In Similarly, the inverter 1822 and the contralateral estimator 1840 work together to compensate for the right contralateral cancellation component S R to generate the right in-band channel T R,In Compensation is performed for the contralateral acoustic component caused by
[0230] In one approach, the inverter 1820 is L,In Receive the received in-band channel T L,In by inverting the polarity of the inverted in-band channel T L,In’ The contralateral estimator 1830 generates the inverted in-band channel T L,In’ , and through filter extraction, an inverted in-band channel T L,In’ The filter extraction extracts a portion of the inverted in-band channel T L,In’ , the portion extracted by the contralateral estimator 1830 is the in-band channel T L,In Therefore, the part extracted by the contralateral estimator 1830 is the left contralateral cancellation component S L This corresponds to the in-band channel T R,In In addition, the in-band channel T L,InIn some embodiments, the inverter 1820 and the contralateral estimator 1830 are implemented in different sequences.
[0231] The inverter 1822 and the contralateral estimator 1840 are connected to the in-band channel T R,In The same operation is performed for the right contralateral cancellation component S R Therefore, for the sake of brevity, a detailed description thereof will be omitted herein.
[0232] In one implementation, the contralateral estimator 1830 includes a filter 1832, an amplifier 1834, and a delay unit 1836. The filter 1832 receives an inverting input channel T L,In’ Receive the inverted in-band channel T corresponding to the contralateral acoustic component through a filter extraction function. L,In’ An example implementation of the filter is a notch filter or a high-pass filter with a center frequency chosen between 5000 Hz and 10000 Hz, and a Q chosen between 0.5 and 1 / 0. The gain in decibels (G dB ) can be derived from equation (26).
[0233]
number
[0234] where D is the delay in samples by the delay unit 1836, for example at a sampling rate of 48 KHz. An alternative implementation is a low pass filter with a corner frequency selected between 5000 Hz and 10000 Hz, and a Q selected between 0.5 and 1.0. Additionally, the amplifier 1834 divides the extracted portion by a corresponding gain factor G L,In and a delay unit 1836 delays the amplified output from the amplifier 1834 according to a delay function D to produce the left contralateral cancellation component S L The contralateral estimator 1840 generates a filter 1842, an amplifier 1844, and an inverted in-band channel TR,In’Perform a similar operation on the right contralateral cancellation component S R In one example, the contralateral estimators 1830, 1840 generate the left and right contralateral cancellation components S according to the following equation: L , S R That is,
[0235]
number
[0236] where F[] is the filter function and D[] is the delay function.
[0237] The crosstalk cancellation configuration can be determined by the parameters of the speakers. In one example, the filter center frequency, delay, amplifier gain, and filter gain can be determined according to the angle formed between the two speakers relative to the listener. In some embodiments, values between the speaker angles are used to interpolate other values.
[0238] The combiner 1850 cancels the right contralateral component S R Left in-band channel T L,In The left in-band crosstalk channel U L and the combiner 1852 generates the left contralateral cancellation component S L Right in-band channel T R,In to form the right in-band crosstalk channel U R The in-band / out-band combiner 1860 generates the left in-band crosstalk channel U L out-of-band channel T L,Out Combined with the left output channel O L and generates the right in-band crosstalk channel U R out-of-band channel T R,Out Combined with the right output channel O R Generate.
[0239] Therefore, the left output channel O L is the in-band channel T that belongs to the contralateral sound. R,InThe right contralateral cancellation component S R Right output channel O R is the in-band channel T that belongs to the contralateral sound. L,In The left contralateral cancellation component S L In this configuration, the right output channel O R When the wavefront of the ipsilateral acoustic component output by the right loudspeaker according to L Similarly, the left output channel O L When the wavefront of the ipsilateral acoustic component output by the left loudspeaker according to R The wavefront of the contralateral sound component output by the right speaker can be cancelled according to: Thus, the contralateral sound component can be reduced to enhance spatial detectability.
[0240] [Example of PSM process flow] 19 is a flow chart illustrating a process 1900 of PSM processing according to one or more embodiments. Process 1900 may include fewer or additional steps, and the steps may be performed in a different order. In some embodiments, PSM processing may be performed using a Hilbert Transform Based Perceptual Sound Stage Modification (HPSM) module.
[0241] In step 1905, an audio processing system (such as, for example, the PSM module 102 of the audio processing system 100 or 1200) separates the input channel into low and high frequency components. A crossover frequency that defines the boundary between the low and high frequency components may be adjustable so that frequencies subject to PSM processing are included in the high frequency components. In some embodiments, the audio processing system applies gain to the low and / or high frequency components.
[0242] An input channel may be a particular portion of an audio signal that is extracted for PSM processing. In some embodiments, an input channel is a mid or side component of an audio signal (e.g., stereo or multi-channel). In some embodiments, an input channel is a hyper-mid, hyper-side, residual mid, or residual side component of an audio signal. In some embodiments, an input channel is associated with an audio source, such as a voice or instrument, that is to be combined with other sounds into an audio mix.
[0243] In step 1910, the audio processing system applies a first Hilbert transform to the high frequency components to generate a first left leg component and a first right leg component, the first left leg component being 90 degrees out of phase with respect to the first right leg component.
[0244] In step 1915, the speech processing system applies a second Hilbert transform to the first right leg component to generate a second left leg component and a second right leg component, the first left leg component being 90 degrees out of phase with the first right leg component.
[0245] In some embodiments, the audio processing system applies a delay and / or gain to the first, left leg component. The audio processing system may apply a delay and / or gain to the second, right leg component. These gains and delays may be used to manipulate the perceptual results of the PSM processing.
[0246] In step 1920, the audio processing system combines the first left leg component with the low frequency components to generate a left channel. In step 1925, the audio processing system combines the second right leg component with the low frequency components to generate a right channel. The left channel may be provided to a left speaker and the right channel may be provided to a right speaker.
[0247] 20 is a flow chart illustrating another process 2000 for PSM processing using a first-order non-orthogonal rotation-based decorrelation (FNORD) filter network, according to some embodiments. The process shown in FIG. 20 may be performed by components of an audio system (such as systems 100, 202, or 1200). Other entities may perform some or all of the steps in FIG. 2 in other embodiments. Embodiments may include different and / or additional steps, or may perform steps in a different order.
[0248] In step 2005, the audio system determines a target amplitude response that defines one or more spatial suggestions to be coded into the mono audio signal to generate the resulting channels, the one or more spatial suggestions being associated with one or more frequency-dependent amplitude suggestions to be coded into the mid-space / side-space of the resulting channels, the mid-space / side-space of the channels not changing the overall coloration of the resulting channels. The one or more spatial suggestions may include at least one elevation angle suggestion associated with a target elevation angle. Each elevation angle suggestion may correspond to one or more frequency-dependent amplitude suggestions to be coded into the mid-space / side-space of the audio signal, such as a target amplitude function corresponding to a narrow region of infinite attenuation at one or more specific frequencies. On the other hand, left and right suggestions for elevation angles are typically symmetric in coloration, so that the left and right signals may be constrained to be colorless. In some embodiments, the spatial suggestions may be based on sampled HRTFs.
[0249] In some embodiments, the target amplitude response may further define one or more parametric spatial suggestions, which may include a target broadband attenuation, a target subband attenuation, a critical point, a filter characteristic, and / or a location of the sound stage where the suggestion should be embedded. The critical point may be an inflection point at 3 dB. The filter characteristic may include one of a high-pass filter characteristic, a low-pass characteristic, a band-pass characteristic, or a band-stop characteristic. The sound stage location may include a mid-channel or a side channel, or, if the number of output channels is more than two, other subspaces in the output space, such as those determined via pairwise and / or hierarchical addition and / or difference. The one or more spatial suggestions may be determined based on the characteristics of the presentation equipment (e.g., the frequency response of the speaker, the location of the speaker, etc.), the expected content of the audio data, the perceptual capabilities of the listener in the scene, or the minimum quality expected of the audio presentation system involved. For example, if the speaker is unable to reproduce frequencies below 200 Hz satisfactorily, spatial suggestions embedded in this range should be avoided. Similarly, if the expected audio content is speech, the audio system may select a target amplitude response that affects only those frequencies within the expected bandwidth of speech to which the ear is most sensitive. If the listener will derive audible cues from other sources in the situation, such as an array of speakers in the location, the audio system may determine a target amplitude response that complements those simultaneous cues.
[0250] In step 2010, the audio system determines a transfer function for a single-input, multiple-output all-pass filter based on the target magnitude response. The transfer function defines the relative rotations in phase angle of the output channels. The transfer function represents for each output the effect that the filter network has on its input in terms of phase angle rotations as a function of frequency.
[0251] In step 2015, the audio system determines coefficients of an all-pass filter based on the transfer function. These coefficients are selected in a manner that best suits the type of suggestions and / or constraints and are to be applied to the incoming audio stream. Some examples of coefficient sets are defined in equations (12), (13), (17), and (19). In some embodiments, determining the coefficients of the all-pass filter based on the transfer function includes using an inverse discrete Fourier transform (idft). In this case, the coefficient set may be determined as defined by equation (19). In some embodiments, determining the coefficients of the all-pass filter based on the transfer function includes using a phase-vocoder. In this case, the coefficient set may be determined as defined by equation (19), except that the coefficient set is to be applied to the frequency domain before resynthesizing the time domain data. In some embodiments, the coefficients include at least a rotation control parameter and a primary coefficient that are determined based on the received critical point parameters, filter characteristic parameters, and sound stage position parameters.
[0252] The audio system 2020 processes the mono channel with coefficients of an all-pass filter to generate a plurality of channels. For example, in some embodiments, the all-pass filter module receives the mono audio channel, performs wideband phase rotation on the mono audio channel to generate a plurality of wideband rotated component channels (e.g., left and right wideband rotated components) based on the rotation control parameter, and performs narrowband phase rotation on at least one of the plurality of wideband rotated component channels based on a primary coefficient to determine a narrowband rotated component channel, which together with one or more remaining channels of the wideband rotated component channel form the plurality of channels output by the audio system.
[0253] In some embodiments, if the system is operating in the time domain using an IIR implementation, as in equation (8), these coefficients may adjust the appropriate feedback and feedforward delays. If an FIR implementation is used, as in equation (19), only a feedforward delay may be used. If the coefficients are determined and applied in the spectral domain, the coefficients may be applied to the spectral data as complex multiplications before resynthesis. An audio system may provide multiple output channels to presentation equipment, such as user equipment connected to the audio system via a network.
[0254] The example PSM processing flows described above each utilize a network of all-pass filters to encode spatial suggestion by perceptually placing mono content at a particular location in the sound stage (e.g., a location associated with a target elevation angle). Because the networks of all-pass filters described herein are colorless, these filters allow the user to decouple the spatial placement of the sound from its overall coloration.
[0255] [Spatial processing of orthogonal components] FIG. 21 is a flow chart illustrating a process 2100 for spatial processing using at least one of the hyper-mid, residual mid, hyper-side, or residual side components, according to one or more embodiments. Spatial processing may include application of gain, amplitude or delay-based panning, binaural processing, reverberation processing, dynamic range processing such as compression and limiting, linear or non-linear audio processing techniques and effects, chorus effects, flanging effects, machine learning-based approaches to vocal or instrument style transfer, transformation, or resynthesis, among other techniques. This process may be performed to provide a spatially enhanced audio to a user's device. This process may include fewer or additional steps, and the steps may be performed in a different order.
[0256] In step 2110, an audio processing system (such as, for example, audio processing system 1200) receives an input audio signal (such as, for example, left channel 1202 and right input channel 1204). In some embodiments, the input audio signal may be a multi-channel audio signal including multiple left and right channel pairs. Each left and right channel pair may be processed as described herein for the left and right input channels.
[0257] In step 2120, the audio processing system generates a non-spatial mid component (such as, for example, the mid component 1208) and a spatial side component (such as, for example, the side component 1210) from the input audio signal. In some embodiments, an L / RM / S converter (such as, for example, the L / RM / S converter module 1206) performs the conversion of the input audio signal into the mid and side components.
[0258] In step 2130, the audio processing system generates at least one of a hyper-mid component (e.g., hyper-mid component M1, etc.), a hyper-side component (e.g., hyper-side component S1, etc.), a residual mid component (e.g., residual mid component M2, etc.), and a residual side component (e.g., residual side component S2, etc.). The audio processing system may generate at least one and / or all of the components listed above. The hyper-mid component includes the spectral energy of the side component removed from the spectral energy of the mid component. The residual mid component includes the spectral energy of the hyper-mid component removed from the spectral energy of the side component. The hyper-side component includes the spectral energy of the mid component removed from the spectral energy of the side component. The residual side component includes the spectral energy of the hyper-side component removed from the spectral energy of the side component. The processing used to generate M1, M2, S1, or S2 may be performed in the frequency domain or the time domain.
[0259] In step 2140, the audio processing system filter-extracts at least one of the hyper-mid, residual mid, hyper-side, and residual side components to enhance the audio signal. The filter-extraction may include HPSM processing, in which a series of Hilbert transforms are applied to the high frequency components of the hyper-mid, residual mid, hyper-side, or residual side components. In one example, the hyper-mid component undergoes HPSM processing, while one or more of the residual mid, hyper-side, or residual side components undergo other types of filter-extraction.
[0260] The filter extraction may include PSM processing, where the spatial implications are colorlessly encoded either through a parametric specification of the spatial implications, as described above in more detail in connection with Figures 10A and 10B, or via anthropometric sample extraction of the HRTF data, as described above in connection with equation (20). In one example, the hyper-mid component undergoes PSM processing, while one or more of the residual mid, hyper-side, or residual side components undergo no or other types of filter extraction.
[0261] Filter extraction may include other types of filter extraction, such as spatially-indicated processing. Spatially-indicated processing may include adjusting the frequency-dependent amplitude or frequency-dependent delay of the hyper-mid, residual mid, hyper-side, or residual side components. Examples of spatially-indicated processing include amplitude or delay-based panning, or binaural processing.
[0262] The filter extraction may include dynamic range processing, such as compression or limiting. For example, the hyper-mid, residual mid, hyper-side, or residual side components may be compressed according to a compression ratio when exceeding a threshold level for compression. In another example, the hyper-mid, residual mid, hyper-side, or residual side components may be limited to a maximum level when exceeding a threshold level for limiting.
[0263] Filter extraction may include machine learning based modifications to the hyper-mid, residual mid, hyper-side, or residual side components. Some examples include machine learning based vocal or instrument style transfer, transformation, or resynthesis.
[0264] The filter extraction of the hyper-mid, residual mid, hyper-side, or residual side components may include application of gain, reverberation processing, and other linear or non-linear audio processing techniques and effects ranging from chorus and / or flanging, or other types of processing. In some embodiments, the filter extraction may include filter extraction for sub-band spatial processing and crosstalk compensation, as described in more detail below in connection with FIG.
[0265] The filter extraction may be performed in the frequency domain or the time domain. In some embodiments, the mid and side components are transformed from the time domain to the frequency domain, the hyper and / or residual components are generated in the frequency domain and the filter extraction is performed in the frequency domain, the filter extracted components are transformed to the time domain and the filter extraction is performed in the time domain on these components.
[0266] In step 2150, the audio processing system generates a left output channel (e.g., left output channel 1242) and a right output channel (e.g., right output channel 1244) using one or more of the filter-extracted hyper / residual components. For example, the M / S to L / R conversion can be performed using a mid component or a side component generated from at least one of the filter-extracted hyper-mid component, the filter-extracted residual mid component, the filter-extracted hyper-side component, or the filter-extracted residual side component. In another example, the filter-extracted hyper-mid component or the filter-extracted residual mid component can be used as the mid component for the M / S to L / R conversion, or the filter-extracted hyper-side component or the residual side component can be used as the side component for the M / S to L / R conversion.
[0267] [Subband spatial processing of orthogonal components and crosstalk processing] FIG. 22 is a flow chart illustrating a process 2200 for sub-band spatial processing and compensation for crosstalk processing using at least one of hyper-mid, residual mid, hyper-side, or residual side components, according to one or more embodiments. Crosstalk processing may include crosstalk cancellation or crosstalk simulation. Sub-band spatial processing may be performed to provide audio content with enhanced spatial detectability (e.g., sound stage enhancement, etc.), such as by creating the perception that sounds are directed to the listener from a wide area rather than a specific point in space corresponding to the loudspeaker's location, thereby providing the listener with a more immersive listening experience. Crosstalk simulation may be used on audio output to headphones to simulate a loudspeaker experience with contralateral crosstalk. Crosstalk cancellation may be used on audio output to loudspeakers to remove the effects of crosstalk interference. Crosstalk compensation compensates for spectral defects caused by crosstalk cancellation or crosstalk simulation. The process may include fewer or additional steps, and the steps may be performed in a different order. The hyper and residual mid / side components can be manipulated in different ways for different purposes. For example, in the case of crosstalk compensation, a targeted subband filter extraction may be applied only to the hyper-mid component M1 (where most of the vocal dialog energy in most movie content occurs) in an effort to remove spectral artifacts resulting from crosstalk processing in only that component. When enhancing the sound stage with or without crosstalk processing, a targeted subband gain may be applied to the residual mid component M2 and the residual side component S2.For example, the residual mid component M2 may be attenuated and the residual side component S2 may be de-amplified to increase the distance between these components in terms of gain, while also avoiding attenuation in the hyper-mid M1 component (e.g., which is often that part of the signal that contains the majority of the voiced energy), without causing a dramatic overall change in the perceived sound of the final L / R signal (which, if done tastefully, can increase spatial detectability).
[0268] In step 2210, the audio processing system receives an input audio signal including a left channel and a right channel. In some embodiments, the input audio signal may be a multi-channel audio signal including multiple left and right channel pairs. Each left and right channel pair may be processed as described herein for left and right input channels.
[0269] In step 2220, the audio processing system applies crosstalk processing to the received input audio signal. The crosstalk processing includes at least one of crosstalk simulation and crosstalk cancellation.
[0270] In steps 2230 through 2260, the audio processing system performs subband spatial processing and crosstalk compensation for crosstalk processing using one or more of the hyper-mid, hyper-side, residual mid, or residual side components. In some embodiments, the crosstalk processing may be performed after the processing of steps 2230 through 2260.
[0271] In step 2230, the audio processing system generates a mid component and a side component from the (eg, crosstalk processed) audio signal.
[0272] In step 2240, the audio processing system generates at least one of a hyper-mid component, a residual mid component, a hyper-side component, and a residual side component. The audio processing system may generate at least one and / or all of the components listed above.
[0273] In step 2250, the audio processing system filter extracts at least one subband of the hyper-mid component, the residual mid component, the hyper-side component, and the residual side component, and applies subband spatial processing to the audio signal. Each subband may include a frequency range, such as may be defined by a set of critical bands. In some embodiments, the subband spatial processing further includes time-delaying at least one subband of the hyper-mid component, the residual mid component, the hyper-side component, and the residual side component. In some embodiments, the filter extraction includes applying HPSM processing.
[0274] In step 2260, the audio processing system filters out at least one of the hyper-mid, residual mid, hyper-side, and residual side components to compensate for spectral imperfections from crosstalk processing of the input audio signal. The spectral imperfections may include peaks or troughs in a frequency response plot of the hyper-mid, residual mid, hyper-side, or residual side components that exceed a predetermined threshold (e.g., 10 dB) that occur as artifacts of crosstalk processing. The spectral imperfections may be estimated spectral imperfections.
[0275] In some embodiments, the filter extraction of spectral orthogonal components for subband spatial processing in step 2250 and the crosstalk compensation in step 2260 may be combined into a single filter extraction operation for each spectral orthogonal component selected for filter extraction.
[0276] In some embodiments, filter extraction on the hyper / residual mid / side components for subband spatial processing or crosstalk compensation may be performed in conjunction with filter extraction for other purposes, such as application of gain, amplitude or delay based panning, binaural processing, reverberation processing, dynamic range processing such as compression and limiting, linear or non-linear audio processing techniques and effects ranging from chorus and / or flanging, machine learning based approaches to vocal or instrument style transfer, transformation or resynthesis, or other types of processing using any of the hyper-mid, residual mid, hyper-side or residual side components.
[0277] The filter extraction may be performed in the frequency domain or the time domain. In some embodiments, the mid and side components are transformed from the time domain to the frequency domain, the hyper and / or residual components are generated in the frequency domain, the filter extraction is performed in the frequency domain, and the filter extracted components are transformed to the time domain. In other embodiments, the hyper and / or residual components are transformed to the time domain, and the filter extraction is performed on these components in the time domain.
[0278] In step 2270, the audio processing system generates left and right output channels from the filtered-extracted hyper-mid component. In some embodiments, the left and right output channels are additionally based on at least one of the filtered-extracted residual mid component, the filtered-extracted hyper-side component, and the filtered-extracted residual side component.
[0279] [Computer example] 23 is a block diagram illustrating a computer 2300, according to some embodiments. The computer 2300 is an example of a computing device that includes circuitry to implement an audio system, such as the audio systems 100, 202, or 1200. Illustrated is at least one processor 2302 coupled to a chipset 2304. The chipset 2304 includes a memory controller hub 2320 and an input / output (I / O) controller hub 2322. The memory 2306 and the graphics adapter 2312 are coupled to the memory controller hub 2320, and the display 2318 is coupled to the graphics adapter 2312. The storage device 2308, the keyboard 2310, the pointing device 2314, and the network adapter 2316 are coupled to the I / O controller hub 2322. The computer 2300 may include various types of input or output devices. Other embodiments of the computer 2300 have different architectures. For example, in some embodiments, the memory 2306 is directly connected to the processor 2302.
[0280] The storage device 2308 includes one or more non-transitory computer-readable storage media, such as a hard drive, a compact disc read-only memory (CD-ROM), a DVD, or a solid-state memory device. The memory 2306 holds program code (configured by one or more instructions) and data used by the processor 2302. This program code may correspond to the processing aspects described with reference to Figures 1-3.
[0281] 23. A pointing device 2314 is used in combination with the keyboard 2310 to input data into the computer system 2300. The graphics adapter 2312 displays images and other information on a display 2318. In some embodiments, the display 2318 includes touch screen capabilities for receiving user input and user selections. The network adapter 2316 connects the computer system 2300 to a network. Some embodiments of the computer 2300 have different and / or other components than those shown in FIG.
[0282] The circuitry may include one or more processors executing program code stored on a non-transitory computer readable medium that, when executed by the one or more processors, configures the one or more processors to implement an audio system or a module of the audio system. Other examples of circuitry for implementing an audio system or a module of the audio system may include integrated circuits, such as an application specific integrated circuit (AS1C), a field programmable gate array (FPGA), or other types of computer circuitry.
[0283] [Additional considerations] Examples of benefits and advantages of the disclosed configuration include dynamic audio enhancement resulting from an audio system that is adaptively enhanced to the device and associated audio rendering system, as well as other relevant information made available by the device OS, such as use case information (e.g., indicating that the audio signal is used for music playback, but not for gaming). The enhanced audio system can either be integrated into the device (using a software development kit) or stored on a remote server for on-demand accessibility. In this way, the device does not need to allocate storage or processing resources to the maintenance of an audio enhancement system that is specific to its audio rendering system or audio rendering configuration. In some embodiments, the enhanced audio system allows for different levels of querying of rendering system information, such that effective audio enhancements can be applied across different levels of available device-specific rendering information.
[0284] Throughout this specification, a component, operation, or structure described as a single instance may be implemented by multiple instances. Although individual operations in one or more methods are illustrated and described as separate operations, one or more individual operations may be performed simultaneously and do not require the operations to be performed in the order illustrated. Structures and functions presented as separate components in example configurations may be implemented as combined structures or components. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are included within the scope of the subject matter herein.
[0285] Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. A module may constitute either a software module (e.g., code embodied on a machine-readable medium or in a transmission signal) or a hardware module. A hardware module is a tangible unit capable of performing certain operations and may be configured or arranged in a particular way. In example embodiments, one or more computer systems (e.g., standalone, client or server computer systems, etc.) or one or more hardware modules of a computer system (e.g., a processor or group of processors, etc.) are configured as hardware modules that operate by software (e.g., an application or portion of an application) to perform certain operations described herein.
[0286] Various operations in the example methods described herein may be performed, at least in part, by one or more processors that are configured temporarily (e.g., by software) or permanently configured to perform the associated operations. Such processors, whether configured temporarily or permanently, may constitute processor-implemented modules that operate to perform one or more operations or functions. Modules referred to herein may, in some example embodiments, include processor-implemented modules.
[0287] Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations in the methods may be performed by one or more processors or processor-implemented hardware modules. Performance for a particular operation may be distributed among one or more processors that reside within a single machine, as well as spread across multiple machines. In some example embodiments, a processor or processors may be located in a single location (e.g., in a home environment, in an office environment, or as a server farm), while in other embodiments, the processors may be distributed across multiple locations.
[0288] Unless otherwise noted, descriptions herein using terms such as "processing," "computing," "calculating," "determining," "presenting," or "displaying" may refer to machine (e.g., a computer) operations or processes that manipulate or transform data represented as physical (e.g., electronic, magnetic, or optical) quantities in one or more memories (e.g., volatile memory, nonvolatile memory, or a combination thereof), registers, or other machine components that receive, store, send, transmit, or display information.
[0289] As used herein, a reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment.
[0290] Some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited in this context.
[0291] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements and may include other elements not expressly listed or that are inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), or both A and B are true (or absent).
[0292] Furthermore, the use of "a" or "an" is used to describe elements and components of the embodiments herein. This is done merely for convenience and to understand the ordinary meaning in the present invention. This description should be interpreted to include one or at least one, and the singular also includes the plural, unless it is clear that it is meant otherwise.
[0293] In some parts of this description, embodiments are described in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. These operations, while described functionally, computationally, or logically, will be understood to be implemented by computer programs, or equivalent electrical circuits, or microcode, or the like. Further, without loss of generality, it has proven convenient at times to refer to arrangements of these operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.
[0294] Any of the steps, operations, or processes described herein may be performed or implemented by one or more hardware or software modules, alone or in combination with other devices. In some embodiments, the software modules are implemented by a computer program product that includes a computer readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations, or processes described.
[0295] The embodiments may also relate to an apparatus for performing the operations herein. The apparatus may be specifically constructed for the required purposes and / or may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions that may be coupled to a computer system bus. Furthermore, all computing systems referred to herein may include a single processor or may be architectures employing multiple processor designs to increase computing power.
[0296] Embodiments may also relate to articles of manufacture produced by the computing processes described herein. Such articles of manufacture may include information resulting from the computing processes, which information may be stored on a non-transitory, tangible, computer-readable storage medium, and may include any of the embodiments in combination with a computer program product or other data described herein.
[0297] Upon reading this disclosure, those skilled in the art will appreciate additional and alternative structural and functional designs of systems and processes for audio content decorrelation according to the principles disclosed herein. Thus, while specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims.
[0298] Finally, the language used herein has been selected primarily for ease of reading and educational purposes, and may not have been selected to delineate or limit patent rights. Accordingly, it is intended that the scope of patent rights be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to illustrate, but not limit, the scope of patent rights, which are set forth in the following claims.
Claims
1. 1. A system comprising: one or more processors; A non-transitory computer-readable medium that, when executed by the one or more processors, causes the one or more processors to: Separating an audio channel into low and high frequency components; applying a first Hilbert transform to the high frequency components to generate a first left leg component and a first right leg component, the first left leg component being 90 degrees out of phase with the first right leg component; applying a second Hilbert transform to the first right leg component to generate a second left leg component and a second right leg component, the second left leg component being 90 degrees out of phase with the second right leg component; combining the first left leg component with the low frequency component to generate a left channel; combining the second right leg component with the low frequency component to generate a right channel; a non-transitory computer readable medium having stored thereon program code configured to perform the steps of: A system comprising:
2. 2. The system of claim 1 , wherein the program code further configures the one or more processors to apply a first gain to the low frequency components and a second gain to the high frequency components, the first gain and the second gain being different.
3. 2. The system of claim 1, wherein the program code further configures the one or more processors to apply a first delay to the first left leg component and a second delay to the second right leg component, the first delay and the second delay being different.
4. 2. The system of claim 1, wherein the program code further configures the one or more processors to apply a first gain to the first left leg component and a second gain to the second right leg component, the first gain and second gain being different.
5. The program code for configuring the one or more processors to apply the first Hilbert transform to the high frequency components includes: applying a first series of all pass filters to the high frequency component to generate the first left leg component; applying a first delay and a second series of all-pass filters to the high frequency component to generate the first right leg component; and configuring the device to: The program code configuring the one or more processors to apply the second Hilbert transform to the first right leg component further comprises: applying a third series of all-pass filters to the first right leg component to generate the second left leg component; applying a second delay and a fourth series of all-pass filters to the first right leg component to generate the second right leg component; Configure to do 2. The system of claim 1 .
6. The program code causes the one or more processors to: generating mid and side components from left and right input channels of an audio signal; generating a hyper-mid component that includes the spectral energy of the side component removed from the spectral energy of the mid component; 2. The system of claim 1, further configured to generate the audio channel by:
7. The program code causes the one or more processors to: generating a mid component and a side component from the left channel and the right channel; applying a filter extraction to the mid and side components; generating a left output channel and a right output channel from the filtered mid component and the filtered side component; The system of claim 1 , further configured to:
8. 2. The system of claim 1, wherein the program code further configures the one or more processors to generate the audio channels by combining channels of a multi-channel audio signal.
9. 2. The system of claim 1, wherein the program code further configures the one or more processors to generate the audio channels by isolating portions of an audio signal.
10. 2. The system of claim 1, wherein the high frequency components include sounds for vocalization.
11. A non-transitory computer readable medium having program code stored thereon, the program code, when executed by one or more processors, causing the one or more processors to: Separating an audio channel into low and high frequency components; applying a first Hilbert transform to the high frequency components to generate a first left leg component and a first right leg component, the first left leg component being 90 degrees out of phase with the first right leg component; applying a second Hilbert transform to the first right leg component to generate a second left leg component and a second right leg component, the second left leg component being 90 degrees out of phase with the second right leg component; combining the first left leg component with the low frequency component to generate a left channel; combining the second right leg component with the low frequency component to generate a right channel; 16. A non-transitory computer-readable medium configured to:
12. 12. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to apply a first gain to the low frequency components and a second gain to the high frequency components, the first gain and the second gain being different.
13. 12. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to apply a first delay to the first left leg component and a second delay to the second right leg component, wherein the first delay and the second delay are different.
14. 12. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to apply a first gain to the first left leg component and a second gain to the second right leg component, wherein the first gain and the second gain are different.
15. The program code for configuring the one or more processors to apply the first Hilbert transform to the high frequency components includes: applying a first series of all pass filters to the high frequency component to generate the first left leg component; applying a first delay and a second series of all-pass filters to the high frequency component to generate the first right leg component; and configuring the device to: The program code configuring the one or more processors to apply the second Hilbert transform to the first right leg component further comprises: applying a third series of all-pass filters to the first right leg component to generate the second left leg component; applying a second delay and a fourth series of all-pass filters to the first right leg component to generate the second right leg component; Configure to do 12. The computer readable medium of claim 11.
16. The program code causes the one or more processors to: generating mid and side components from left and right input channels of an audio signal; generating a hyper-mid component that includes the spectral energy of the side component removed from the spectral energy of the mid component; 12. The computer-readable medium of claim 11, further configured to generate the audio channel by:
17. The program code causes the one or more processors to: generating a mid component and a side component from the left channel and the right channel; applying a filter extraction to the mid and side components; generating a left output channel and a right output channel from the filtered mid component and the filtered side component; 12. The computer readable medium of claim 11, further configured to:
18. 12. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to generate the audio channels by combining channels of a multi-channel audio signal.
19. 12. The computer readable medium of claim 11, wherein the program code further configures the one or more processors to generate the audio channels by isolating portions of an audio signal.
20. 12. The computer-readable medium of claim 11, wherein the high frequency components include sounds for vocalization.
21. 1. A method, comprising: Separating an audio channel into low and high frequency components; applying a first Hilbert transform to the high frequency components to generate a first left leg component and a first right leg component, the first left leg component being 90 degrees out of phase with the first right leg component; applying a second Hilbert transform to the first right leg component to generate a second left leg component and a second right leg component, the second left leg component being 90 degrees out of phase with the second right leg component; combining the first left leg component with the low frequency component to generate a left channel; combining the second right leg component with the low frequency component to generate a right channel; A method comprising:
22. applying, by the one or more processors, a first gain to the low frequency components and a second gain to the high frequency components, the first gain and the second gain being different.
22. The method of claim 21 further comprising:
23. applying, by the one or more processors, a first delay to the first left leg component and a second delay to the second right leg component, the first delay and the second delay being different.
22. The method of claim 21 further comprising:
24. applying, by the one or more processors, a first gain to the first left leg component and a second gain to the second right leg component, the first gain and the second gain being different.
22. The method of claim 21 further comprising:
25. 2. The step of applying a first Hilbert transform to the high frequency components, applying a first series of all-pass filters to the high frequency components to generate the first left leg component; applying a first delay and a second series of all-pass filters to the high frequency component to generate the first right leg component. Including, The step of applying the second Hilbert transform to the first right leg component comprises: applying a third series of all-pass filters to the first right leg component to generate the second left leg component; applying a second delay and a fourth series of all-pass filters to the first right leg component to generate the second right leg component.
22. The method of claim 21 further comprising:
26. by said one or more processors; generating mid and side components from left and right input channels of an audio signal; generating a hyper-mid component that includes the spectral energy of the side component removed from the spectral energy of the mid component; 22. The method of claim 21, further comprising generating the audio channel by:
27. by said one or more processors; generating a mid component and a side component from the left channel and the right channel; applying a filter extraction to the mid and side components; generating a left output channel and a right output channel from the filtered mid component and the filtered side component; 22. The method of claim 21 further comprising:
28. 22. The method of claim 21, further comprising generating, by the one or more processors, the audio channels by combining channels of a multi-channel audio signal.
29. 22. The method of claim 21, further comprising the step of generating, by the one or more processors, the audio channels by isolating portions of an audio signal.
30. 22. The method of claim 21, wherein the high frequency components include sounds for vocalization.
Citation Information
Patent Citations
Signal processing method and signal processing apparatus
JP2009276268A
Modulating device, demodulating device, information transmission system, modulating method and demodulating method
JP2010016625A
Hybrid derivation of surround sound audio channels by controllably combining ambient signal components and matrix-decoded signal components.
JP2010529780A
Audio spatial environment up-mixer
US20060093152A1
Audio Spatial Environment Engine
US20090060204A1
