Decoding of multiple audio signals

Abstract

A residual scaling unit is configured to determine a scaling factor for a residual channel based on an inter-channel mismatch value. The inter-channel mismatch value is indicative of a time alignment between a reference channel and a target channel. The residual scaling unit is further configured to scale (e.g., attenuate) the residual channel according to the scaling factor to generate a scaled residual channel. A residual channel encoder is configured to encode the scaled residual channel as part of a bitstream.

Description

[0001] This application is a divisional application of the invention patent application filed on December 11, 2017, with application number 201780081733.4 and invention title "Decoding of Multiple Audio Signals".

[0002] Priority Claim

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 448,287, filed January 19, 2017, entitled "Carding of Multiple Audio Signals," and U.S. Non-Provisional Patent Application No. 15 / 836,604, filed December 8, 2017, entitled "Carding of Multiple Audio Signals," the entire contents of each of which are expressly incorporated herein by reference. Technical Field

[0004] This invention generally relates to the decoding (e.g., encoding or decoding) of multiple audio signals. Background Technology

[0005] Technological advancements have led to smaller and more powerful computing devices. For example, a variety of portable personal computing devices exist today, including cordless phones (such as mobile phones and smartphones), tablets, and laptops. These devices are small, lightweight, and easily carried by the user. They can transmit voice and data packets via wireless networks. Furthermore, many of these devices incorporate additional functionality, such as digital still cameras, digital video cameras, digital recorders, and audio file players. Moreover, these devices can process executable instructions, including software applications, such as web browser applications that can access the internet. Therefore, these devices can contain significant computing power.

[0006] A computing device may include or be coupled to multiple microphones to receive audio signals. Generally, the sound source is closer to the first microphone than to the second microphone among multiple microphones. Therefore, due to the corresponding distance between the microphone and the sound source, the second audio signal received from the second microphone may be delayed relative to the first audio signal received from the first microphone. In other embodiments, the first audio signal may be delayed relative to the second audio signal. In stereo coding, the audio signal from the microphone may be encoded to produce a center channel signal and one or more side channel signals. The center channel signal may correspond to the sum of the first and second audio signals. The side channel signals may correspond to the difference between the first and second audio signals. Because of the delay in receiving the second audio signal relative to the first audio signal, the first audio signal may not be aligned with the second audio signal. This misalignment (e.g., timing mismatch) of the first audio signal relative to the second audio signal can increase the difference between the two audio signals.

[0007] When the time mismatch between the first and second channels (e.g., the first signal and the second signal) is significant, the analysis and synthesis windows in the Discrete Fourier Transform (DFT) parameter estimation process tend to become inappropriately mismatched. Summary of the Invention

[0008] In a particular embodiment, an apparatus includes a first transformation unit configured to perform a first transformation operation on a reference channel to generate a frequency domain reference channel. The apparatus further includes a second transformation unit configured to perform a second transformation operation on a target channel to generate a frequency domain target channel. The apparatus further includes a stereo channel adjustment unit configured to determine an inter-channel mismatch value indicating time misalignment between the frequency domain reference channel and the frequency domain target channel. The stereo channel adjustment unit is also configured to adjust the frequency domain target channel based on the inter-channel mismatch value to generate an adjusted frequency domain target channel. The apparatus further includes a downmixer configured to perform a downmixing operation on the frequency domain reference channel and the adjusted frequency domain target channel to generate a center channel and side channels. The apparatus further includes a residual generation unit configured to generate a predicted side channel based on the center channel. The predicted side channel corresponds to a prediction of the side channel. The residual generation unit is further configured to generate a residual channel based on the side channels and the predicted side channels. The apparatus further includes a residual scaling unit configured to determine a scaling factor for the residual channel based on the inter-channel mismatch value. The residual scaling unit is further configured to scale the residual channel according to the scaling factor to generate a scaled residual channel. The apparatus further includes a middle channel encoder configured to encode the middle channel as a portion of the bitstream. The apparatus further includes a residual channel encoder configured to encode the scaled residual channel as a portion of the bitstream.

[0009] In another specific embodiment, a communication method includes performing a first transformation operation on a reference channel at an encoder to generate a frequency-domain reference channel. The method further includes performing a second transformation operation on a target channel to generate a frequency-domain target channel. The method further includes determining an inter-channel misalignment value indicating a time misalignment between the frequency-domain reference channel and the frequency-domain target channel. The method further includes adjusting the frequency-domain target channel based on the inter-channel misalignment value to generate an adjusted frequency-domain target channel. The method further includes performing a downmixing operation on the frequency-domain reference channel and the adjusted frequency-domain target channel to generate a center channel and side channels. The method further includes generating a predicted side channel based on the center channel. The predicted side channel corresponds to a prediction of the side channel. The method further includes generating a residual channel based on the side channel and the predicted side channel. The method further includes determining a scaling factor for the residual channel based on the inter-channel misalignment value. The method further includes scaling the residual channel according to the scaling factor to produce a scaled residual channel. The method further includes encoding the intermediate channel and the scaled residual channel as a portion of a bitstream.

[0010] In another specific embodiment, a non-transitory computer-readable medium includes instructions that, when executed by a processor within an encoder, cause the processor to perform operations, the operations including performing a first transformation operation on a reference channel to generate a frequency-domain reference channel. The operations also include performing a second transformation operation on a target channel to generate a frequency-domain target channel. The operations further include determining an inter-channel misalignment value indicating a time misalignment between the frequency-domain reference channel and the frequency-domain target channel. The operations further include adjusting the frequency-domain target channel based on the inter-channel misalignment value to generate an adjusted frequency-domain target channel. The operations further include performing a downmixing operation on the frequency-domain reference channel and the adjusted frequency-domain target channel to generate a center channel and side channels. The operations further include generating a predicted side channel based on the center channel. The predicted side channel corresponds to a prediction of the side channel. The operations further include generating a residual channel based on the side channel and the predicted side channel. The operations further include determining a scaling factor for the residual channel based on the inter-channel misalignment value. The operation further includes scaling the residual channel according to the scaling factor to produce a scaled residual channel. The operation also includes encoding the intermediate channel and the scaled residual channel as a portion of a bitstream.

[0011] In another specific embodiment, an apparatus includes means for performing a first transformation operation on a reference channel to generate a frequency-domain reference channel. The apparatus further includes means for performing a second transformation operation on a target channel to generate a frequency-domain target channel. The apparatus further includes means for determining an inter-channel mismatch value indicating a time misalignment between the frequency-domain reference channel and the frequency-domain target channel. The apparatus further includes means for adjusting the frequency-domain target channel based on the inter-channel mismatch value to generate an adjusted frequency-domain target channel. The apparatus further includes means for performing a downmixing operation on the frequency-domain reference channel and the adjusted frequency-domain target channel to generate a center channel and side channels. The apparatus further includes means for generating a predicted side channel based on the center channel. The predicted side channel corresponds to a prediction of the side channel. The apparatus further includes means for generating a residual channel based on the side channel and the predicted side channel. The apparatus further includes means for determining a scaling factor for the residual channel based on the inter-channel mismatch value. The device further includes means for scaling the residual channel according to the scaling factor to produce a scaled residual channel. The device also includes means for encoding the intermediate channel and the scaled residual channel as portions of a bitstream.

[0012] After reviewing the entire application, other embodiments, advantages and features of the invention will become apparent, the entire application comprising the following sections: description of drawings, detailed description and claims. Attached Figure Description

[0013] Figure 1 A block diagram of a specific illustrative example of a system containing an encoder operable to encode multiple audio signals;

[0014] Figure 2 For illustration Figure 1 A diagram of an instance of an encoder;

[0015] Figure 3 For illustration Figure 1 A diagram of another instance of an encoder;

[0016] Figure 4 A diagram illustrating an instance of a decoder;

[0017] Figure 5 A flowchart illustrating a method for decoding audio signals;

[0018] Figure 6 A block diagram of a specific illustrative example of a device operable to encode multiple audio signals;

[0019] Figure 7 A block diagram for a specific illustrative example of a base station. Detailed Implementation

[0020] Specific aspects of the invention are described below with reference to the accompanying drawings. In the description, common features are indicated by common reference numerals. As used herein, various terms are used only for the purpose of describing particular embodiments and are not intended to limit the embodiments. For example, unless the context clearly indicates otherwise, the singular forms “a / an” and “the” are intended to also include the plural forms. It will be further understood that the terms “comprises” and “comprising” are used interchangeably with “includes” or “including”. Additionally, it will be understood that the terms “wherein” and “where” are used interchangeably. As used herein, ordinal terms (e.g., “first,” “second,” “third,” etc.) used to modify an element (e.g., structure, component, operation, etc.) do not themselves indicate any priority or order of said element relative to another element, but merely distinguish said element from another element having the same name (unless an ordinal term is used). As used herein, the term “set” refers to one or more of a particular element, and the term “a plurality” refers to a plurality of a particular element (e.g., two or more).

[0021] In this invention, terms such as “determine,” “calculate,” “shift,” and “adjust” are used to describe how one or more operations are performed. It should be noted that these terms should not be considered limiting and other techniques can be used to perform similar operations. Additionally, as mentioned herein, “generate,” “calculate,” “use,” “select,” “access,” and “determine” are used interchangeably. For example, “generate,” “calculate,” or “determine” a parameter (or signal) can refer to actively generating, calculating, or determining a parameter (or signal), or it can refer to using, selecting, or accessing a parameter (or signal) that has been generated, for example, by another component or device.

[0022] Systems and apparatuses operable to encode multiple audio signals are disclosed. The apparatus may include an encoder configured to encode multiple audio signals. Multiple recording devices (e.g., multiple microphones) can be used to capture multiple audio signals simultaneously in time. In some instances, multiple audio signals (or multichannel audio) can be synthesized (e.g., artificially) by multiplexing several audio channels recorded simultaneously or at different times. As illustrative examples, simultaneous recording or multiplexing of audio channels can produce 2-channel configurations (i.e., stereo: left and right), 5.1-channel configurations (left, right, center, left surround, right surround, and low-frequency accent (LFE) channels), 7.1-channel configurations, 7.1+4-channel configurations, 22.2-channel configurations, or N-channel configurations.

[0023] An audio capture device in a telephone conference room (or telepresence room) may include multiple microphones for acquiring spatial audio. The spatial audio may include speech and encoded, transmitted background audio. Depending on the arrangement of the microphones and the location of a given source (e.g., a speaker) relative to the microphones and room dimensions, speech / audio from that source (e.g., a speaker) may arrive at the multiple microphones at different times. For example, a sound source (e.g., a speaker) may be closer to a first microphone associated with the device than a second microphone associated with the device. Therefore, sound from the sound source may arrive at the first microphone earlier than at the second microphone. The device may receive a first audio signal via the first microphone and a second audio signal via the second microphone.

[0024] Mid-side (MS) decoding and parametric stereo (PS) decoding are stereo decoding techniques that offer improved efficiency compared to dual-mono decoding. In dual-mono decoding, the left (L) channel (or signal) and right (R) channel (or signal) are decoded independently without utilizing inter-channel correlation. Before decoding, MS decoding reduces redundancy between correlated L / R channel pairs by transforming the left and right channels into sum and difference channels (e.g., side channels). The sum and difference signals are decoded either by waveform decoding or based on a model in MS decoding. The sum signal requires relatively more bits than the side signal. PS decoding reduces redundancy in each subband by transforming the L / R signals into a sum signal and a set of side parameters. The side parameters can indicate inter-channel intensity difference (IID), inter-channel phase difference (IPD), inter-channel time difference (ITD), side or residual prediction gain, etc. The sum signal is the decoded waveform and is transmitted along with the side parameters. In a hybrid system, side channels may be waveform-decoded in a lower frequency band (e.g., less than 2 kHz) and PS-decoded in a higher frequency band (e.g., greater than or equal to 2 kHz), where inter-channel phase maintenance is less perceptibly critical. In some implementations, PS decoding may also be used in the lower frequency band prior to waveform decoding to reduce inter-channel redundancy.

[0025] MS and PS decoding can be performed in the frequency domain or sub-band domain. In some instances, the left and right channels may be uncorrelated. For example, the left and right channels may contain uncorrelated synthesized signals. When the left and right channels are uncorrelated, the decoding efficiency of MS, PS, or both can approach that of dual-mono decoding.

[0026] Depending on the recording configuration, there may be time mismatch and other spatial effects (such as echo and room reverberation) between the left and right channels. If the time and phase mismatch between channels is not compensated, the sum and difference channels may contain comparable energy that reduces the decoding gain associated with MS or PS techniques. The reduction in decoding gain can be based on the amount of time (or phase) mismatch. The comparable energy of the sum and difference signals can limit the use of MS decoding in certain frames where the channels are time-mismatched but highly correlated. In stereo decoding, the center channel (e.g., the sum channel) and side channels (e.g., the difference channel) can be generated based on the following formula:

[0027] M=(L+R) / 2, S=(LR) / 2, Equation 1

[0028] Where M corresponds to the center channel, S corresponds to the side channel, L corresponds to the left channel, and R corresponds to the right channel.

[0029] In some cases, the center channel and side channels can be generated based on the following formula:

[0030] M=c(L+R), S=c(LR), Equation 2

[0031] Where c corresponds to a frequency-dependent complex value. The generation of the center and side channels based on Equation 1 or Equation 2 can be called "downmixing". The reverse process of generating the left and right channels from the center and side channels based on Equation 1 or Equation 2 can be called "upmixing".

[0032] In some cases, the middle channel can be based on other methods, such as:

[0033] M=(L+g D R) / 2, or equation 3

[0034] M = g1L + g2R (Equation 4)

[0035] Where g1 + g2 = 1.0, and where g D Here is the gain parameter. In other instances, demixing can be performed in the frequency band, where mid(b) = c1L(b) + c2R(b), where c1 and c2 are complex numbers, and side(b) = c3L(b) - c4R(b), where c3 and c4 are complex numbers.

[0036] A particular method for selecting between MS decoding and dual-mono decoding for a specific frame may include: generating an intermediate signal and a side signal, calculating the energy of the intermediate signal and the side signal, and determining whether to perform MS decoding based on the energy. For example, MS decoding may be performed in response to determining that the ratio of the energy of the side signal to the energy of the intermediate signal is less than a threshold. For illustration, for a spoken audio frame, if the right channel is shifted at least for a first time (e.g., about 0.001 seconds or 48 samples at 48 kHz), then the first energy of the intermediate signal (corresponding to the sum of the left and right signals) may be equivalent to the second energy of the side signal (corresponding to the difference between the left and right signals). When the first energy is equivalent to the second energy, a higher number of bits can be used to encode the side channel, thereby reducing the decoding efficiency of MS decoding relative to dual-mono decoding. Therefore, when the first energy is equivalent to the second energy (e.g., when the ratio of the first energy to the second energy is greater than or equal to a threshold), dual-mono decoding can be used. In alternative methods, a decision can be made on whether to use MS decoding or dual-mono decoding for a specific frame based on a comparison of a threshold with the normalized cross-correlation values ​​of the left and right channels.

[0037] In some instances, the encoder may determine a mismatch value indicating the amount of time mismatch between a first audio signal and a second audio signal. As used herein, the terms "time shift value," "shift value," and "mismatch value" are used interchangeably. For example, the encoder may determine a time shift value indicating a shift (e.g., time mismatch) of the first audio signal relative to the second audio signal. The mismatch value may correspond to the amount of time mismatch between the reception of the first audio signal at a first microphone and the reception of the second audio signal at a second microphone. Furthermore, the encoder may determine the mismatch value on a frame-by-frame basis (e.g., based on every 20 milliseconds (ms) of voice / audio frames). For example, the mismatch value may correspond to the amount of time delay of a second frame of the second audio signal relative to a first frame of the first audio signal. Alternatively, the mismatch value may correspond to the amount of time delay of a first frame of the first audio signal relative to a second frame of the second audio signal.

[0038] When the sound source is closer to the first microphone than the second microphone, the frames of the second audio signal may be delayed relative to the frames of the first audio signal. In this case, the first audio signal may be referred to as the "reference audio signal" or "reference channel," and the delayed second audio signal may be referred to as the "target audio signal" or "target channel." Alternatively, when the sound source is closer to the second microphone than the first microphone, the frames of the first audio signal may be delayed relative to the frames of the second audio signal. In this case, the second audio signal may be referred to as the reference audio signal or reference channel, and the delayed first audio signal may be referred to as the target audio signal or target channel.

[0039] Depending on where the sound source (e.g., a speaker) is located in the conference room or telepresence room, or how the location of the sound source (e.g., a speaker) changes relative to the microphone, the reference channel and target channel can change between frames; similarly, the time mismatch value can also change between frames. However, in some implementations, the time mismatch value can always be positive to indicate the amount of delay of the "target" channel relative to the "reference" channel. Furthermore, the time mismatch value can be used to determine a "non-causal shift" value (referred to herein as the "shift value"), the delayed target channel being "pulled back" in time to the shift value, such that the target channel is aligned (e.g., maximized aligned) with the "reference" channel. A downmixing algorithm to determine the center and side channels can be performed on the reference channel and the non-causal shifted target channel.

[0040] The encoder can determine the time mismatch value based on a reference audio channel and multiple time mismatch values ​​applied to the target audio channel. For example, the first frame X of the reference audio channel can be received at a first time (m1). The first specific frame Y of the target audio channel can be received at a second time (n1) corresponding to the first time mismatch value (e.g., mismatch1 = n1 - m1). Additionally, the second frame of the reference audio channel can be received at a third time (m2). The second specific frame of the target audio channel can be received at a fourth time (n2) corresponding to the second time mismatch value (e.g., mismatch2 = n2 - m2).

[0041] The device can perform a framing or buffering algorithm at a first sampling rate (e.g., a 32kHz sampling rate (i.e., 640 samples per frame)) to generate frames (e.g., 20ms samples). In response to determining that the first frame of the first audio signal and the second frame of the second audio signal arrive at the device simultaneously, the encoder can estimate a shift value (e.g., shift1) equal to zero samples. The left channel (e.g., corresponding to the first audio signal) and the right channel (e.g., corresponding to the second audio signal) can be aligned in time. In some situations, even when aligned, the left and right channels may still differ in energy due to various reasons (e.g., microphone calibration).

[0042] In some instances, the left and right channels may be temporally misaligned for various reasons (e.g., the sound source (e.g., a speaker) may be closer to one microphone than the other, and the distance between the two microphones may be greater than a threshold (e.g., 1 to 20 cm)). The position of the sound source relative to the microphones can introduce different delays in the first and second channels. Additionally, there may be gain differences, energy differences, or level differences between the first and second channels.

[0043] In some instances where there are more than two channels, the reference channel is initially selected based on the channel's level or energy, and subsequently refined based on the time mismatch values ​​between different channel pairs (e.g., t1(ref,ch2), t2(ref,ch3), t3(ref,ch4), ..., t3(ref,chN)), where ch1 is the initial reference channel and t1(.), t2(.), etc., are functions of the estimated mismatch values. If all time mismatch values ​​are positive, then ch1 is considered the reference channel. Alternatively, if any of the mismatch values ​​is negative, then the reference channel is reconfigured to the channel associated with the mismatch value that produced the negative value, and the above process continues until the optimal selection of the reference channel is achieved (i.e., based on maximizing the decorrelation of the maximum number of side channels). Hysteresis can be used to overcome any abrupt changes in the selection of the reference channel.

[0044] In some instances, when multiple speakers speak alternately (e.g., without overlap), the arrival times of audio signals from multiple sound sources (e.g., speakers) at the microphone can vary. In this situation, the encoder can dynamically adjust the time mismatch value based on the speaker to identify the reference channel. In other instances, multiple speakers may speak simultaneously, depending on which speaker is loudest, closest to the microphone, etc., which can lead to varying time mismatch values. In this situation, the identification of the reference and target channels can be based on the varying time shift values ​​in the current frame and the estimated time mismatch values ​​in previous frames, and on the energy or time evolution of the first and second audio signals.

[0045] In some instances, when the first and second audio signals potentially exhibit little (e.g., no) correlation, the two signals may be synthesized or artificially generated. It should be understood that the examples described herein are illustrative and may be instructive in determining the relationship between the first and second audio signals in similar or different contexts.

[0046] The encoder can generate comparison values ​​(e.g., difference or cross-correlation values) based on comparisons between a first frame of the first audio signal and multiple frames of the second audio signal. Each of the multiple frames can correspond to a specific time mismatch value. The encoder can generate a first estimated shift value based on the comparison values. For example, the first estimated shift value can correspond to a comparison value indicating a higher time similarity (or lower difference) between the first frame of the first audio signal and a corresponding first frame of the second audio signal.

[0047] The encoder can determine the final shift value by refining a series of estimated shift values ​​in multiple stages. For example, the encoder can first estimate a "provisional" shift value based on comparison values ​​generated from stereo preprocessed and resampled versions of the first and second audio signals. The encoder can generate interpolated comparison values ​​associated with shift values ​​close to the estimated "provisional" shift value. The encoder can determine a second estimated "interpolated" shift value based on the interpolated comparison values. For example, the second estimated "interpolated" shift value can correspond to a specific interpolated comparison value that indicates higher temporal similarity (or smaller difference) compared to the remaining interpolated comparison values ​​and the first estimated "provisional" shift value. If the second estimated "interpolated" shift value of the current frame (e.g., the first frame of the first audio signal) differs from the final shift value of a previous frame (e.g., a frame of the first audio signal preceding the first frame), then the "interpolated" shift value of the current frame is further "refined" to improve the temporal similarity between the first audio signal and the shifted second audio signal. Specifically, by searching around a second estimated “interpolated” shift value for the current frame and a final estimated shift value for the previous frame, a third estimated “corrected” shift value corresponds to a more accurate measure of temporal similarity. The third estimated “corrected” shift value is further tuned to estimate the final shift value by limiting any spurious changes in shift values ​​between frames, and is further controlled to prevent switching from negative shift values ​​to positive shift values ​​(or vice versa) in two successive (or consecutive) frames as described herein.

[0048] In some instances, the encoder may avoid switching between positive and negative shift values ​​in consecutive or adjacent frames, or vice versa. For example, the encoder may set the final shift value to a specific value (e.g., 0) indicating no time shift, based on an estimated "interpolated" or "corrected" shift value for the first frame and a corresponding estimated "interpolated" or "corrected" or final shift value in a specific frame preceding the first frame. For illustration, in response to determining that one of the estimated "provisional" or "interpolated" or "corrected" shift values ​​for the current frame (e.g., the first frame) is positive and the other of the estimated "provisional" or "interpolated" or "corrected" or "final" shift values ​​for a previous frame (e.g., a frame preceding the first frame) is negative, the encoder may set the final shift value for the current frame to indicate no time shift, i.e., shift1 = 0. Alternatively, in response to determining that one of the estimated "provisional", "interpolated", or "corrected" shift values ​​for the current frame (e.g., the first frame) is negative and the other of the estimated "provisional", "interpolated", "corrected", or "final" shift values ​​for a previous frame (e.g., a frame preceding the first frame) is positive, the encoder may also set the final shift value for the current frame to indicate no time shift, i.e., shift1 = 0.

[0049] The encoder can select a frame of either a first audio signal or a second audio signal as a "reference" or a "target" based on a shift value. For example, in response to determining that the final shift value is positive, the encoder can generate a reference channel or signal indicator with a first value (e.g., 0) indicating that the first audio signal is a "reference" signal and the second audio signal is a "target" signal. Alternatively, in response to determining that the final shift value is negative, the encoder can generate a reference channel or signal indicator with a second value (e.g., 1) indicating that the second audio signal is a "reference" signal and the first audio signal is a "target" signal.

[0050] The encoder can estimate the relative gain (e.g., relative gain parameter) associated with a reference signal and a non-causal shifted target signal. For example, in response to determining that the final shift value is positive, the encoder can estimate a gain value to normalize or equalize the energy or power level of the first audio signal relative to a second audio signal offset by a non-causal shift value (e.g., the absolute value of the final shift value). Alternatively, in response to determining that the final shift value is negative, the encoder can estimate a gain value to normalize or equalize the power or amplitude level of the non-causal shifted first audio signal relative to the second audio signal. In some instances, the encoder can estimate a gain value to normalize or equalize the amplitude or power level of the "reference" signal relative to the non-causal shifted "target" signal. In other instances, the encoder can estimate a gain value (e.g., a relative gain value) based on the reference signal relative to the target signal (e.g., the unshifted target signal).

[0051] The encoder can generate at least one encoded signal (e.g., a center channel signal, a side channel signal, or both) based on a reference signal, a target signal, a non-causal shift value, and a relative gain parameter. In other embodiments, the encoder can generate at least one encoded signal (e.g., a center channel, a side channel, or both) based on a reference channel and a time mismatch adjusted target channel. The side channel signal can correspond to the difference between a first sample of a first frame of a first audio signal and a selected sample of a selected frame of a second audio signal. The encoder can select the selected frame based on the final shift value. Because the difference between the first sample and the selected sample is reduced compared to other samples of the second audio signal corresponding to a frame of the second audio signal received by the device simultaneously with the first frame, fewer bits are available to encode the side channel signal. The transmitter of the device can transmit at least one encoded signal, a non-causal shift value, a relative gain parameter, a reference channel or signal indicator, or a combination thereof.

[0052] The encoder can generate at least one encoded signal (e.g., an intermediate signal, a side signal, or both) based on a reference signal, a target signal, a non-causal shift value, a relative gain parameter, low-frequency band parameters of a specific frame of a first audio signal, high-frequency band parameters of a specific frame, or a combination thereof. The specific frame may precede the first frame. Certain low-frequency band parameters, high-frequency band parameters, or combinations thereof from one or more previous frames can be used to encode the intermediate signal, side signal, or both of the first frame. Encoding the intermediate signal, side signal, or both based on low-frequency band parameters, high-frequency band parameters, or combinations thereof may include estimates of non-causal shift values ​​and inter-channel relative gain parameters. Low-frequency band parameters, high-frequency band parameters, or combinations thereof may include pitch parameters, phonation parameters, decoder type parameters, low-frequency band energy parameters, high-frequency band energy parameters, tilt parameters, pitch gain parameters, FCB gain parameters, decoding mode parameters, speech activity parameters, noise estimation parameters, signal-to-noise ratio parameters, formant shaping parameters, voice / music decision parameters, non-causal shift, inter-channel gain parameters, or combinations thereof. The transmitter of the device can transmit at least one encoded signal, a non-causal shift value, a relative gain parameter, a reference channel (or signal) indicator, or a combination thereof. In this invention, terms such as "determine," "calculate," "shift," and "adjust" are used to describe how one or more operations are performed. It should be noted that these terms should not be considered limiting and that other techniques can be used to perform similar operations.

[0053] This invention discloses systems and apparatuses operable to modify or decode residual channel signals (e.g., side channel (or signal) or error channel (or signal)). For example, residual channels can be modified or encoded based on time misalignment or mismatch values ​​between target and reference channels to reduce interharmonic noise introduced by windowing effects in signal-adaptive "flexible" stereo decoders. Signal-adaptive "flexible" stereo decoders can transform one or more time-domain signals (e.g., reference channel and adjusted target channel) into frequency-domain signals. Window mismatch in analysis-synthesis can lead to significant interharmonic noise or spectral leakage in the estimated side channels during downmixing.

[0054] Some encoders improve the timing alignment of two channels by shifting both channels. For example, the first channel may be causally shifted by half the mismatch, and the second channel may be noncausally shifted by half the mismatch, thus causing the two channels to be aligned in time. However, the proposed system uses only the noncausative shift of one channel to improve the timing alignment of the channels. For example, the target channel (e.g., the hysteresis channel) may be noncausally shifted to align the reference channel with the target channel. Since only the target channel is shifted to align the channels in time, the target channel is shifted by a larger amount compared to the amount shifted in cases where both causative and noncausative shifts are used to align the channels. When one channel (i.e., the target channel) is the only channel shifted based on a determined mismatch value, the middle and side channels (obtained by downmixing the first and second channels) will show an increase in interharmonic noise or spectral leakage. When the window rotation (e.g., the amount of non-causal shift) is relatively large (e.g., greater than 1 to 2 ms), this interharmonic noise (e.g., artifacts) is more pronounced in the side channels.

[0055] Target channel shifting can be performed in the time or frequency domain. If the target channel is shifted in the time domain, the shifted target channel and reference channel undergo DFT analysis using an analysis window to transform them to the frequency domain. Alternatively, if the target channel is shifted in the frequency domain, the target channel (before shifting) and reference channel undergo DFT analysis using an analysis window to transform them to the frequency domain, and the target channel is shifted (using a phase rotation operation) after the DFT analysis. In either case, after shifting and DFT analysis, the frequency domain versions of the shifted target and reference channels are downmixed to produce the center and side channels. In some implementations, an error channel may be generated. The error channel indicates the difference between the side channel and the estimated side channel determined based on the center channel. The term "residual channel" is used herein to refer to either the side channel or the error channel. Subsequently, DFT synthesis is performed using a synthesis window to transform the signal to be transmitted (e.g., the middle channel and residual channel) back into the time domain.

[0056] To avoid introducing artifacts, the synthesis window should match the analysis window. However, when the time alignment of the target and reference channels is large, using only a non-causal shift of the target channel to align the target and reference channels can cause a large mismatch between the synthesis and analysis windows of the target channel corresponding to the residual channel. Artifacts introduced by this window mismatch are prevalent in the residual channel.

[0057] The residual channel can be modified to reduce these artifacts. In one instance, the residual channel can be attenuated (e.g., by applying gain to the side channels or by applying gain to the error channels) before the bitstream is generated for transmission. The residual channel can be fully attenuated (e.g., zeroed out) or only partially attenuated. As another instance, the number of bits in the bitstream used to encode the residual channel can be modified. For example, when the time misalignment between the target channel and the reference channel is small (e.g., below a threshold), a first number of bits can be allocated for transmitting the residual channel information. However, when the time misalignment between the target channel and the reference channel is large (e.g., above a threshold), a second number of bits can be allocated for transmitting the residual channel information, where the second number is less than the first number.

[0058] See Figure 1 This describes a specific illustrative instance of the system and designates it as 100 in its entirety. System 100 includes a first device 104 communicatively coupled to a second device 106 via a network 120. The network 120 may include one or more wireless networks, one or more wired networks, or a combination thereof.

[0059] The first device 104 may include an encoder 114, a transmitter 110, and one or more input interfaces 112. At least one of the input interfaces 112 may be coupled to a first microphone 146, and at least one other input interface 112 may be coupled to a second microphone 148. The encoder 114 may include a transformation unit 202, a transformation unit 204, a stereo channel adjustment unit 206, a downmixer 208, a residual generation unit 210, a residual scaling unit 212 (e.g., a residual channel modifier), a center channel encoder 214, a residual channel encoder 216, and a signal-adaptive "flexible" stereo decoder 109. The signal-adaptive "flexible" stereo decoder 109 may include a time-domain (TD) decoder, a frequency-domain (FD) decoder, or a modified discrete cosine transform (MDCT) domain decoder. The residual signal or error signal modification described herein may be applied to each stereo downmixing mode (e.g., TD downmixing mode, FD downmixing mode, or MDCT downmixing mode). The first device 104 may also include a memory 153 configured to store analytical data.

[0060] The second device 106 may include a decoder 118. The decoder 118 may include a time equalizer 124 and a frequency domain stereo decoder 125. The second device 106 may be coupled to a first loudspeaker 142, a second loudspeaker 144, or both.

[0061] During operation, the first device 104 may receive a reference channel 220 (e.g., a first audio signal) from a first microphone 146 via a first input interface and a target channel 222 (e.g., a second audio signal) from a second microphone 148 via a second input interface. The reference channel 220 may correspond to a channel that is time-advanced (e.g., a leading channel), and the target channel 222 may correspond to a channel that is time-delayed (e.g., a lagging channel). For example, a sound source 152 (e.g., a user, speaker, ambient noise, musical instrument, etc.) may be closer to the first microphone 146 than to the second microphone 148. Therefore, the audio signal from the sound source 152 may be received at the input interface 112 via the first microphone 146 at an earlier time than via the second microphone 148. This natural delay in multi-channel signal acquisition via multiple microphones can introduce a time misalignment between the first audio channel 130 and the second audio channel 132. The reference channel 220 can be either the right channel or the left channel, and the target channel 222 can be either the right channel or the left channel.

[0062] Such as about Figure 2 As described in more detail, the target channel 222 can be adjusted (e.g., shifted in time) to be substantially aligned with the reference channel 220. According to one embodiment, the reference channel 220 and the target channel 222 can be varied on a frame-by-frame basis.

[0063] See Figure 2 This demonstrates an example of encoder 114A. Encoder 114A can correspond to... Figure 1 The encoder 114a includes a conversion unit 202, a conversion unit 204, a stereo channel adjustment unit 206, a downmixer 208, a residual generation unit 210, a residual scaling unit 212, a center channel encoder 214, and a residual channel encoder 216.

[0064] A reference channel 220 captured by the first microphone 146 is provided to the transformation unit 202. The transformation unit 202 is configured to perform a first transformation operation on the reference channel 220 to generate a frequency domain reference channel 224. For example, the first transformation operation may include one or more Discrete Fourier Transform (DFT) operations, Fast Fourier Transform (FFT) operations, Modified Discrete Cosine Transform (MDCT) operations, etc. According to some embodiments, a Quadrature Mirror Filter Bank (QMF) operation (using a filter bank, such as a composite low-delay filter bank) can be used to split the reference channel 220 into multiple sub-bands. The frequency domain reference channel 224 is provided to the stereo channel adjustment unit 206.

[0065] The target channel 222, captured by the second microphone 148, is provided to the transformation unit 204. The transformation unit 204 is configured to perform a second transformation operation on the target channel 222 to generate a frequency domain target channel 226. For example, the second transformation operation may include a DFT operation, an FFT operation, an MDCT operation, etc. According to some embodiments, a QMF operation may be used to split the target channel 222 into multiple sub-bands. The frequency domain target channel 226 is also provided to the stereo channel adjustment unit 206.

[0066] In some alternative implementations, additional processing steps may be performed on the reference and target channels captured by the microphone prior to the transformation operation. For example, in one implementation, the channels may be shifted in the time domain (e.g., causally, non-causally, or both) to align with each other based on a mismatch value estimated in a previous frame. The transformation operation is then performed on the shifted channels.

[0067] Stereo channel adjustment unit 206 is configured to determine an inter-channel mismatch value 228 indicating a time misalignment between the frequency domain reference channel 224 and the frequency domain target channel 226. Therefore, the inter-channel mismatch value 228 can be an inter-channel time difference (ITD) parameter indicating how much the target channel 222 lags behind the reference channel 220 (in the frequency domain). Stereo channel adjustment unit 206 is further configured to adjust the frequency domain target channel 226 based on the inter-channel mismatch value 228 to produce an adjusted frequency domain target channel 230. For example, stereo channel adjustment unit 206 may shift the frequency domain target channel 226 to the inter-channel mismatch value 228 to produce an adjusted frequency domain target channel 230 that is time-synchronized with the frequency domain reference channel 224. The frequency domain reference channel 224 is passed to demixer 208, and the adjusted frequency domain target channel 230 is provided to demixer 208. The inter-channel mismatch value 228 is provided to the residual scaling unit 212.

[0068] Demixer 208 is configured to perform demixing operations on frequency domain reference channel 224 and adjusted frequency domain target channel 230 to produce center channel 232 and side channels 234. Center channel (M... fr (b))232 can be a frequency domain reference channel (L) fr (b))224 and the adjusted frequency domain target channel (R) fr (b))230 functions. For example, the center channel (M fr (b))232 can be expressed as M fr (b)=(L fr (b)+R fr (b)) / 2. According to another implementation, the center channel (M) fr (b))232 can be expressed as M fr (b)=c1(b)*Lfr (b)+c2*R fr (b), where c1(b) and c2(b) are complex values. In some embodiments, the complex values ​​c1(b) and c2(b) are based on stereo parameters (e.g., inter-channel phase difference (IPD) parameters). For example, in one embodiment, c1(b) = (cos(-γ) - i*sin(-γ)) / 2 0.5 And c2(b)=(cos(IPD(b)-γ)+i*sin(IPD(b)-γ)) / 2 0.5 , where i is an imaginary number representing the square root of -1. The middle channel 232 is provided to the residual generation unit 210 and the middle channel encoder 214.

[0069] Side channel (S fr (b))234 can also be a frequency domain reference channel (L fr (b))224 and the adjusted frequency domain target channel (R) fr (b))230 functions. For example, the lateral channel (S fr (b))234 can be expressed as S fr (b)=(L fr (b)-R fr (b)) / 2. According to another embodiment, the side channel (S fr (b))234 can be expressed as S fr (b)=(L fr (b)-c(b)*R fr (b)) / (1+c(b)), where c(b) can be a function of the inter-channel level difference (ILD(b)) or (ILD(b)) (e.g., c(b) = 10^(ILD(b) / 20)). The side channel 234 is provided to the residual generation unit 210 and the residual scaling unit 212. In some embodiments, the side channel 234 is provided to the residual channel encoder 216. In some embodiments, the residual channel is the same as the side channel.

[0070] The residual generation unit 210 is configured to generate a predicted side channel 236 based on the center channel 232. The predicted side channel 236 corresponds to the prediction of the side channel 234. For example, the predicted side channel... 236 can be expressed as Where g is the predicted residual gain for each parameter band operation and is a function of ILD. The residual generation unit 210 is further configured to generate a residual channel 238 based on the side channel 234 and the predicted side channel 236. For example, the residual channel (e) 238 can be expressed as... The error signal. According to some embodiments, the predicted side channel 236 may be equal to zero (or may not be estimated) in certain frequency bands. Therefore, in some situations (or frequency bands), the residual channel 238 is the same as the side channel 234. The residual channel 238 is provided to the residual scaling unit 212. According to some embodiments, the downmixer 208 generates the residual channel 238 based on the frequency domain reference channel 224 and the adjusted frequency domain target channel 230.

[0071] If the inter-channel mismatch value 228 between the frequency domain reference channel 224 and the frequency domain target channel 226 meets a threshold (e.g., relatively large), then the analysis window and synthesis window used for DFT parameter estimation can be substantially mismatched. Larger temporal mismatches are more tolerant if one window is causally shifted and the other is non-causally shifted. However, if the frequency domain target channel 226 is a single channel shifted based on the inter-channel mismatch value 228, then the middle channel 232 and side channels 234 can indicate an increase in inter-harmonic noise or spectral leakage. Inter-harmonic noise is more pronounced in the side channels 234 when the window rotation is relatively large (e.g., greater than 2 milliseconds). As a result, the residual scaling unit 212 scales (e.g., attenuates) the residual channel 238 before decoding.

[0072] For illustrative purposes, the residual scaling unit 212 is configured to determine a scaling factor 240 for the residual channel 238 based on the inter-channel mismatch value 228. The larger the inter-channel mismatch value 228, the larger the scaling factor 240 (e.g., the more the residual channel 238 is attenuated). According to one embodiment, the scaling factor (fac_att) 240 is determined using the following pseudocode:

[0073] fac_att = 1.0f;

[0074] if(fabs(hStereoDft->itd[k_offset])>80.0f)

[0075] {

[0076] fac_att=min(1.0f, max(0.2f, 2.6f-0.02f*fabs(hStereoDft->itd[1])));

[0077] }

[0078] pDFT_RES[2*i]*=fac_att;

[0079] pDFT_RES[2*i+1]*=fac_att;

[0080] Therefore, the scaling factor 240 can be determined based on the inter-channel mismatch value 228 (e.g., itd[k_offset]) being greater than a threshold (e.g., 80). The residual scaling unit 212 is further configured to scale the residual channel 238 according to the scaling factor 240 to produce a scaled residual channel 242. Thus, if the inter-channel mismatch value 228 is substantially large, the residual scaling unit 212 attenuates the residual channel 238 (e.g., the error signal) because the side channel 234 indicates significant spectral leakage in some situations. The scaled residual channel 242 is provided to the residual channel encoder 216.

[0081] According to some embodiments, the residual scaling unit 212 is configured to determine residual gain parameters based on the inter-channel mismatch value 228. The residual scaling unit 212 may also be configured to zero out one or more frequency bands of the residual channel 238 based on the inter-channel mismatch value 228. According to one embodiment, the residual scaling unit 212 is configured to zero out (or substantially zero out) each frequency band of the residual channel 238 based on the inter-channel mismatch value 228.

[0082] A center channel encoder 214 is configured to encode center channel 232 to produce an encoded center channel 244. The encoded center channel 244 is provided to a multiplexer (MUX) 218. A residual channel encoder 216 is configured to encode scaled residual channel 242, residual channel 238, or side channel 234 to produce an encoded residual channel 246. The encoded residual channel 246 is provided to the multiplexer 218. The multiplexer 218 can combine the encoded center channel 244 and the encoded residual channel 246 as part of a bitstream 248A. According to one embodiment, bitstream 248A corresponds to... Figure 1 Bit stream 248 (or contained in the bit stream).

[0083] According to one embodiment, the residual channel encoder 216 is configured to set the number of bits in the bit stream 248A used to encode the scaled residual channel 242 based on the inter-channel mismatch value 228. The residual channel encoder 216 can compare the inter-channel mismatch value 228 with a threshold. If the inter-channel mismatch value is less than or equal to the threshold, then a first number of bits are used to encode the scaled residual channel 242. If the inter-channel mismatch value 228 is greater than the threshold, then a second number of bits are used to encode the scaled residual channel 242. The second number of bits is different from the first number of bits. For example, the second number of bits is less than the first number of bits.

[0084] Return to view Figure 1The signal-adaptive "flexible" stereo decoder 109 can transform one or more time-domain channels (e.g., reference channel 220 and target channel 222) into frequency-domain channels (e.g., frequency-domain reference channel 224 and frequency-domain target channel 226). For example, the signal-adaptive "flexible" stereo decoder 109 can perform a first transformation operation on the reference channel 222 to generate the frequency-domain reference channel 224. Additionally, the signal-adaptive "flexible" stereo decoder 109 can perform a second transformation operation on an adjusted version of the target channel 222 (e.g., a target channel 222 shifted in the time domain by an equivalent amount of inter-channel mismatch value 228) to generate an adjusted frequency-domain target channel 230.

[0085] The signal-adaptive "flexible" stereo decoder 109 is further configured to determine, based on a first time-shift operation, whether to perform a second time-shift (e.g., non-causal) operation on the adjusted frequency-domain target channel 230 in the transform domain to produce a modified adjusted frequency-domain target channel (not shown). The modified adjusted frequency-domain target channel may correspond to a target channel 222 shifted to a time mismatch value and a second time-shift value. For example, encoder 114 may shift target channel 222 to a time mismatch value to produce an adjusted version of target channel 222, the signal-adaptive "flexible" stereo decoder 109 may perform a second transform operation on the adjusted version of target channel 122 to produce an adjusted frequency-domain target channel, and the signal-adaptive "flexible" stereo decoder 109 may time-shift the adjusted frequency-domain target channel in the transform domain.

[0086] Frequency domain channels 224 and 226 can be used to estimate stereo parameters 162 (e.g., parameters that represent the spatial properties associated with frequency domain channels 224 and 226). Examples of stereo parameters 162 may include parameters such as: inter-channel intensity difference (IID) parameters (e.g., inter-channel level difference (ILD)), inter-channel time difference (ITD) parameters, IPD parameters, inter-channel correlation (ICC) parameters, non-causal shift parameters, spectral tilt parameters, inter-channel phonation parameters, inter-channel pitch parameters, inter-channel gain parameters, etc. Stereo parameters 162 may also be transmitted as part of bitstream 248.

[0087] For example, regarding Figure 2 In a similar manner as described, the signal-adaptive "flexible" decoder 109 can utilize the mid-frequency channel M... fr The information in (b) and the stereo parameters 162 (e.g., ILD) corresponding to frequency band (b) are obtained from the middle channel M fr (b) Predicting the lateral vocal tract S PRED (b). For example, the predicted sideband S PRED (b) can be expressed as M fr(b)*(ILD(b)-1) / (ILD(b)+1). This can be based on the sideband channel S. fr and the predicted sideband S PRED The error signal (e) is then calculated. For example, the error signal e can be expressed as S. fr -S PRED The error signal (e) can be decoded using time-domain or transform-domain decoding techniques to generate the decoded error signal e. CODED For certain frequency bands, the error signal e can be expressed as the mid-band channel M_PAST from those frequency bands of the previous frame. fr A scaled version. For example, the decoded error signal e CODED It can be expressed as g PRED *M_PAST fr In some implementation schemes, g PRED It can be estimated that eg PRED *M_PAST fr The energy on the human body is reduced (e.g., reduced to a minimum). The M_PAST frames used can be based on the window shape used for analysis / synthesis and can be restricted to using only even-numbered window jumps.

[0088] For example, regarding Figure 2 In a similar manner as described, residual scaling unit 212 can be configured to adjust, modify, or encode residual channels (e.g., side channels or error channels) based on the inter-channel mismatch value 228 between the frequency domain target channel 226 and the frequency domain reference channel 224 to reduce inter-harmonic noise introduced by the windowing effect in DFT stereo coding. In one example, for illustration, residual scaling unit 212 attenuates residual channels (e.g., by applying gain to the side channels or by applying gain to the error channels) before generating the bitstream for transmission. Residual channels can be completely attenuated (e.g., zeroed out) or only partially attenuated.

[0089] As another example, the number of bits used to encode the residual channel in the bitstream can be modified. For instance, when the time misalignment between the target channel and the reference channel is small (e.g., below a threshold), a first number of bits can be allocated to transmit the residual channel information. However, when the time misalignment between the target channel and the reference channel is large (e.g., above a threshold), a second number of bits can be allocated to transmit the residual channel information. The second number is less than the first number.

[0090] Decoder 118 can perform decoding operations based on stereo parameters 162, encoded residual channel 246, and encoded intermediate channel 244. For example, IPD information included in stereo parameters 162 can indicate whether decoder 118 will use IPD parameters. Decoder 118 can generate a first channel and a second channel based on bitstream 248 and determination. For example, frequency domain stereo decoder 125 and time equalizer 124 can perform upmixing to generate a first output channel 126 (e.g., corresponding to reference channel 220), a second output channel 128 (e.g., corresponding to target channel 222), or both. Second device 106 can output the first output channel 126 via first amplifier 142. Second device 106 can output the second output channel 128 via second amplifier 144. In an alternative embodiment, the first output channel 126 and the second output channel 128 can be transmitted as a stereo signal pair to a single output amplifier.

[0091] It should be noted that the residual scaling unit 212 modifies the residual channel 238 estimated by the residual generation unit 210 based on the inter-channel mismatch value 228. The residual channel encoder 216 encodes the scaled residual channel 242 (e.g., the modified residual signal) and transmits it to the decoder via the encoded bitstream 248A. In some embodiments, the residual scaling unit 212 may reside in the decoder, and the operation of the residual scaling unit 212 may be skipped at the encoder. This skipping is possible because the inter-channel mismatch value 228 is available at the decoder (since the inter-channel mismatch value 228 is encoded and transmitted to the decoder as part of the stereo parameter 162). Based on the inter-channel mismatch value 228 available at the decoder, the residual scaling unit residing at the decoder can modify the decoded residual channel.

[0092] about Figures 1 to 2 The described technique can adjust, modify, or encode residual channels (e.g., side channels or error channels) based on time misalignment or mismatch values ​​between target channel 222 and reference channel 220 to reduce interharmonic noise introduced by windowing effects in DFT stereo coding. For example, to reduce the introduction of artifacts that can be caused by windowing effects in DFT stereo coding, residual channels can be attenuated (e.g., gain can be applied), one or more frequency bands of the residual channels can be zeroed, the number of bits used to encode the residual channels can be adjusted, or a combination thereof.

[0093] As an example of attenuation, the attenuation factor that varies with the mismatch value can be expressed by the following equation:

[0094] attenuation_factor = 2.6 - 0.02 * |mismatch value|

[0095] Additionally, the attenuation factor (e.g., attenuation_factor) calculated according to the above equation can be reduced (or saturated) to maintain it within a range. For example, the attenuation factor can be reduced to maintain it within the limits of 0.2 and 1.0.

[0096] See Figure 3 This demonstrates another example of encoder 114B. Encoder 114B can correspond to... Figure 1 Encoder 114. For example, Figure 3 The components described herein can be integrated into the signal-adaptive "flexible" stereo decoder 109. It should also be understood that it can be implemented using hardware (e.g., a dedicated circuit system), software (e.g., instructions executed by a processor), or a combination thereof. Figure 3 The various components shown in the diagram (e.g., transformers, signal generators, encoders, modifiers, etc.)

[0097] The reference channel 220 and the adjusted target channel 322 are provided to the transformation unit 302. The adjusted target channel 322 can be generated by adjusting the target channel 222 in time to an equivalent amount of the inter-channel mismatch value 228 in the time domain. Therefore, the adjusted target channel 322 is substantially aligned with the reference channel 220. The transformation unit 302 can perform a first transformation operation on the reference channel 220 to generate a frequency domain reference channel 224, and the transformation unit 302 can perform a second transformation on the adjusted target channel 322 to generate an adjusted frequency domain target channel 230.

[0098] Therefore, the transform unit 302 can generate frequency domain (or sub-band domain or filtered low-frequency core and high-frequency bandwidth extension) channels. As a non-limiting example, the transform unit 302 can perform DFT operations, FFT operations, MDCT operations, etc. According to some embodiments, quadrature mirror filter bank (QMF) operations (using filter banks, such as composite low-delay filter banks) can be used to split the input channels 220, 322 into multiple sub-bands. The signal adaptive “flexible” stereo decoder 109 is further configured to determine, based on the first time shift operation, whether to perform a second time shift (e.g., non-causal) operation on the adjusted frequency domain target channel 230 in the transform domain to generate a modified adjusted frequency domain target channel. The frequency domain reference channel 224 and the adjusted frequency domain target channel 230 are provided to the stereo parameter estimator 306 and the downmixer 307.

[0099] The stereo parameter estimator 206 can extract (e.g., generate) stereo parameters 162 based on the frequency domain reference channel 224 and the adjusted frequency domain target channel 230. For illustrative purposes, IID(b) can be the energy E of the left channel in frequency band (b). L (b) and the energy E of the right channel in frequency band (b). R(b) is a function. For example, IID(b) can be expressed as 20*log 10 (E L (b) / E R (b)). The IPD estimated and transmitted at the encoder provides an estimate of the phase difference in the frequency domain between the left and right channels in band (b). Stereo parameter 162 may include additional (or alternative) parameters, such as ICC, ITD, etc. Stereo parameter 162 can be transmitted to Figure 1 The second device 106 is provided to the downmixer 207 (e.g., the side channel generator 308) or both. In some embodiments, stereo parameters 162 may optionally be provided to the side channel encoder 310.

[0100] Stereo parameters 162 can be provided to the IPD, ITD adjuster (or modifier) ​​350. In some embodiments, the IPD, ITD adjuster (or modifier) ​​350 may generate a modified IPD' or a modified ITD'. Alternatively or additionally, the IPD, ITD adjuster (or modifier) ​​350 may determine the residual gain (e.g., a residual gain value) to be applied to the residual signal (e.g., a side channel). In some embodiments, the IPD, ITD adjuster (or modifier) ​​350 may also determine the value of an IPD flag. The value of the IPD flag indicates whether the IPD values ​​of one or more frequency bands should be ignored or set to zero. For example, when the IPD flag is asserted, the IPD values ​​of one or more frequency bands may be ignored or set to zero.

[0101] IPD, ITD adjuster (or modifier) ​​350 may provide modified IPD', modified ITD', IPD flag, residual gain, or a combination thereof to downmixer 307 (e.g., side channel generator 308). IPD, ITD adjuster (or modifier) ​​350 may provide ITD, IPD flag, residual gain, or a combination thereof to side channel modifier 330. IPD, ITD adjuster (or modifier) ​​350 may provide ITD, IPD value, IPD flag, or a combination thereof to side channel encoder 310.

[0102] A frequency-domain reference channel 224 and an adjusted frequency-domain target channel 230 can be provided to a demixer 307. The demixer 307 includes a center channel generator 312 and a side channel generator 308. According to some embodiments, stereo parameters 162 can also be provided to the center channel generator 312. The center channel generator 312 can generate a center channel M based on the frequency-domain reference channel 224 and the adjusted frequency-domain target channel 230. fr(b)232. According to some implementation schemes, the middle channel 232 can also be generated based on stereo parameters 162. Some methods for generating the middle channel 232 based on the frequency domain reference channel 224, the adjusted frequency domain target channel 230, and the stereo parameters 162 are as follows, including M fr (b)=(L fr (b)+R fr (b)) / 2 or M fr (b)=c1(b)*L fr (b)+c2*R fr (b), where c1(b) and c2(b) are complex values. In some implementations, the complex values ​​c1(b) and c2(b) are based on the stereo parameter 162. For example, in one implementation of mid-side downmixing, when estimating IPD, c1(b) = (cos(-γ) - i*sin(-γ)) / 2 0.5 And c2(b)=(cos(IPD(b)-γ)+i*sin(IPD(b)-γ)) / 2 0.5 , where i is the imaginary number representing the square root of -1.

[0103] The center channel 232 is provided to the DFT synthesizer 313. The DFT synthesizer 313 provides its output to the center channel encoder 316. For example, the DFT synthesizer 313 can synthesize the center channel 232. The synthesized center channel can be provided to the center channel 316. The center channel encoder 316 can generate the encoded center channel 244 based on the synthesized center channel.

[0104] Side channel generator 308 can generate side channels (S) based on frequency domain reference channel 224 and adjusted frequency domain target channel 230. fr (b) 234. The side channel 234 can be estimated in the frequency domain. In each frequency band, the gain parameter (g) can be different and can be based on the inter-channel level difference (e.g., based on stereo parameter 162). For example, the side channel 234 can be expressed as (L fr (b)-c(b)*R fr (b)) / (1+c(b)), where c(b) can be a function of ILD(b) or ILD(b) (e.g., c(b) = 10^(ILD(b) / 20)). Side channel 234 can be provided to side channel 330. Side channel modifier 330 also receives ITD, IPD flag, residual gain, or a combination thereof from IPD, IPD adjuster 350. Side channel modifier 330 generates modified side channels based on side channel 234, the frequency domain center channel, and one or more of ITD, IPD flag, or residual gain.

[0105] The modified side channel is provided to the DFT synthesizer 332 to generate the synthesized side channel. The synthesized side channel is provided to the side channel encoder 310. The side channel encoder 310 generates the encoded residual channel 246 based on the stereo parameters 162 received from the DFT and the ITD, IPD values ​​or IPD flags received from the IPD, ITD adjuster 350. In some embodiments, the side channel encoder 310 receives a residual decoding enable / disable signal 354 and generates the encoded residual channel 246 based on the residual decoding enable / disable signal 354. For illustrative purposes, when the residual decoding enable / disable signal 354 indicates that residual coding is disabled, the side channel encoder 310 may not generate the encoded side channel 246 for one or more frequency bands.

[0106] Multiplexer 352 is configured to generate bitstream 248B based on coded intermediate channel 244, coded residual channel 246, or both. In some embodiments, multiplexer 352 receives stereo parameter 162 and generates bitstream 248B based on stereo parameter 162. Bitstream 248B may correspond to Figure 1 The bitstream is 248.

[0107] See Figure 4 This demonstrates an example of decoder 118A. Decoder 118A can correspond to... Figure 1 Decoder 118. Bitstream 248 is provided to a multiplexer (DEMUX) 402 of decoder 118A. Bitstream 248 includes stereo parameters 162, an encoded center channel 244, and an encoded residual channel 246. Multiplexer 402 is configured to extract the encoded center channel 244 from bitstream 248 and provide the encoded center channel 244 to center channel decoder 404. Multiplexer 402 is also configured to extract the encoded residual channel 246 and stereo parameters 162 from bitstream 248. The encoded residual channel 246 and stereo parameters 162 are provided to side channel decoder 406.

[0108] The encoded residual channel 246, stereo parameter 162, or both are provided to the IPD, ITD adjuster 468. The IPD, ITD adjuster 468 is configured to generate an IPD flag value (e.g., the encoded residual channel 246 or stereo parameter 162) identified in the bitstream 248. The IPD flag can be provided as shown in [reference]. Figure 3 The described indication. Alternatively, the IPD flag may indicate to the decoder 118A whether to process or ignore the received residual signal information for one or more frequency bands. Based on the IPD flag value (e.g., whether the flag is asserted or not), the IPD, ITD adjuster 468 is configured to adjust the IPD, adjust the ITD, or both.

[0109] The center channel decoder 404 can be configured to decode the encoded center channel 244 to produce the center channel (m CODED (t))450. If the center channel 450 is a time-domain signal, then transform 408 can be applied to the center channel 450 to generate a frequency-domain center channel (M). CODED (b)) 452. The frequency domain center channel 452 can be provided to the upmixer 410. However, if the center channel 450 is a frequency domain signal, then the center channel 450 can be provided directly to the upmixer 410.

[0110] The side channel decoder 406 can generate side channels (S) based on the encoded residual channel 246 and stereo parameters 162. CODED (b) 454. For example, this can be used to address decoding errors in the low and high frequency bands (e). The side channel 454 can be expressed as S PRED (b)+e CODED (b), where S PRED (b) = M CODED (b)*(ILD(b)-1) / (ILD(b)+1). In some implementations, the side channel decoder 406 further generates the side channel 454 based on the IPD flag. Transformation 456 can be applied to the side channel 454 to generate the frequency domain side channel (S CODED (b))455. The frequency domain side channel 455 can also be provided to the upmixer 410.

[0111] Upmixer 410 can perform upmixing operations on center channel 452 and side channels 455. For example, upmixer 410 can generate a first upmixed channel (L) based on center channel 452 and side channels 455. fr )456 and the second-level mixing channel (R fr 458. Therefore, in the described example, the first upmix signal 456 can be a left channel signal, and the second upmix signal 458 can be a right channel signal. The first upmix signal 456 can be expressed as M CODED (b)+S CODED (b) and the second upmixing signal 458 can be expressed as M CODED (b)-S CODED (b)

[0112] A synthesis and windowing operation 457 is performed on the first upmix signal 456 to generate a synthesized first upmix signal 460. The synthesized first upmix signal 460 is provided to the interchannel aligner 464. A synthesis and windowing operation 416 is performed on the second upmix signal 458 to generate a synthesized second upmix signal 466. The synthesized second upmix signal 466 is provided to the interchannel aligner 464. The interchannel aligner 464 can align the synthesized first upmix signal 460 and the synthesized second upmix signal 466 to generate a first output signal 470 and a second output signal 472.

[0113] It should be noted that Figure 2 Encoder 114A, Figure 3 Encoder 114B and Figure 4 The decoder 118A may contain a portion, but not all, of the encoder or decoder architecture. For example, Figure 2 Encoder 114A, Figure 3 Encoder 114B, Figure 4 The decoder 118A or combinations thereof may also include parallel paths for high-frequency band (HB) processing. Alternatively, in some embodiments, time-domain downmixing may be performed at encoders 114A, 114B. Alternatively, time-domain upmixing may follow... Figure 4 The decoder 118A is used to obtain the left and right channels after decoder shift compensation.

[0114] See Figure 5 This demonstrates communication method 500. Method 500 can be derived from... Figure 1 First device 104 Figure 1 Encoder 114, Figure 2 Encoder 114A, Figure 3 The encoder 114B or a combination thereof is used for execution.

[0115] Method 500 includes: at 502, performing a first transform operation on the reference channel at the encoder to generate a frequency domain reference channel. For example, see... Figure 2 The transformation unit 202 performs a first transformation operation on the reference channel 220 to generate a frequency domain reference channel 224. The first transformation operation may include DFT operation, FFT operation, MDCT operation, etc.

[0116] Method 500 further includes: at 504, performing a second transform operation on the target channel to generate a frequency domain target channel. For example, see... Figure 2 The transformation unit 204 performs a second transformation operation on the target channel 222 to generate a frequency domain target channel 226. The second transformation operation may include DFT operation, FFT operation, MDCT operation, etc.

[0117] Method 500 further includes, at 506, determining an inter-channel mismatch value indicating time misalignment between the frequency domain reference channel and the frequency domain target channel. For example, see... Figure 2 The stereo channel adjustment unit 206 determines an inter-channel mismatch value 228 indicating time misalignment between the frequency domain reference channel 224 and the frequency domain target channel 226. Therefore, the inter-channel mismatch value 228 can be an inter-channel time difference (ITD) parameter indicating how much the target channel 222 lags behind the reference channel 220 (in the frequency domain).

[0118] Method 500 further includes: at 508, adjusting the frequency domain target channel based on the inter-channel mismatch value to generate an adjusted frequency domain target channel. For example, see... Figure 2 The stereo channel adjustment unit 206 adjusts the frequency domain target channel 226 based on the inter-channel mismatch value 228 to generate the adjusted frequency domain target channel 230. For illustrative purposes, the stereo channel adjustment unit 206 shifts the frequency domain target channel 226 to the inter-channel mismatch value 228 to generate the adjusted frequency domain target channel 230 that is time-synchronized with the frequency domain reference channel 224.

[0119] Method 500 further includes: at 510, performing a downmixing operation on the frequency domain reference channel and the adjusted frequency domain target channel to produce the center channel and side channels. For example, see... Figure 2 The downmixer 208 performs downmixing on the frequency domain reference channel 224 and the adjusted frequency domain target channel 230 to produce the center channel 232 and the side channels 234. fr (b))232 can be a frequency domain reference channel (L) fr (b))224 and the adjusted frequency domain target channel (R) fr (b))230 functions. For example, the center channel (M fr (b))232 can be expressed as M fr (b)=(L fr (b)+R fr (b)) / 2. Side channels (S fr (b))234 can also be a frequency domain reference channel (L fr (b))224 and the adjusted frequency domain target channel (R) fr (b))230 functions. For example, the lateral channel (S fr (b))234 can be expressed as S fr (b)=(L fr (b)-R fr (b)) / 2.

[0120] Method 500 further includes: at 512, generating predicted side channels based on the middle channel. The predicted side channels correspond to the predictions of the side channels. For example, see Figure 2 The residual generation unit 210 generates a predicted side channel 236 based on the center channel 232. The predicted side channel 236 corresponds to the prediction of the side channel 234. For example, the predicted side channel... 236 can be expressed as Where g is the predicted residual gain for each parameter band operation and is a function of ILD.

[0121] Method 500 further includes: at 514, generating a residual channel based on the side channels and the predicted side channels. For example, see... Figure 2 The residual generation unit 210 generates a residual channel 238 based on the side channel 234 and the predicted side channel 236. For example, the residual channel (e) 238 can be expressed as... Error signal.

[0122] Method 500 also includes: at 516, determining a scaling factor for the residual channel based on the inter-channel mismatch value. For example, see... Figure 2 The residual scaling unit 212 determines the scaling factor 212 for the residual channel 238 based on the inter-channel mismatch value 228. The larger the inter-channel mismatch value 228, the larger the scaling factor 240 (e.g., the more the residual channel 238 is attenuated).

[0123] Method 500 further includes: at 518, scaling the residual channel according to a scaling factor to produce a scaled residual channel. For example, see... Figure 2 The residual scaling unit 212 scales the residual channel 238 according to the scaling factor 240 to produce the scaled residual channel 242. Therefore, if the inter-channel mismatch value 228 is generally large, the residual scaling unit 212 attenuates the residual channel 238 (e.g., the error signal) because the side channel 234 indicates a large amount of spectral leakage.

[0124] Method 500 also includes, at 520, encoding the middle channel and the scaled residual channel as a portion of the bitstream. For example, see... Figure 2 The center channel encoder 214 encodes the center channel 232 to produce the encoded center channel 244, and the residual channel encoder 216 encodes either the scaled residual channel 242 or the side channel 234 to produce the encoded residual channel 246. The multiplexer 218 combines the encoded center channel 244 and the encoded residual channel 246 as part of the bit stream 248A.

[0125] Method 500 may adjust, modify, or encode residual channels (e.g., side channels or error channels) based on time misalignment or mismatch values ​​between target channel 222 and reference channel 220 to reduce interharmonic noise introduced by windowing effects in DFT stereo coding. For example, to reduce the introduction of artifacts that may be caused by windowing effects in DFT stereo coding, residual channels may be attenuated (e.g., gain may be applied), one or more frequency bands of the residual channels may be zeroed, or the number of bits used to encode the residual channels, or a combination thereof, may be adjusted.

[0126] See Figure 6 This is a block diagram illustrating a specific illustrative example of a device 600 (e.g., a wireless communication device). In various embodiments, device 600 may have more than Figure 6 The few or many components are illustrated. In the illustrative embodiment, device 600 may correspond to... Figure 1 First device 104 Figure 1 The second device 106 or a combination thereof. In an illustrative embodiment, device 600 may perform [see reference 106]. Figures 1 to 5 The system and methods described one or more operations.

[0127] In a particular embodiment, device 600 includes a processor 606 (e.g., a central processing unit (CPU)). Device 600 may include one or more additional processors 610 (e.g., one or more digital signal processors (DSPs)). Processor 610 may include a media (e.g., voice and music) codec (CODEC) 608 and an echo canceller 612. Media CODEC 608 may include a decoder 118, an encoder 114, or a combination thereof. Encoder 114 may include a residual generation unit 210 and a residual scaling unit 212.

[0128] Device 600 may include memory 153 and CODEC 634. Although media CODEC 608 is illustrated as a component of processor 610 (e.g., dedicated circuitry and / or executable programmable code), in other embodiments, one or more components of media CODEC 608, such as decoder 118, encoder 114, or combinations thereof, may be included in processor 606, CODEC 634, another processing component, or combinations thereof.

[0129] Device 600 may include a transmitter 110 coupled to antenna 642. Device 600 may include a display 628 coupled to display controller 626. One or more speakers 648 may be coupled to CODEC 634. One or more microphones 646 may be coupled to CODEC 634 via input interface 112. In a particular embodiment, speaker 648 may include Figure 1The first loudspeaker 142, the second loudspeaker 144, or a combination thereof. In a particular embodiment, the microphone 646 may include... Figure 1 The first microphone 146, the second microphone 148, or a combination thereof. The CODEC 634 may include a digital-to-analog converter (DAC) 602 and an analog-to-digital converter (ADC) 604.

[0130] Memory 153 may contain processing units that can be executed by processor 606, processor 610, CODEC 634, device 600, or a combination thereof to perform [see reference]. Figures 1 to 5 Instruction 660, which describes one or more operations.

[0131] One or more components of device 600 may be implemented via dedicated hardware (e.g., a circuit system), by a processor that executes instructions to perform one or more tasks, or a combination thereof. As an example, memory 153 or one or more components of processor 606, processor 610, and / or CODEC 634 may be memory devices, such as random access memory (RAM), magnetoresistive random access memory (MRAM), spin torque transfer MRAM (STT-MRAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, or compact optical disc read-only memory (CD-ROM). The memory device may be included when executed by a computer (e.g., the processor, processor 606, and / or processor 610 in CODEC 634) to cause the computer to perform [see reference]. Figures 1 to 4 Instructions (e.g., instruction 660) describing one or more operations. As an example, one or more components of memory 153 or processor 606, processor 610, and / or CODEC 634 may be non-transitory computer-readable media containing instructions (e.g., instruction 660) that, when executed by a computer (e.g., the processor, processor 606, and / or processor 610 in CODEC 634), cause the computer to perform [see reference]. Figures 1 to 5 The one or more operations described.

[0132] In certain embodiments, device 600 may be included in a system-in-package or system-on-a-chip device (e.g., a mobile station modem (MSM)) 622. In certain embodiments, processor 606, processor 610, display controller 626, memory 153, CODEC 634, and transmitter 110 are included in the system-in-package or system-on-a-chip device 622. In certain embodiments, input devices 630, such as a touchscreen and / or keypad, and power supply 644 are coupled to the system-on-a-chip device 622. Furthermore, in certain embodiments, such as... Figure 6 As illustrated, the display 628, input device 630, speaker 648, microphone 646, antenna 642, and power supply 644 are external to the system-on-a-chip device 622. However, each of the display 628, input device 630, speaker 648, microphone 646, antenna 642, and power supply 644 may be coupled to components of the system-on-a-chip device 622, such as an interface or controller.

[0133] Device 600 may include: wireless telephone, mobile communication device, mobile phone, smartphone, cellular phone, laptop computer, desktop computer, computer, tablet computer, set-top box, personal digital assistant (PDA), display device, television, game console, music player, radio, video player, entertainment unit, communication device, fixed location data unit, personal media player, digital video player, digital video disc (DVD) player, tuner, camera, navigation device, decoder system, encoder system, or any combination thereof.

[0134] In conjunction with the techniques described above, an apparatus includes means for performing a first transformation operation on a reference channel to generate a frequency domain reference channel. For example, the means for performing the first transformation operation may include... Figures 1 to 2 Transformation unit 202, Figure 3 One or more components of encoder 114B, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0135] The device further includes means for performing a second transformation operation on the target channel to generate a frequency domain target channel. For example, the means for performing the second transformation operation may include... Figures 1 to 2 Transformation unit 204 Figure 3 One or more components of encoder 114B, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0136] The device further includes means for determining an inter-channel misalignment value indicating a time misalignment between a frequency domain reference channel and a frequency domain target channel. For example, the means for determining the inter-channel misalignment value may include... Figures 1 to 2 Stereo channel adjustment unit 206, Figure 3 One or more components of encoder 114B, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0137] The device further includes means for adjusting a frequency-domain target channel based on an inter-channel mismatch value to generate an adjusted frequency-domain target channel. For example, the means for adjusting the frequency-domain target channel may include... Figures 1 to 2 Stereo channel adjustment unit 206, Figure 3 One or more components of encoder 114B, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0138] The device further includes means for performing downmixing on a frequency-domain reference channel and an adjusted frequency-domain target channel to generate a center channel and side channels. For example, the means for performing the downmixing operation may include... Figures 1 to 2 downmixer 208 Figure 3 downmixer 307, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0139] The device further includes means for generating predicted side channels based on the center channel. The predicted side channels correspond to predictions of the side channels. For example, the means for generating predicted side channels may include... Figures 1 to 2 Residual generation unit 210, Figure 3 IPD, ITD adjuster or modifier 350, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0140] The device further includes means for generating a residual channel based on the side channels and the predicted side channels. For example, the means for generating the residual channel may include... Figures 1 to 2 Residual generation unit 210, Figure 3 IPD, ITD adjuster or modifier 350, Figure 6Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0141] The device further includes means for determining a scaling factor for the residual channel based on the inter-channel mismatch value. For example, the means for determining the scaling factor may include... Figures 1 to 2 Residual scaling unit 212, Figure 3 IPD, ITD adjuster or modifier 350, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0142] The device further includes means for scaling the residual channels according to a scaling factor to produce scaled residual channels. For example, the means for scaling the residual channels may include... Figures 1 to 2 Residual scaling unit 212, Figure 3 Side channel modifier 330 Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0143] The device also includes means for encoding the middle channel and a portion of the scaled residual channel as a bitstream. For example, the means for encoding may include... Figures 1 to 2 The middle channel encoder 214 Figures 1 to 2 Residual channel encoder 216 Figure 3 The middle channel encoder 316 Figure 3 Side channel encoder 310, Figure 6 Processor 610, Figure 6 Processor 606 Figure 6 CODEC 634, instruction 660 executed by one or more processing units, one or more other modules, devices, components, circuits or combinations thereof.

[0144] In certain embodiments, one or more components of the systems and apparatuses disclosed herein may be integrated into a decoding system or device (e.g., an electronic device, a codec, or a processor therein), an encoding system or device, or both. In other embodiments, one or more components of the systems and apparatuses disclosed herein may be integrated into a wireless telephone, tablet computer, desktop computer, laptop computer, set-top box, music player, video player, entertainment unit, television, game console, navigation device, communication device, personal digital assistant (PDA), fixed location data unit, personal media player, or another type of device.

[0145] See Figure 7 A block diagram depicting a specific illustrative example of a base station 700. In various implementations, compared to... Figure 7 The components illustrated herein may vary; base station 700 may have more or fewer components. In the illustrative example, base station 700 may be based on... Figure 5 Method 500 operation.

[0146] Base station 700 may be part of a wireless communication system. The wireless communication system may include multiple base stations and multiple wireless devices. The wireless communication system may be a Long Term Evolution (LTE) system, a fourth-generation (4G) LTE system, a fifth-generation (5G) system, a Code Division Multiple Access (CDMA) system, a Global System for Mobile Communications (GSM) system, a Wireless Local Area Network (WLAN) system, or some other wireless system. The CDMA system may implement Wideband CDMA (WCDMA), CDMA1X, Evolved Data Optimized (EVDO), Time Division Synchronous CDMA (TD-SCDMA), or some other version of CDMA.

[0147] Wireless devices can also be referred to as user equipment (UE), mobile station, terminal, access terminal, subscriber unit, workstation, etc. Wireless devices may include: cellular phones, smartphones, tablet computers, wireless modems, personal digital assistants (PDAs), handheld devices, laptop computers, smartbooks, netbooks, tablet computers, cordless phones, wireless local loop (WLL) stations, Bluetooth devices, etc. Wireless devices may include or correspond to... Figure 6 Device 600.

[0148] Various functions may be performed by one or more components of base station 700 (and / or in other components not shown), such as sending and receiving messages and data (e.g., audio data). In a particular instance, base station 700 includes processor 706 (e.g., CPU). Base station 700 may include transcoder 710. Transcoder 710 may include audio CODEC 708 (e.g., voice and music CODEC). For example, transcoder 710 may include one or more components (e.g., circuitry) configured to perform operations on audio CODEC 708. As another example, transcoder 710 is configured to execute one or more computer-readable instructions to perform operations on audio CODEC 708. Although audio CODEC 708 is depicted as a component of transcoder 710, in other instances, one or more components of audio CODEC 708 may be included in processor 706, another processing component, or a combination thereof. For example, decoder 118 (e.g., vocoder decoder) may be included in receiver data processor 764. As another example, encoder 114 (e.g., vocoder encoder) may be included in transmission data processor 782.

[0149] Transcoder 710 can be used to transcode messages and data between two or more networks. Transcoder 710 is configured to convert message and audio data from a first format (e.g., digital format) to a second format. For illustration, decoder 118 can decode an encoded signal having the first format, and encoder 114 can encode the decoded signal into an encoded signal having the second format. Alternatively or additionally, transcoder 710 is configured to perform data rate adaptation. For example, transcoder 710 can down-convert or up-convert the data rate without changing the format of the audio data. For illustration, transcoder 710 can down-convert a 64 kbit / s signal to a 16 kbit / s signal. Audio CODEC 708 may include encoder 114 and decoder 118. Decoder 118 may include stereo parameter adjuster 618.

[0150] Base station 700 includes memory 732. Memory 732 (an example of a computer-readable storage device) may contain instructions. These instructions may be executable by processor 706, transcoder 710, or a combination thereof. Figure 5 Method 500 includes one or more instructions. Base station 700 may include multiple transmitters and receivers (e.g., transceivers) coupled to an antenna array, such as a first transceiver 752 and a second transceiver 754. The antenna array may include a first antenna 742 and a second antenna 744. The antenna array is configured to communicate wirelessly with one or more wireless devices, such as... Figure 6The device 600. For example, the second antenna 744 can receive a data stream 714 (e.g., a bit stream) from the wireless device. The data stream 714 may contain messages, data (e.g., encoded voice data) or a combination thereof.

[0151] Base station 700 may include network connection 760, such as a backhaul connection. Network connection 760 is configured to communicate with one or more base stations of a core network or wireless communication network. For example, base station 700 may receive a second data stream (e.g., message or audio data) from the core network via network connection 760. Base station 700 may process the second data stream to generate message or audio data and provide the message or audio data to one or more wireless devices via one or more antennas in an antenna array, or provide it to another base station via network connection 760. In certain embodiments, as an illustrative and non-limiting example, network connection 760 may be a wide area network (WAN) connection. In some embodiments, the core network may include or correspond to a public switched telephone network (PSTN), a packet backbone, or both.

[0152] Base station 700 may include media gateway 770 coupled to network connection 760 and processor 706. Media gateway 770 is configured to convert media streams between different telecommunications technologies. For example, media gateway 770 may convert between different transport protocols, different decoding schemes, or both. For illustrative purposes, as a non-limiting example, media gateway 770 may convert from PCM signals to Real-Time Transport Protocol (RTP) signals. Media gateway 770 may convert data between: packet-switched networks (e.g., Voice over Internet Protocol (VoIP) networks, IP Multimedia Subsystem (IMS), fourth-generation (4G) wireless networks such as LTE, WiMax, and UMB, fifth-generation (5G) wireless networks, etc.), circuit-switched networks (e.g., PSTN), and hybrid networks (e.g., second-generation (2G) wireless networks such as GSM, GPRS, and EDGE, third-generation (3G) wireless networks such as WCDMA, EV-DO, and HSPA, etc.).

[0153] Additionally, media gateway 770 may include a transcoder, such as transcoder 710, configured to transcode data when codecs are incompatible. For example, as an illustrative, non-limiting example, media gateway 770 may perform transcoding between an Adaptive Multiple Rate (AMR) codec and a G.711 codec. Media gateway 770 may include a router and multiple physical interfaces. In some embodiments, media gateway 770 may also include a controller (not shown). In certain embodiments, the media gateway controller may be external to media gateway 770, external to base station 700, or both. The media gateway controller can control and coordinate the operation of multiple media gateways. Media gateway 770 may receive control signals from the media gateway controller and can be used to bridge different transmission technologies and add services to end-user capabilities and connectivity.

[0154] Base station 700 may include demodulator 762 coupled to transceivers 752, 754, receiver data processor 764, and processor 706, and receiver data processor 764 may be coupled to processor 706. Demodulator 762 is configured to demodulate modulated signals received from transceivers 752, 754, and may provide the demodulated data to receiver data processor 764. Receiver data processor 764 is configured to extract message or audio data from the demodulated data and transmit the message or audio data to processor 706.

[0155] Base station 700 may include a transmission data processor 782 and a transmission multiple-input multiple-output (MIMO) processor 784. Transmission data processor 782 may be coupled to processor 706 and transmission MIMO processor 784. Transmission MIMO processor 784 may be coupled to transceivers 752, 754 and processor 706. In some embodiments, transmission MIMO processor 784 may be coupled to media gateway 770. As an illustrative, non-limiting example, transmission data processor 782 is configured to receive message or audio data from processor 706 and decode the message or audio data based on a decoding scheme such as CDMA or Orthogonal Frequency Division Multiplexing (OFDM). Transmission data processor 782 may provide the decoded data to transmission MIMO processor 784.

[0156] CDMA or OFDM technologies can be used to multiplex decoded data with other data, such as pilot data, to produce multiplexed data. The multiplexed data can then be modulated (i.e., symbol-mapped) by the data processor 782 based on a specific modulation scheme (e.g., binary phase shift keying (“BPSK”), quadrature phase shift keying (“QSPK”), M-ary phase shift keying (“M-PSK”), M-ary quadrature amplitude modulation (“M-QAM”), etc.) to produce modulated symbols. In certain embodiments, different modulation schemes can be used to modulate decoded data and other data. The data rate, decoding, and modulation for each data stream can be determined by instructions executed by the processor 706.

[0157] The transport MIMO processor 784 is configured to receive modulation symbols from the transport data processor 782 and can further process the modulation symbols and perform beamforming on the data. For example, the transport MIMO processor 784 can apply beamforming weights to the modulation symbols.

[0158] During operation, the second antenna 744 of base station 700 can receive data stream 714. The second transceiver 754 can receive data stream 714 from the second antenna 744 and can provide data stream 714 to demodulator 762. Demodulator 762 can demodulate the modulated signal of data stream 714 and provide the demodulated data to receiver data processor 764. Receiver data processor 764 can extract audio data from the demodulated data and provide the extracted audio data to processor 706.

[0159] Processor 706 may provide audio data to transcoder 710 for transcoding. Transcoder 710's decoder 118 may decode the audio data from a first format into decoded audio data, and encoder 114 may encode the decoded audio data into a second format. In some embodiments, encoder 114 may encode the audio data using a data rate higher than the data rate received from the wireless device (e.g., upsampling) or a lower data rate (e.g., downsampling). In other embodiments, the audio data may not be transcoded. Although transcoding (e.g., decoding and encoding) is illustrated as being performed by transcoder 710, the transcoding operation (e.g., decoding and encoding) may be performed by multiple components of base station 700. For example, decoding may be performed by receiver data processor 764, and encoding may be performed by transport data processor 782. In other embodiments, processor 706 may provide audio data to media gateway 770 for conversion into another transport protocol, decoding scheme, or both. Media gateway 770 may provide the converted data to another base station or core network via network connection 760.

[0160] Encoded audio data (e.g., transcoded data) generated at encoder 114 can be provided to transmission data processor 782 or network connection 760 via processor 706. Transcoded audio data from transcoder 710 can be provided to transmission data processor 782 for decoding according to a modulation scheme such as OFDM to generate modulation symbols. Transmission data processor 782 can provide the modulation symbols to transmission MIMO processor 784 for further processing and beamforming. Transmission MIMO processor 784 can apply beamforming weights and can provide the modulation symbols to one or more antennas in an antenna array, such as first antenna 742, via first transceiver 752. Therefore, base station 700 can provide transcoded data stream 716, corresponding to data stream 714 received from a wireless device, to another wireless device. Transcoded data stream 716 may have a different encoding format, data rate, or both than data stream 714. In other embodiments, transcoded data stream 716 can be provided to network connection 760 for transmission to another base station or core network.

[0161] It should be noted that the various functions performed by one or more components of the systems and apparatuses disclosed herein are described as being performed by certain components or modules. This division of components and modules is for illustrative purposes only. In alternative embodiments, the functions performed by a particular component or module may be divided among multiple components or modules. Furthermore, in alternative embodiments, two or more components or modules may be integrated into a single component or module. Each component or module may be implemented using hardware (e.g., field-programmable gate array (FPGA) devices, application-specific integrated circuits (ASICs), DSPs, controllers, etc.), software (e.g., instructions executable by a processor), or any combination thereof.

[0162] Those skilled in the art will further appreciate that the various illustrative logic blocks, configurations, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software executed by a processing device such as a hardware processor, or a combination of both. The foregoing has generally described various illustrative components, blocks, configurations, modules, circuits, and steps in terms of functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in varying ways for each specific application, but these implementation decisions should not be construed as departing from the scope of the invention.

[0163] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or a combination of both. The software module may reside in a memory device, such as random access memory (RAM), magnetoresistive random access memory (MRAM), spin torque transfer MRAM (STT-MRAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, or compact optical disc read-only memory (CD-ROM). An exemplary memory device is coupled to a processor, allowing the processor to read information from and write information to the memory device. In an alternative example, the memory device may be integrated with the processor. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or user terminal. In an alternative example, the processor and storage medium may reside as discrete components in a computing device or user terminal.

[0164] A prior description of the disclosed embodiments has been provided to enable those skilled in the art to make or use the disclosed embodiments. Various modifications to these embodiments will readily be apparent to those skilled in the art, and the principles defined herein can be applied to other embodiments without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the embodiments shown herein, but should be accorded the widest scope consistent with the principles and novel features as defined in the appended claims.

Claims

1. An apparatus for audio processing, comprising: A first transformation unit is configured to perform a first transformation operation on a reference channel to generate a frequency domain reference channel; The second transformation unit is configured to perform a second transformation operation on the target channel to generate a frequency domain target channel; A stereo channel adjustment unit configured to perform the following operations: Determine the inter-channel mismatch value that indicates phase misalignment between the frequency domain reference channel and the frequency domain target channel; and The frequency domain target channel is adjusted based on the inter-channel mismatch value to generate an adjusted frequency domain target channel; A downmixer configured to perform downmixing on the frequency domain reference channel and the adjusted frequency domain target channel to produce a middle channel and side channels. The residual generation unit is configured to perform the following operations: Based on the middle channel, a predicted side channel is generated, the predicted side channel corresponding to the prediction of the side channel; as well as The residual channel is generated based on the side channel and the predicted side channel; The residual scaling unit is configured to perform the following operations: The scaling factor for the residual channel is determined based on the inter-channel mismatch value; and The residual channel is scaled according to the scaling factor to produce a scaled residual channel; A middle channel encoder configured to encode the middle channel as a portion of the bitstream; and A residual channel encoder configured to encode the scaled residual channel as a portion of the bitstream.

2. The apparatus according to claim 1, wherein the residual channel includes an error channel signal.

3. The apparatus of claim 1, wherein the residual scaling unit is further configured to determine a residual gain parameter based on the inter-channel mismatch value.

4. The apparatus of claim 1, wherein one or more frequency bands of the residual channel are set to zero based on the inter-channel mismatch value.

5. The apparatus of claim 1, wherein each frequency band of the residual channel is set to zero based on the inter-channel mismatch value.

6. The apparatus of claim 1, wherein the residual channel encoder is further configured to set the number of bits in the bit stream used to encode the residual channel based on the inter-channel mismatch value.

7. The apparatus of claim 1, wherein the residual channel encoder is further configured to compare the inter-channel mismatch value with a threshold.

8. The apparatus of claim 7, wherein if the inter-channel mismatch value is less than or equal to the threshold, then a first number of bits are used to encode the scaled residual channel.

9. The apparatus of claim 8, wherein if the inter-channel mismatch value is greater than the threshold, then a second number of bits are used to encode the scaled residual channel.

10. The apparatus of claim 9, wherein the second number of bits is different from the first number of bits.

11. The apparatus of claim 9, wherein the second number of bits is less than the first number of bits.

12. The apparatus of claim 1, wherein the residual generation unit and the residual scaling unit are integrated into the mobile device.

13. The apparatus according to claim 1, wherein the residual generation unit and the residual scaling unit are integrated into the base station.

14. A communication method, the method comprising: A first transformation operation is performed on the reference channel at the encoder to generate a frequency domain reference channel; Perform a second transformation operation on the target channel to generate the frequency domain target channel; Determine the inter-channel mismatch value that indicates phase misalignment between the frequency domain reference channel and the frequency domain target channel; The frequency domain target channel is adjusted based on the inter-channel mismatch value to generate an adjusted frequency domain target channel; Perform downmixing on the frequency domain reference channel and the adjusted frequency domain target channel to generate the middle channel and side channels; Based on the middle channel, a predicted side channel is generated, the predicted side channel corresponding to the prediction of the side channel; The residual channel is generated based on the side channel and the predicted side channel; The scaling factor for the residual channel is determined based on the inter-channel mismatch value; and The residual channel is scaled according to the scaling factor to produce a scaled residual channel; Encoding the middle channel as part of the bitstream; and The scaled residual channel is encoded as part of the bitstream.

15. The method of claim 14, wherein the residual channel comprises an error channel signal.

16. The method of claim 14, further comprising determining a residual gain parameter based on the inter-channel mismatch value.

17. The method of claim 14, wherein one or more frequency bands of the residual channel are set to zero based on the inter-channel mismatch value.

18. The method of claim 14, wherein each frequency band of the residual channel is set to zero based on the inter-channel mismatch value.

19. The method of claim 14, further comprising setting the number of bits in the bitstream used to encode the residual channel based on the inter-channel mismatch value.

20. The method of claim 14, further comprising comparing the inter-channel mismatch value with a threshold.

21. The method of claim 20, wherein if the inter-channel mismatch value is less than or equal to the threshold, then a first number of bits are used to encode the scaled residual channel.

22. The method of claim 21, wherein if the inter-channel mismatch value is greater than the threshold, then a second number of bits are used to encode the scaled residual channel.

23. The method of claim 22, wherein the second number of bits is different from the first number of bits.

24. The method of claim 14, wherein scaling the residual channel is performed at the mobile device.

25. The method of claim 14, wherein scaling the residual channel is performed at the base station.

26. A non-transitory computer-readable medium comprising, when executed by a processor within an encoder, instructions that cause the processor to perform operations including: Perform a first transformation operation on the reference channel to generate a frequency domain reference channel; Perform a second transformation operation on the target channel to generate the frequency domain target channel; Determine the inter-channel mismatch value that indicates phase misalignment between the frequency domain reference channel and the frequency domain target channel; The frequency domain target channel is adjusted based on the inter-channel mismatch value to generate an adjusted frequency domain target channel; Perform downmixing on the frequency domain reference channel and the adjusted frequency domain target channel to generate the middle channel and side channels; Based on the middle channel, a predicted side channel is generated, the predicted side channel corresponding to the prediction of the side channel; The residual channel is generated based on the side channel and the predicted side channel; The scaling factor for the residual channel is determined based on the inter-channel mismatch value; and The residual channel is scaled according to the scaling factor to produce a scaled residual channel; Encoding the middle channel as part of the bitstream; and The scaled residual channel is encoded as part of the bitstream.

27. The non-transitory computer-readable medium of claim 26, wherein the residual channel includes an error channel signal.

28. An apparatus for audio processing, comprising: A means for performing a first transformation operation on a reference channel to generate a frequency domain reference channel; A means for performing a second transformation operation on a target channel to generate a frequency domain target channel; A means for determining an inter-channel mismatch value indicating phase misalignment between the frequency domain reference channel and the frequency domain target channel; A means for adjusting the frequency domain target channel based on the inter-channel mismatch value to produce an adjusted frequency domain target channel; A means for performing downmixing operation on the frequency domain reference channel and the adjusted frequency domain target channel to generate a middle channel and side channels; A means for generating predicted side channels based on the middle channel, the predicted side channels corresponding to predictions of the side channels; Apparatus for generating a residual channel based on the side channel and the predicted side channel; A means for determining a scaling factor for the residual channel based on the inter-channel mismatch value; and A means for scaling the residual channel according to the scaling factor to produce a scaled residual channel; and A means for encoding the middle channel and the scaled residual channel as a portion of the bitstream.

29. The device of claim 28, wherein the means for scaling the residual channel is integrated into the mobile device.

30. The device of claim 28, wherein the means for scaling the residual channel is integrated into the base station.

Images