Post-processing of binaural signals

Abstract

An audio processing method includes: performing spatial analysis on binaural signals to estimate the level difference and phase difference of binaural filters characterizing the binaural signals; and performing object extraction on the binaural audio signals using the estimated level difference and phase difference to generate left / right principal component signals and left / right residual component signals. The system can perform different processing on the left / right principal components and left / right residual components using different object processing parameters, such as repositioning, equalization, compression, upmixing, channel remapping, or storage, to generate processed binaural signals, thereby providing an improved listening experience. Repositioning can be based on head-tracking sensor data.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 155,471, filed March 2, 2021, and Spanish Patent Application No. P202031265, filed December 17, 2020, both of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to audio processing, and more particularly to the post-processing of binaural audio signals. Background Technology

[0004] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims of this application and are not acknowledged as prior art by virtue of their inclusion in this section.

[0005] Audio source separation typically refers to extracting specific components from an audio mix in order to isolate or manipulate the level, location, or other properties of objects present in the other sound mix. Source separation methods can be based on algebraic derivation, use machine learning, etc. After extraction, some manipulation can be applied, and then the separated components may be mixed with the background audio. Similarly, for stereo or multichannel audio, there are many models about how to isolate or manipulate objects present in the mix at specific spatial locations. These models are based on linear real-valued mixture models, for example, assuming that the object of interest used for extraction or manipulation exists in the mix signal through a linear frequency-independent gain. In other words, for object signal x... i (where i is the object index) and the mixed signal s j The assumed model uses an unknown linear gain g. ij As shown in equation (1):

[0006]

[0007] Binaural audio content (e.g., stereo signals used for playback on headphones) is becoming widely available. Sources of binaural audio include rendered binaural audio and captured binaural audio.

[0008] Rendered binaural audio typically refers to audio generated through computation. For example, object-based audio (such as Dolby Atmos). TMAudio can be rendered for headphones using Head-Related Transfer Functions (HRTFs), which incorporate interaural time difference (ITD) and interaural level difference (ILD), as well as reflections occurring within the ear. If manipulated correctly, the perceived location of objects can be manipulated to be anywhere around the listener. Additionally, room reflections and post-reverberation can be added to create a sense of distance. One product with a binaural renderer for locating sound source objects around the listener is the Dolby Atmos Production Suite. TM (DAPS) system.

[0009] Binaural audio capture typically refers to audio generated by capturing signals from microphones located at the ears. One way to capture binaural audio is to place the microphones at the ears of a headset. Another method has been made possible by the strong growth of the wireless in-ear headphone market; because in-ear headphones may also include microphones (e.g., for making phone calls), capturing binaural audio has become easy for consumers.

[0010] Both rendered and captured binaural audio typically require some form of post-processing. Examples of such post-processing include: reorienting or rotating the scene to compensate for head movement; rebalancing the levels of specific objects relative to the background, such as boosting the level of speech or dialogue, attenuating background sounds and room reverberation; applying equalization or dynamic range adjustment to specific objects within the mix or only in a specific direction (e.g., in front of the listener); and so on. Summary of the Invention

[0011] Existing systems for audio post-processing have many problems. One issue is that many existing decomposition and upmixing processes use linear gain. While linear gain works well for channel-based signals such as stereo audio, it doesn't work well for binaural audio because binaural audio has frequency-dependent level and time differences. An improved upmixing process that works well for binaural audio is needed.

[0012] While methods exist for redirecting or rotating binaural signals, these methods typically operate to perform relative changes resulting from rotations occurring on the entire mix or only on coherent elements. It is necessary to separate the binaural rendered objects from the mix and perform different processing based on different objects.

[0013] The embodiments relate to a method for extracting and processing one or more objects from binaural rendering or binaural capture. The method focuses on: (1) estimating HRTF properties used during rendering or present in the capture; (2) performing source separation based on the estimation of the estimated HRTF properties; and (3) processing one or more of the separated sources.

[0014] According to an embodiment, a computer-implemented audio processing method includes: performing a signal transformation on binaural signals, the signal transformation including transforming the binaural signals from a first signal domain to a second signal domain; and generating a transformed binaural signal, wherein the first signal domain is a time domain and the second signal domain is a frequency domain. The method further includes performing spatial analysis on the transformed binaural signals, wherein performing spatial analysis includes generating estimated rendering parameters, and wherein the estimated rendering parameters include level difference and phase difference. The method further includes extracting an estimated object from the transformed binaural signals using at least a first subset of the estimated rendering parameters, wherein extracting the estimated object includes generating a left principal component signal, a right principal component signal, a left residual component signal, and a right residual component signal. The method further includes performing object processing on the estimated object using at least a second subset of the estimated rendering parameters, wherein performing object processing includes generating a processed signal based on the left principal component signal, the right principal component signal, the left residual component signal, and the right residual component signal.

[0015] Therefore, the listener experience is improved because the system can apply different frequency-related level differences and time differences to binaural signals.

[0016] Generating the processing signals may include: generating a left primary processing signal and a right primary processing signal from the left primary component signal and the right primary component signal using a first set of object processing parameters; and generating a left residual processing signal and a right residual processing signal from the left residual component signal and the right residual component signal using a second set of object processing parameters. The second set of object processing parameters differs from the first set of object processing parameters. In this way, the primary components can be processed in a different manner than the residual components.

[0017] According to another embodiment, an apparatus includes a processor. The processor is configured to control the apparatus to implement one or more of the methods described herein. The apparatus may additionally include details similar to those of the methods described herein.

[0018] According to another embodiment, a non-transitory computer-readable medium stores a computer program that, when executed by a processor, controls means to perform processing including one or more of the methods described herein.

[0019] The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of the various embodiments. Attached Figure Description

[0020] Figure 1 This is a block diagram of the audio processing system 100.

[0021] Figure 2 This is a block diagram of the object processing system 208.

[0022] Figure 3A and Figure 3B The diagram illustrates the object handling system 108 related to re-rendering (see Figure 108). Figure 1 Examples of ).

[0023] Figure 4 This is a block diagram of the object processing system 408.

[0024] Figure 5 This is a block diagram of the object processing system 508.

[0025] Figure 6 This is an apparatus architecture 600 according to embodiments for implementing the features and processes described herein.

[0026] Figure 7 This is a flowchart of audio processing method 700. Detailed Implementation

[0027] This document describes techniques related to audio processing. In the following description, numerous examples and specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure, as defined by the claims, may include some or all of these examples, either alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0028] The following description details various methods, processes, and procedures. While specific steps may be described in a particular order, this order is primarily for convenience and clarity. A particular step may be performed more than once, may occur before or after other steps, even if these steps are described in a different order, and may occur in parallel with other steps. A second step is only necessary if the first step must be completed before the second step can begin. This will be specifically indicated when it is unclear from the context.

[0029] In this document, the terms “and,” “or,” and “and / or” are used. These terms should be understood to have inclusive meanings. For example, “A and B” can at least mean: “both A and B,” or “at least both A and B.” As another example, “A or B” can at least mean: “at least A,” “at least B,” “both A and B,” or “at least both A and B.” As yet another example, “A and / or B” can at least mean: “A and B,” or “A or B.” When XOR is intended, this will be explicitly stated, for example, “either A or B,” or “at most one of A and B,” etc.

[0030] This document describes the various processing functions associated with structures such as blocks, elements, components, and circuits. Typically, these structures can be implemented by a processor controlled by one or more computer programs.

[0031] 1. Binaural after-ear processing system

[0032] As discussed in more detail below, embodiments describe a method for extracting one or more components from binaural mixing and, additionally, for estimating the position or rendering parameters of the one or more components, the parameters (1) being frequency-dependent and (2) including relative time differences. This enables one or more of the following: accurately manipulating the position of one or more objects in binaural reproduction or capture; processing one or more objects in binaural reproduction or capture, wherein the processing depends on the estimated position of each object; and source separation, which includes estimating the position of each source from binaural reproduction or capture.

[0033] Figure 1 This is a block diagram of an audio processing system 100. The audio processing system 100 can be implemented by one or more computer programs executed by one or more processors. The processor can be a component of a device (such as a headset, headphones, mobile phone, laptop computer, etc.) that implements the functionality of the audio processing system 100. The audio processing system 100 includes a signal transformation system 102, a spatial analysis system 104, an object extraction system 106, and an object processing system 108. The audio processing system 100 may include other components and functions not discussed in detail (for brevity). Typically, in the audio processing system 100, the binaural signal is first processed by the signal transformation system 102 using time-frequency transformation. Subsequently, the spatial analysis system 104 estimates rendering parameters (e.g., binaural rendering parameters), which include level differences and time differences applied to one or more objects. These one or more objects are then extracted by the object extraction system 106 and / or processed by the object processing system 108. The following paragraphs provide further details about each component.

[0034] The signal transformation system 102 receives the binaural signal 120, performs signal transformation on the binaural signal 120, and generates the transformed binaural signal 122. The signal transformation includes transforming the binaural signal 120 from a first signal domain to a second signal domain. The first signal domain can be the time domain, and the second signal domain can be the frequency domain. The signal transformation can be one of various time-frequency transformations, including Fourier transforms such as Fast Fourier Transform (FFT) or Discrete Fourier Transform (DFT), Quadrature Mirror Filter (QMF) transform, Complex QMF (CQMF) transform, Hybrid CQMF (HCQMF) transform, etc. The signal transformation may produce a complex-valued signal.

[0035] Typically, signal conversion system 102 provides some time-frequency separation to the binaural signal 120, thereby generating a converted binaural signal 122. For example, signal conversion system 102 can convert blocks or frames of the binaural signal 120, such as blocks of 10 to 100 ms, or even blocks of 20 ms. The converted binaural signal 122 then corresponds to a set of time-frequency tiles for each converted block of the binaural signal 120. The number of time-frequency tiles depends on the number of frequency bands implemented by signal conversion system 102. For example, signal conversion system 102 can be implemented by a filter bank with 10 to 100 frequency bands (such as 20 frequency bands), in which case the converted binaural signal 122 has the same number of time-frequency tiles.

[0036] The spatial analysis system 104 receives the transformed binaural signal 122, performs spatial analysis on the transformed binaural signal 122, and generates multiple estimated rendering parameters 124. Typically, the estimated rendering parameters 124 correspond to parameters of the head-related transfer function (HRTF), head-related impulse response (HRIR), binaural interaural impulse response (BRIR), etc. The estimated rendering parameters 124 include: multiple level differences—parameter h, discussed in more detail below; and multiple phase differences—parameter φ, discussed in more detail below.

[0037] The object extraction system 106 receives the transformed binaural signal 122 and estimated rendering parameters 124, performs object extraction on the transformed binaural signal 122 using the estimated rendering parameters 124, and generates multiple estimated objects 126. Typically, the object extraction system 106 generates one object for each time-frequency slice of the transformed binaural signal 122. For example, for 100 time-frequency slices, the number of estimated objects is 100.

[0038] Each estimated object can be represented as a principal component signal (hereinafter referred to as x) and a residual component signal (hereinafter referred to as d). The principal component signal may include the left principal component signal x. l and the right principal component signal x r The residual component signal may include the left residual component signal d.l and the right residual component signal d r Therefore, for each time-frequency slice, the estimated object 126 includes four component signals.

[0039] Object processing system 108 receives estimated object 126 and estimated rendering parameters 124, performs object processing on estimated object 126 using estimated rendering parameters 124, and generates processing signal 128. The subset of estimated rendering parameters 124 used by object processing system 108 may differ from the subset of estimated rendering parameters used by object extraction system 106. Object processing system 108 may implement multiple different object processing procedures, as further detailed below.

[0040] 2. Spatial analysis and object extraction

[0041] As implemented by the spatial analysis system 104 and the object extraction system 106, the audio processing system 100 can perform multiple calculations as part of performing spatial analysis and object extraction. These calculations may include one or more of HRTF estimation, phase unrolling, object estimation, object separation, and phase alignment.

[0042] 2.1 HRTF Estimation

[0043] In the following text, it is assumed that the signal exists in sub-bands and time frames, using time-frequency transforms (e.g., DFT, CQMF, HCQMF, etc.) that provide complex-valued signals. Within each time-frequency slice, it is assumed that the complex-valued binaural signal pair (l[n], r[n]) (where n is the frequency or time index) can be modeled as shown in equations (2a-2b):

[0044]

[0045]

[0046] Complex phase angle φ l and φ r This indicates the phase shift introduced by the HRTF within the narrow subband; h l and h r This represents the magnitude of the HRTF applied to the principal component signal x; and d l d r These are two unknown residual signals. In most cases, the focus is not on the absolute phase φ of the HRTF. l and φ r Instead, the interaural phase difference (IPD) φ can be used. Applying IPD φ to the right channel signal, the signal model can be expressed by equations (3a-3b):

[0047] l[n]=h l x[n]+d l[n]

[0048] (3a)

[0049] r[n]=h r x[n]e -jφ +d r [n]

[0050] (3b)

[0051] Similarly, perhaps of greatest interest is the estimation of head shadow effects (e.g., interaural level difference ILD), which can therefore be modeled using real-valued head shadow attenuation h, as shown in equations (4a-4b):

[0052] l[n]=x[n]+d l [n]

[0053] (4a)

[0054] r[n]=hx[n]e -jφ +d r [n]

[0055] (4b)

[0056] Assume the expected value of the inner product of the residual signals is zero, as shown in equation (5):

[0057] <d l d r * >=0 (5)

[0059] Furthermore, assume that the expected value of the inner product of signal x and any residual signal is also zero, as shown in equation (6):

[0060] <xd l * >= <xd r * >=0 (6)

[0062] Finally, the two residual signals must have equal energy, as shown in equation (7):

[0063] <d l d l * >= <d r d r * >= <dd * > (7)

[0065] Then, the relative IPD phase angle φ is directly obtained according to equation (8):

[0066] φ=∠ <lr * > (8)

[0068] In other words, the phase difference of each time-frequency slice is calculated as the transformed binaural signal (e.g., Figure 1 The left component l of the transformed binaural signal (122) and the right component r * The phase angle of the inner product.

[0069] Then, the modified right channel signal r is created by applying the relative phase angle, as shown in equation (9):

[0070] r′[n]=r[n]e +jφ =hx[n]+d r [n]e +jφ (9)

[0072] Principal components are estimated from l[n] and r′[n] based on weighted combinations. As shown in equation (10):

[0073]

[0074] In equation (10), the caret or hat symbol ^ represents the estimated value, and the weight w′r can be calculated according to equation (11):

[0075] w′ r =W r e -jφ (11)

[0077] The cost function E can be expressed according to equation (12). x :

[0078] E x′ =||xw l (x+d l )-w′ r (hx+d r e +jφ )|| 2 (12)

[0080] partial derivatives and Setting it to zero yields equation (13a-13b):

[0081]

[0082]

[0083] Then, the equation (14a-14c) can be written:

[0084] <l l * >= <x x * >+ <d d * >

[0085] (14a)

[0086] <r′r′ * >= <x x * h 2 + <d d * >

[0087] (14b)

[0088] <(l+r′)(l+r′) * >= <m m * >= <x x * >(1+h) 2 +2 <d d * >= <x x * >(1+2h+h 2 )+2 <d d * >

[0089] (14c)

[0090] Substituting the values, we get equation (15a-15i):

[0091] <d d * >= <l l * >- <x x * >= <r′r′ * >- <x x * h 2

[0092] (15a)

[0093]

[0094]

[0095] h 2 ( <m m * >- <l l * >- <r′r′ * >)+2h( <l l * >- <r′r′ * >)- <m m * >+ <l l * >+《r′r′ * >=0

[0096] (15d)

[0097] h 2 A+hB+C=0

[0098] (15e)

[0099] A = <m m * >- <l l * >- <r′ r′ * >

[0100] (15f)

[0101] B = 2((ll) * >- <r′r′ * >)

[0102] (15g)

[0103] C = - <m m * >+ <l l * >+《r′r′ * >

[0104] (15h)

[0105] D = B 2 -4AC

[0106] (15i)

[0107] Then, equation (15a-15i) gives the solution for the level difference h existing in the HRTF, as shown in equation (16):

[0108]

[0109] In other words, the level difference for each time-frequency slice is calculated based on a quadratic equation derived from the left component of the transformed binaural signal, the right component of the transformed binaural signal, and the phase difference. An example of the left component of the transformed binaural signal is... Figure 1 The left component of 122, and in expressions A, B, and C, it is determined by variables l and l * The example of the right component of the transformed binaural signal is the right component of 122, and it is represented by variables r′ and r′ in expressions A, B, and C. * The phase difference is represented by the phase difference information of the estimated rendering parameter 124, and is represented by the IPD phase angle φ in equation (8), which is used to calculate r′ according to equation (9).

[0110] As a specific example, spatial analysis system 104 (see...) Figure 1The HRTF can be estimated by operating the transformed binaural signal 122 using equations (1-16), specifically by using equation (8) to generate the IPD phase angle φ and equation (16) to generate the level difference h as part of generating the estimated rendering parameters 124.

[0111] 2.2 Phase Unfolding

[0112] In the previous section, according to equation (8), the estimated IPDφ is always enclosed within the interval of two pi. To accurately determine the location of a given object, phase unwrapping is required. Typically, unwrapping refers to using adjacent frequency bands to determine the most probable location given multiple possible locations indicated by the enclosed IPD. Various strategies can be employed for phase unwrapping: evidence-based unwrapping and model-based unwrapping.

[0113] 2.2.1 Evidence-based development

[0114] For evidence-based phase unrolling, information from adjacent frequency bands can be used to derive the optimal estimate of the unrolled IPD. Assuming an IPD estimate with three adjacent subbands b-1, b, and b+1, denoted as φ... b-1 φ b φ b+1 Then, the expanded phase candidate of frequency band b is given by equation (17).

[0115]

[0116] According to equation (18), each candidate They all have associated ITD

[0117]

[0118] In equation (18), f b This represents the center frequency of frequency band b. It also includes the total energy of the principal components in each frequency band. The estimated value is given by equation (19):

[0119]

[0120] Therefore, the cross-correlation function of frequency band b (expressed as R) can be obtained from equation (20). b (τ) is used for modeling, and the cross-correlation function is used as the principal component x in this frequency band. b ITD τ Functions:

[0121]

[0122] Now, for each expanded IPD candidate, the energy between adjacent frequency bands v can be accumulated, and the maximum value can be taken as an estimate of the majority of the energy in the entire frequency band occupied by a single IPD, as shown in equation (21):

[0123]

[0124] In other words, the system is capable of performing the following operations: estimating the total energy of the left and right principal component signals in each frequency band; calculating the cross-correlation based on each frequency band; and selecting an appropriate phase difference for each frequency band based on the cross-correlation and the energy between adjacent frequency bands.

[0125] 2.2.2 Model-based expansion

[0126] For model-based expansion, given an estimate of the head shadow parameter h according to equation (16), a simple HRTF model (e.g., a spherical head model) can be used to find the value of h in a given frequency band b. The optimal value. In other words, finding the optimal unfolded phase that matches the size of a given head shadow. This unfolding can be performed by calculation given the model and the values ​​of h in each frequency band. In other words, the system selects an appropriate phase difference for a given frequency band from multiple candidate phase differences based on the level difference applied to the head-related transfer function for that frequency band.

[0127] As a concrete example, for the two types of unfolding, spatial analysis system 104 (see...) Figure 1 Phase unrolling can be performed as part of generating estimated rendering parameters 124.

[0128] 2.3 Principal Object Estimation

[0129] Based on equations (15a), (15b) and (16) <x x * >、 <d d * After estimating h, the weight w can be calculated. l w′ r See also equations (10-11). Repeat equations (13a-13b) above as equations (22a-22b):

[0130]

[0131]

[0132] Then, the weight w can be calculated according to equation (23a-23b). l w′b r :

[0133]

[0134]

[0135] As a specific example, spatial analysis system 104 (see...) Figure 1 The main object estimation can be performed by generating weights as part of the generated estimation rendering parameters 124.

[0136] 2.4 Separation of primary and residual objects

[0137] The system can estimate two pairs of binaural signals: one pair for the principal components to be rendered, and the other pair for the residual components. The pair of principal components to be rendered can be expressed by equations (24a-24b):

[0138]

[0139]

[0140] In equation (24a-24b), the signal l x [n] corresponds to the left principal component signal (e.g., Figure 2 (220 in the middle), and signal r x [n] corresponds to the right principal component signal (e.g., Figure 2 (222). Equation (24a-24b) can be represented by the supermatrix M, as shown in equation (25):

[0141]

[0142] residual signal l d [n] and r d [n] can be estimated according to equation (26):

[0143]

[0144] In equation (26), the signal l d [n] corresponds to the left residual component signal (e.g., Figure 2 (224 in the middle), and signal r d [n] corresponds to the right residual component signal (e.g., Figure 2 226 in the middle.

[0145] The perfect reconstruction requires an expression for D, as shown in equation (27):

[0146] D=IM (27)

[0148] In equation (27), I corresponds to the identity matrix.

[0149] As a concrete example, object extraction system 106 (see...) Figure 1 The primary object estimation can be performed as part of generating the estimated object 126. The estimated object 126 can then be, for example, used as component signals 220, 222, 224, and 226 (see...). Figure 2 ) provided to object processing systems (e.g., Figure 1 108 in Figure 2 (e.g., 208 in the middle).

[0150] 2.5 Overall Phase Alignment

[0151] So far, all phase alignments have been applied to the right channel and right channel prediction coefficients, see, for example, equation (9). To obtain a more balanced distribution, one strategy is to align the phases of the extracted principal and residual components with the downmixing m according to equation m = l + r. The phase shift θ to be applied to these two prediction coefficients will then be as shown in equation (28):

[0152]

[0153] Then, the weight equations of equations (10) and (23a-23b) are modified using a phase shift θ to give the signal according to equation (29a-29b). Final prediction coefficients:

[0154] w l，θ =w l e +jθ

[0155] (29a)

[0156] w r，θ =w r e jθ =w′ r e +jφ e +jθ

[0157] (29b)

[0158] This leads to a modification of equation (25), resulting in equation (30):

[0159]

[0160] Therefore, the submix extraction matrix M does not change with θ, but is used for calculation The prediction coefficients do indeed depend on θ, as shown in equation (31):

[0161]

[0162] Finally, for The re-rendering is given by equation (32):

[0163]

[0164] As a specific example, spatial analysis system 104 (see...) Figure 1 The system can perform a portion of the overall phase alignment (which is part of the generation weights) as part of the generation estimated rendering parameters 124, and the object extraction system 106 can perform a portion of the overall phase alignment as part of the generation estimated objects 126.

[0165] 3. Object handling

[0166] As described above, the object processing system 108 can implement several different object processing procedures. These object processing procedures include one or more of the following: repositioning, level adjustment, equalization, dynamic range adjustment, sibilance cancellation, multi-band compression, immersion enhancement, surround enhancement, upmixing, conversion, channel remapping, storage, and archiving. Repositioning typically refers to moving one or more identified objects within a perceived audio scene, for example, by adjusting the HRTF parameters of the left and right component signals in a processed binaural signal. Level adjustment typically refers to adjusting the level of one or more identified objects within a perceived audio scene. Equalization typically refers to adjusting the timbre of one or more identified objects by applying frequency-dependent gain. Dynamic range adjustment typically refers to adjusting the loudness of one or more identified objects to fall within a defined loudness range, for example, adjusting speech volume so that a nearby speaker is not perceived as too loud and a distant speaker is not perceived as too soft. De-essing typically refers to reducing sibilance, such as reducing the listener's perception of harsh consonants like "s," "sh," "x," "ch," "t," and "th." Multi-band compression typically involves applying different loudness adjustments to different frequency bands of one or more identified objects, for example, reducing the loudness and loudness range of noise bands and increasing the loudness of speech bands. Immersion enhancement typically involves adjusting the parameters of one or more identified objects to match other sensory information, such as video signals; for example, matching moving sounds to a moving set of 3D video pixels, or adjusting the dry / wet balance so that the echo corresponds to the perceived visual room size. Surround enhancement typically involves adjusting the position of one or more identified objects to enhance the listener's perception of the sound source's surroundings. Upmixing, conversion, and channel remapping typically refer to changing one type of channel arrangement to another. Upmixing typically refers to increasing the number of channels in an audio signal; for example, upmixing a 2-channel signal (such as binaural audio) to a 12-channel signal (such as 7.1.4 channel surround sound). Conversion typically refers to decreasing the number of channels in an audio signal; for example, converting a 6-channel signal (such as 5.1 channel surround sound) to a 2-channel signal (such as stereo audio). Channel remapping typically refers to operations that include both upmixing and conversion. Storage and archiving typically refer to storing the binaural signal as one or more extractable objects with associated metadata, and a binaural residual signal.

[0167] Various audio processing systems and tools can be used to perform object processing procedures. Examples of such audio processing systems include the Dolby Atmos Production Suite. TM (DAPS) system, Dolby Volume TM System, Dolby MediaEnhance TMSystem, Dolby TM Mobile capture audio processing system, etc.

[0168] The figure below provides further details on object processing in various embodiments of the audio processing system 100.

[0169] Figure 2 This is a block diagram of object processing system 208. Object processing system 208 can be used as object processing system 108 (see [link]). Figure 1 ).

[0170] The object processing system 208 receives the left principal component signal 220, the right principal component signal 222, the left residual component signal 224, the right residual component signal 226, the first set of object processing parameters 230, the second set of object processing parameters 232, and the estimated rendering parameters 124 (see [link]). Figure 1 Component signals 220, 222, 224, and 226 correspond to the estimated object 126 (see [link]). Figure 1 The component signals of ) are estimated. The rendering parameters 124 include those from the spatial analysis system 104 (see Figure 1 The calculated level difference and phase difference.

[0171] Object processing system 208 uses object processing parameters 230 to generate left main processed signal 240 and right main processed signal 242 from left main component signal 220 and right main component signal 222. Object processing system 208 uses object processing parameters 232 to generate left residual processed signal 244 and right residual processed signal 246 from left residual component signal 224 and right residual component signal 226. Processed signals 240, 242, 244, and 246 correspond to processed signal 128 (see...). Figure 1 The object processing system 208 can perform direct feed processing, for example, generating a left (or right) main (or residual) processed signal only from the left (or right) main (or residual) component signal. The object processing system 208 can also perform cross-feed processing, for example, generating a left (or right) main (or residual) processed signal from both the left and right main (or residual) component signals.

[0172] When generating one or more of processing signals 240, 242, 244, and 246, object processing system 208 may use one or more level differences and one or more phase differences from estimated rendering parameters 124, depending on the specific type of processing performed. As an example, repositioning uses at least some (e.g., all) of the level differences and at least some (e.g., all) of the phase differences. As another example, level adjustment uses at least some (e.g., all) of the level differences but not all of the phase differences (e.g., never using any phase differences). As yet another example, repositioning does not use all of the level differences (e.g., never using any level differences) but uses at least some of the phase differences (e.g., low frequencies, such as frequencies below 1.5 kHz). Using only low frequencies is acceptable because inter-channel phase differences above these frequencies do not significantly affect the perceived location of the source, but changing the phase can lead to audible artifacts. Therefore, adjusting only the low-frequency phase differences while keeping the high-frequency phase differences constant may be a better trade-off between audio quality and perceived location.

[0173] Object processing parameters 230 and 232 enable object processing system 208 to process the primary component signals 220 and 222 using one set of parameters and the residual component signals 224 and 226 using another set of parameters. This allows for different processing of the primary and residual components when performing the different object processing procedures discussed above. For example, during repositioning, the primary components can be repositioned as determined by object processing parameter 230, where object processing parameter 232 leaves the residual components unchanged. As another example, during multi-band compression, object processing parameter 230 can be used to compress the bandwidth of the primary components, and different object processing parameters 232 can be used to compress the bandwidth of the residual components.

[0174] The object processing system 208 may include additional components for performing additional processing steps. One such additional component is an inverse transform system. The inverse transform system performs an inverse transform on the processed signals 240, 242, 244, and 246 to generate processed signals in the time domain. The inverse transform is performed by the signal transformation system 102 (see [link to system description]). Figure 1 The reverse process of the transformation performed.

[0175] Another additional component is the time-domain processing system. Some audio processing techniques work well in the time domain, such as delay effects, echo effects, reverb effects, pitch shifting, and timbre modification. Implementing the time-domain processing system after the inverse transform system enables the object processing system 208 to perform time-domain processing on the processed signal to generate a modified time-domain signal.

[0176] The details of object processing system 208 may otherwise be similar to those of object processing system 108.

[0177] Figures 3A to 3B The diagram illustrates the object handling system 108 related to re-rendering (see Figure 108). Figure 1 Examples of ). Figure 3A This is a block diagram of object processing system 308, which can be used as object processing system 108. Object processing system 308 receives left principal component signal 320, right principal component signal 322, left residual component signal 324, right residual component signal 326, and sensor data 330. Component signals 320, 322, 324, and 326 correspond to the estimated object 126 (see [link to relevant documentation]). Figure 1 The component signal of the sensor data 330 corresponds to the data generated by sensors (such as gyroscopes or other types of head tracking sensors) located in devices such as headsets, headphones, in-ear headphones, microphones, etc.

[0178] Object processing system 308 uses sensor data 330 to generate left main processed signal 340 and right main processed signal 342 based on left main component signal 320 and right main component signal 322. Object processing system 308 generates left residual processed signal 344 and right residual processed signal 346 from sensor data 330 without modification. Object processing system 308 can be used in conjunction with object processing system 208 (see...). Figure 2 Similar methods can be used for direct feed processing or cross-feed processing. The object processing system 308 can use binaural translation to generate the main processing signals 340 and 342. In other words, the main component signals 320 and 322 are considered as objects to which binaural translation is applied, and the diffused sound in the residual component signals 324 and 326 remains unchanged.

[0179] Alternatively, object processing system 308 can generate a monoa object from the left principal component signal 320 and the right principal component signal 322, and can perform binaural translation on the monoa object using sensor data 330. Object processing system 308 can generate the monoa object using phase-aligned downmixing.

[0180] Furthermore, as head tracking systems become a common feature in high-end in-ear and over-ear headphones, they can learn the listener's orientation in real time and rotate the scene accordingly, for example, in virtual reality, augmented reality, or other immersive media applications. However, unless object-based rendering is available, the effectiveness and quality of rotation methods are limited in rendered binaural rendering. To address this issue, object extraction system 106 (see...) Figure 1 The principal components are separated and their positions estimated, and the object processing system 308 treats the principal components as objects and applies binaural translation, while not touching the diffuse sound in the residual components. This enables the following applications.

[0181] One application is that the object processing system 308 rotates the audio scene according to the listener's perspective, while keeping the positioning of the objects accurate without compromising the sense of space in the audio scene conveyed by the environment in the remaining objects.

[0182] Another application is that the object processing system 308 compensates for unwanted head rotations that occur when recording with binaural headphones or microphones. Head rotations can be inferred from the position of the principal component. For example, if it is assumed that the principal component should remain stationary, each detected positional change can be compensated for. Head rotations can also be inferred by acquiring head tracking data synchronously with the audio recording.

[0183] Figure 3B It can be used as an object processing system 108 (see Figure 1 A block diagram of the object processing system 358. The object processing system 358 receives a left primary component signal 370, a right primary component signal 372, a left residual component signal 374, a right residual component signal 376, and configuration information 380. Component signals 370, 372, 374, and 376 correspond to the estimated object 126 (see [link to documentation]). Figure 1 The component signal. Configuration information 380 corresponds to the channel layout used for upmixing, conversion, or channel remapping.

[0184] The object processing system 358 uses configuration information 380 to generate a multi-channel output signal 390. The multi-channel output signal 390 corresponds to a specific channel layout as specified in the configuration information 380. For example, when the configuration information 380 specifies upmixing to 5.1 channel surround sound, the object processing system performs upmixing to generate six channels of a 5.1 channel surround sound channel signal from component signals 370, 372, 374, and 376.

[0185] More specifically, preserving the spatial characteristics of binaural recordings presents challenges when playing them through a speaker layout. Typical solutions involve crosstalk cancellation, which is often only effective for a very small listening area in front of the speakers. By separating the primary and residual components and inferring the location of the primary component, the object processing system 358 can treat the primary component as a dynamic object whose associated location changes over time, which can be accurately rendered across various speaker layouts. The object processing system 358 can use a 2- to N-channel upmixer to process the diffusion component to create an immersive, channel-based bed; the dynamic object generated by the primary component and the channel-based bed generated by the residual component together provide an immersive representation of the original binaural recording across any set of speakers. An example system for upmixing to generate diffuse content can be described in the following literature, in which the diffuse content is decorrelated and distributed according to an orthogonal matrix: Mark Vinton, David McGrath, Charles Robinson and Phillip Brown, “Next Generation Surround Decoding and Upmixing for Consumer and Professional Applications”, 57th International Conference: The Future of Audio Entertainment Technology – Film, Television and the Internet (March 2015).

[0186] Compared to many existing systems, the advantage of this time-frequency decomposition is that the retranslation can vary depending on the object, rather than rotating the entire sound field as the head moves. Additionally, in many existing systems, an extra interaural time delay (ITD) is added to the signal, which can result in a greater delay than it would otherwise be. The object processing system 358 helps overcome these problems compared to these existing systems.

[0187] Figure 4 It can be used as an object processing system 108 (see Figure 1 A block diagram of object processing system 408. Object processing system 408 receives left primary component signal 420, right primary component signal 422, left residual component signal 424, right residual component signal 426, and configuration information 430. Component signals 420, 422, 424, and 426 correspond to the estimated object 126 (see [link to diagram]). Figure 1 The component signal. Configuration information 430 corresponds to the configuration settings used for voice improvement processing.

[0188] Object processing system 408 uses configuration information 430 to generate left main processing signal 440 and right main processing signal 442 based on left main component signal 420 and right main component signal 422. Object processing system 408 generates left residual processing signal 444 and right residual processing signal 446 from configuration information 430 without modification. Object processing system 408 can be used in conjunction with object processing system 208 (see...). Figure 2 Similar methods can be used for direct feed processing or cross-feed processing. The object processing system 408 can use manual speech enhancement processing parameters provided by configuration information 430, or the configuration information 430 can correspond to settings for automatic processing performed by a speech enhancement processing system, such as the one described in International Application Publication No. WO 2020 / 014517. In other words, the primary component signals 420 and 422 are considered as objects to which speech enhancement processing is applied, and the diffused sound in the residual component signals 424 and 426 remains unchanged.

[0189] More specifically, binaural recordings of audio content such as podcasts and video logs typically include contextual ambient sounds accompanying the speech, such as crowd noise, natural sounds, and city noise. It is generally desirable to improve speech quality without affecting the background noise, for example, by increasing its level, pitch, and dynamic range. Separating the speech into primary and residual components allows the object processing system 408 to perform independent processing; level adjustments, equalization, sibilance reduction, and dynamic range adjustments can be applied to the primary components based on configuration information 430. After processing, the object processing system 408 reassembles the signals into processed signals 440, 442, 444, and 446 to form an enhanced binaural presentation.

[0190] Figure 5 It can be used as an object processing system 108 (see Figure 1 A block diagram of object processing system 508. Object processing system 508 receives left primary component signal 520, right primary component signal 522, left residual component signal 524, right residual component signal 526, and configuration information 530. Component signals 520, 522, 524, and 526 correspond to the estimated object 126 (see [link to diagram]). Figure 1 The component signal. Configuration information 530 corresponds to the configuration settings used for level adjustment processing.

[0191] Object processing system 508 uses the first set of level adjustment values ​​in configuration information 530 to generate left main processing signal 540 and right main processing signal 542 based on left main component signal 520 and right main component signal 522. Object processing system 508 uses the second set of level adjustment values ​​in configuration information 530 to generate left residual processing signal 540 and right residual processing signal 542 based on left residual component signal 520 and right residual component signal 522. Object processing system 508 can be used in conjunction with object processing system 208 (see...). Figure 2 Similar methods can be used for direct feed processing or cross-feed processing.

[0192] More specifically, recordings made in reverberant environments (such as large indoor spaces, rooms with reflective surfaces, etc.) may contain significant amounts of reverberation, especially when the source of interest is not close to the microphone. Excessive reverberation reduces the intelligibility of the sound source. In binaural recording, reverberation and ambient sounds (e.g., non-localized noise from nature or machinery) are often uncorrelated in the left and right channels and therefore remain primarily in the residual signal after decomposition. This characteristic allows the object processing system 508 to control ambient quantities (e.g., perceived reverberation) in the recording by controlling the relative levels of the principal and residual components and then summing them into a modified binaural signal. The modified binaural signal then has, for example, less residual to enhance intelligibility, or a smaller principal component to enhance perceived immersion.

[0193] The desired balance between the primary and residual components, as set according to configuration information 530, can be manually defined, for example, by controlling a volume controller or a "balance" knob, or it can be automatically obtained based on an analysis of the relative levels of the components and the definition of the desired balance between their levels. In one embodiment, this analysis compares the root mean square (RMS) levels of the primary and residual components throughout the recording. In another embodiment, the analysis is adaptive over time, and the relative levels of the primary and residual signals are adjusted accordingly in a time-varying manner. For speech content, content analysis, such as speech activity detection, can be performed prior to the process to modify the relative balance of the primary and residual components during speech or non-speech portions in different ways.

[0194] 4. Hardware and Software Details

[0195] The following paragraphs describe the various hardware and software details related to the binaural post-processing discussed above.

[0196] Figure 6This is a device architecture 600 according to embodiments for implementing the features and processes described herein. Architecture 600 can be implemented in any electronic device, including but not limited to: desktop computers, consumer audio / video (AV) devices, radio broadcasting equipment, mobile devices (e.g., smartphones, tablets, laptops, wearable devices, etc.). In the example embodiments shown, architecture 600 is for a laptop computer and includes processor(s) 601, peripheral interface 602, audio subsystem 603, speaker 604, microphone 605, sensor(s) 606 (e.g., accelerometer, gyroscope, barometer, magnetometer, camera, etc.), location processor 607 (e.g., GNSS receiver, etc.), wireless communication subsystem 608 (e.g., Wi-Fi, Bluetooth, cellular, etc.), and I / O subsystem(s) 609, which includes a touch controller 610 and other input controllers 611, a touch surface 612, and other input / control devices 613. Other architectures with more or fewer components can also be used to implement the disclosed embodiments.

[0197] Memory interface 414 is coupled to processor 601, peripheral interface 602, and memory 615 (e.g., flash memory, RAM, ROM, etc.). Memory 615 stores computer program instructions and data, including but not limited to: operating system instructions 616, communication instructions 617, GUI instructions 618, sensor processing instructions 619, telephone instructions 620, electronic messaging instructions 621, web browsing instructions 622, audio processing instructions 623, GNSS / navigation instructions 624, and application / data 625. Audio processing instructions 623 include instructions for performing the audio processing described herein.

[0198] According to an embodiment, architecture 600 may correspond to implementing audio processing system 100 (see [link]). Figure 1 Computer systems (such as laptop computers), object processing systems described herein (e.g., Figure 2 208 in Figure 3A 308 in Figure 3B 358 in Figure 4 408 in Figure 5 One or more of the following (e.g., 508, etc.).

[0199] According to embodiments, architecture 600 may correspond to multiple devices; these devices may communicate via wired or wireless connections, such as an IEEE 802.15.1 standard connection. For example, architecture 600 may correspond to a computer system or mobile phone implementing processor(s) 601 and a headset implementing an audio subsystem 603 (e.g., a speaker); one or more of sensors 606, such as a gyroscope or other head-tracking sensors; and so on. As another example, architecture 600 may correspond to an in-ear headphone implementing processor(s) 601 or a mobile phone and an audio subsystem 603 (e.g., a microphone and a speaker).

[0200] Figure 7 This is a flowchart of audio processing method 700. Method 700 can be performed by a... Figure 6 The components of the architecture 600 are used by a device (e.g., a laptop computer, mobile phone, etc.) to perform, for example, by executing one or more computer programs to implement the audio processing system 100 (see Figure 1 ), the object processing system described in this paper (e.g., Figure 2 208 in Figure 3A 308 in Figure 3B 358 in Figure 4 408 in Figure 5 One or more functions of 508, etc.

[0201] At point 702, a signal transformation is performed on the binaural signal. Performing the signal transformation includes transforming the binaural signal from a first signal domain to a second signal domain and generating the transformed binaural signal. The first signal domain can be the time domain, and the second signal domain can be the frequency domain. For example, signal transformation system 102 (see...) Figure 1 It can transform the binaural signal 120 to generate the transformed binaural signal 122.

[0202] At 704, spatial analysis is performed on the transformed binaural signal. Performing spatial analysis includes generating estimated rendering parameters, which include level difference and phase difference. For example, spatial analysis system 104 (see...) Figure 1 Spatial analysis is performed on the transformed binaural signal 122 to generate estimated rendering parameters 124.

[0203] At 706, the estimated object is extracted from the transformed binaural signal using at least a first subset of the estimated rendering parameters. Extracting the estimated object involves generating the left principal component signal, the right principal component signal, the left residual component signal, and the right residual component signal. For example, object extraction system 106 (see...) Figure 1One or more of the estimated rendering parameters 124 can be used to perform object extraction on the transformed binaural signal 122 to generate an estimated object 126. The estimated object 126 may correspond to component signals, such as the left principal component signal 220, the right principal component signal 222, the left residual component signal 224, and the right residual component signal 226 (see [link to documentation]). Figure 2 (Figure 3 shows component signals 320, 322, 324, and 326, etc.)

[0204] At 708, object processing is performed on the estimated object using at least a second subset of multiple estimated rendering parameters. Performing object processing includes generating a processing signal based on the left principal component signal, the right principal component signal, the left residual component signal, and the right residual component signal. For example, object processing system 108 (see...) Figure 1 The estimated object 126 can be processed using one or more of the estimated rendering parameters 124 to generate a processing signal 128. As another example, processing system 208 (see...) Figure 2 Object processing can be performed on component signals 220, 222, 224, and 226 using one or more of the estimated rendering parameters 124 and object processing parameters 230 and 232.

[0205] Method 700 may include additional steps corresponding to other functions of one or more of the audio processing system 100, object processing systems 108, 208, 308, etc., as described herein. For example, method 700 may include receiving sensor data, head tracking data, etc., and performing processing based on the sensor data or head tracking data. As another example, object processing (see 708) may include processing the principal component using one set of processing parameters and processing the residual component using another set of processing parameters. As another example, method 700 may include performing an inverse transform, performing time-domain processing on the inversely transformed signal, etc.

[0206] Implementation details

[0207] The embodiments may be implemented in hardware, an executable module stored on a computer-readable medium, or a combination of both (e.g., a programmable logic array, etc.). Unless otherwise stated, the steps performed by the embodiments do not need to be inherently associated with any particular computer or other device, although they may be relevant in some embodiments. Specifically, various general-purpose machines may be used with programs written in accordance with the teachings herein, or more specialized devices (e.g., integrated circuits, etc.) may be more readily constructed to perform the desired method steps. Thus, the embodiments may be implemented in one or more computer programs that execute on one or more programmable computer systems, each of which includes at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0208] Each such computer program is preferably stored or downloaded to a storage medium or device (e.g., solid-state memory or medium, magnetic or optical medium, etc.) readable by a general-purpose or special-purpose programmable computer, for configuring and operating the computer to execute the program described herein when the computer system reads the storage medium or device. The system of the present invention can also be considered as an embodiment of a computer-readable storage medium configured with a computer program, wherein such a storage medium causes the computer system to operate in a specific and predefined manner to perform the functions described herein. Software itself and intangible or transient signals are excluded in the sense that they are not patentable subject matter.

[0209] The aspects of the system described herein can be implemented in a suitable computer-based audio processing network environment to process digital or digitized audio files. Parts of the adaptive audio system may include one or more networks comprising any desired number of independent machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks can be built on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0210] One or more components, blocks, processes, or other functional units may be implemented by a computer program executed by a processor-based computing device controlling the system. It should also be noted that any number of combinations of hardware, firmware, and / or data and / or instructions embodied in various machine-readable or computer-readable media may be used to describe the various functions disclosed herein in terms of behavior, register transfers, logical components, and / or other characteristics. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical, non-transitory, non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0211] The foregoing description illustrates various embodiments of this disclosure and examples of how aspects of this disclosure may be implemented. The foregoing examples and embodiments should not be considered as limited embodiments, but are presented to illustrate the flexibility and advantages of this disclosure as defined by the appended claims. Other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art based on the foregoing disclosure and the appended claims, and may be employed without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A computer-implemented audio processing method, the method comprising: Performing signal transformation on binaural signals, wherein the binaural signals are either binaural reproduction or binaural capture, wherein performing the signal transformation includes: Transform the binaural signals from the first signal domain to the second signal domain; and A transformed binaural signal is generated, wherein the first signal domain is the time domain and the second signal domain is the frequency domain, wherein the signal transformation is a time-frequency transformation, and wherein the transformed binaural signal includes multiple time-frequency slices transformed within a given time period; Spatial analysis is performed on each of the plurality of time-frequency slices of the transformed binaural signal, wherein performing the spatial analysis includes generating a plurality of estimated rendering parameters, wherein a given time-frequency slice of the plurality of time-frequency slices is associated with a given subset of the plurality of estimated rendering parameters, wherein the plurality of estimated rendering parameters includes a plurality of level differences and a plurality of phase differences, and wherein the plurality of estimated rendering parameters corresponds to at least one of a head-related transfer function, a head-related impulse response, and a binaural room impulse response used during the binaural reproduction or present in the binaural capture; Multiple objects are generated from the transformed binaural signal using at least a first subset of the multiple estimated rendering parameters, wherein the objects are represented by a corresponding left principal component signal, right principal component signal, left residual component signal, and right residual component signal of each corresponding time-frequency slice of the transformed binaural signal; and Object processing is performed on the plurality of objects using at least a second subset of the plurality of estimated rendering parameters, wherein performing the object processing includes generating a processing signal based on the left principal component signal, the right principal component signal, the left residual component signal, and the right residual component signal. The object processing includes at least one of repositioning, level adjustment, equalization, dynamic range adjustment, hissing elimination, multi-band compression, immersion enhancement, surround enhancement, upmixing, conversion, channel remapping, storage, and archiving.

2. The method of claim 1, wherein, Generating the processing signal includes: Using the first set of object processing parameters, a left primary processing signal and a right primary processing signal are generated from the left primary component signal and the right primary component signal; and Using a second set of object processing parameters, a left residual processing signal and a right residual processing signal are generated from the left residual component signal and the right residual component signal, wherein the second set of object processing parameters differs from the first set of object processing parameters. The object processing includes using the left main processing signal, the right main processing signal, the left residual processing signal, and the right residual processing signal.

3. The method of claim 1, further comprising: Receive sensor data from a sensor, wherein the sensor is a component of at least one of a headset, headphones, in-ear headphones, and a microphone. The object processing includes generating the processing signal based on the sensor data.

4. The method of claim 1, wherein, Performing the object processing includes: Based on sensor data, binaural translation is applied to the left principal component signal and the right principal component signal, wherein applying the binaural translation includes generating a left principal processing signal and a right principal processing signal; and The left residual processing signal and the right residual processing signal are generated from the left residual component signal and the right residual component signal without applying the binaural translation.

5. The method of claim 1, wherein, Performing the object processing includes: A monoear object is generated from the left principal component signal and the right principal component signal; Based on sensor data, binaural translation is applied to the monoaural object; and The left residual processing signal and the right residual processing signal are generated from the left residual component signal and the right residual component signal without applying the binaural translation.

6. The method of claim 1, wherein, Performing the object processing includes: A multi-channel output signal is generated from the left principal component signal, the right principal component signal, the left residual component signal, and the right residual component signal. The multi-channel output signal includes at least one left channel and at least one right channel. The at least one left channel includes at least one of the left front channel, left side channel, left rear channel, and left high channel. The at least one right channel includes at least one of the right front channel, right side channel, right rear channel, and right high channel.

7. The method of claim 1, wherein, Performing the object processing includes: Speech enhancement processing is performed on the left main component signal and the right main component signal, wherein applying the speech enhancement includes generating a left main processed signal and a right main processed signal; and A left residual processing signal is generated from the left residual component signal and a right residual processing signal is generated from the right residual component signal without applying the speech improvement processing.

8. The method of claim 1, wherein, Generating the processing signal includes: Applying a level adjustment to the left primary component signal and the right primary component signal using a first level adjustment value, wherein applying the level adjustment includes generating a left primary processed signal and a right primary processed signal; and Level adjustment is applied to the left residual component signal and the right residual component signal using a second level adjustment value, wherein applying the level adjustment includes generating a left residual processed signal and a right residual processed signal, and wherein the second level adjustment value is different from the first level adjustment value. The object processing includes using the left main processing signal, the right main processing signal, the left residual processing signal, and the right residual processing signal.

9. The method of any one of claims 1 to 8, wherein, The plurality of phase differences are plurality of unfolded phase differences, wherein the plurality of unfolded phase differences are unfolded by performing at least one of evidence-based unfolding and model-based unfolding.

10. The method of claim 9, wherein, Performing the evidence-based unfolding includes: In each frequency band, estimate the total energy of the left principal component signal and the right principal component signal; Cross-correlation is calculated based on each frequency band; and Based on the cross-correlation, the multiple expanded phase differences are selected from multiple candidate phase differences according to the energy between adjacent frequency bands.

11. The method of claim 9, wherein, Performing the model-based expansion includes: The plurality of expanded phase differences are selected from a plurality of candidate phase differences based on the level difference applied to the head-related transfer function in a given frequency band.

12. The method according to any one of claims 1 to 8, wherein, For a given index in the second signal domain, a given phase difference among the plurality of phase differences is calculated as the phase angle of the inner product of the left component of the transformed binaural signal and the right component of the transformed binaural signal.

13. The method according to any one of claims 1 to 8, wherein, The given level difference among the plurality of level differences is calculated based on a quadratic equation of the left component of the transformed binaural signal, the right component of the transformed binaural signal, and the given phase difference among the plurality of phase differences.

14. The method of any one of claims 2, 4, 7, and 8, further comprising: Perform inverse signal transformation on the left main processing signal, the right main processing signal, the left residual processing signal, and the right residual processing signal to generate a processing signal, wherein the processing signal is located in the first signal domain.

15. The method of any one of claims 1 to 8, further comprising: The processed signal is subjected to time-domain processing, wherein performing time-domain processing includes generating a modified time-domain signal.

16. A non-transitory computer-readable medium storing a computer program that, when executed by a processor, controls means to perform processing including the method as described in any one of claims 1 to 15.

17. An apparatus for audio processing, the apparatus comprising: A processor and a sensor, wherein the processor is configured to control the device to perform processing including the method as described in any one of claims 1 to 15.

Images