Target mid-side signals for audio applications

By using the target pan and phase difference parameters to calculate the target middle and side signals in the stereo audio signal, the problem of difficult to separate non-central pan audio sources in the prior art is solved, and the time and frequency variable audio sources are realized Efficient extraction and separation.

CN119072931APending Publication Date: 2024-12-03DOLBY LABORATORIES LICENSING CORP
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202380035874.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-11-08
Filing Date
2023-03-03
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, when processing stereo audio signals, it is difficult to effectively separate audio sources with non-center sound pans, especially moving audio sources and reverb sources, resulting in poor extraction of intermediate and side signals.

Method used

By obtaining the target pan parameters and target phase difference parameters in multiple continuous time periods and frequency bands of the stereo audio signal, partial intermediate signals and side signals are calculated and combined to form target intermediate and side audio signals to ensure signal alignment Audio source with variable time and frequency and allows separation of multiple audio sources.

Benefits of technology

Effective extraction and separation of time and frequency variable audio sources in stereo audio signals is achieved, improving the extraction accuracy and integrity of intermediate and side signals, and ensuring lossless reconstruction of left and right audio signals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072931A_ABST
    Figure CN119072931A_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method and an audio processing configuration for extracting a target intermediate (and optionally a target side) audio signal from a stereo audio signal. The method comprises obtaining (S1) a plurality of consecutive time periods of the stereo audio signal, and for each of a plurality of frequency bands of each time period of the stereo audio signal, obtaining (S2) at least one of a target acoustic phase parameter ([theta]) and a target phase difference parameter ([phi]). The method further comprises, for each time period and each frequency band, extracting (S3) a partial intermediate signal representation (211, 212) based on at least one of the target acoustic phase parameter ([theta]) and the target phase difference parameter ([phi]) for each frequency band, and forming (S4) a target intermediate audio signal (M) by combining the partial intermediate signal representations (211, 212) for each frequency band and time period.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 318,226 filed on March 9, 2022, U.S. Provisional Application No. 63 / 423,786 filed on November 8, 2022, and European Patent Application No. 22183794.1 filed on July 8, 2022. Technical Field

[0003] The present invention relates to source separation, signal enhancement and signal processing. The present invention also relates to a method for extracting target mid and side audio signals from a stereo audio signal. The present invention also relates to a processing device using the aforementioned method. Background Art

[0004] In the field of audio processing, audio source separation is important in some applications. In a simple application, the separated speech audio signal is maintained or provided with additional gain, while the background audio signal is omitted or attenuated relative to the speech audio signal. This can enhance the intelligibility of the speech audio signal.

[0005] In audio source separation, the signal of interest that is the target of estimation or extraction is called the target signal. The target signal is not limited to speech, and can be any ordinary audio signal (such as a musical instrument), or even multiple audio signals to be separated from an audio mixture including noise and / or additional audio signals of no interest.

[0006] For a stereo audio signal comprising two audio signals associated with a left and right audio channel, respectively, one processing method (which also implicitly performs some source separation) is to extract a mid and side audio signal from the left and right audio signals, wherein the mid and side audio signals are proportional to the sum and difference of the left and right audio signals, respectively. The mid signal will emphasize the audio components that are equal in amplitude and in phase between the left and right audio signals, while the side audio signal will attenuate or eliminate such signals. Thus, calculating the mid and side signals from the left and right audio signals constitutes an effective method for enhancing or attenuating the in-phase center-panned sources, respectively. The mid and side signals can be further converted back to left and right (traditional stereo) audio signals.

[0007] With respect to a center-phase source of interest, note that source separation has been implicitly performed: the mid signal will contain that source, while the side signal will not. The side audio signal will primarily include potentially uninteresting background audio signals that can be attenuated or omitted to enhance the intelligibility of the center-phase, in-phase audio source.

[0008] A drawback of conventional mid and side signal calculations is that implicit source separation based on mid / side audio signal separation fails when the desired audio source is not center-phase. To address this problem, specific mid and side signal calculation techniques have been developed for stereo audio signals with specific phase characteristics that can successfully align fixed non-center-phase audio sources. See, e.g., Master, Aaron, Ph.D. thesis "Stereo Music Source Separation via Bayesian Modeling", Stanford University, 2006.

[0009] While these techniques alleviate some of the shortcomings of basic mid and side signal extraction, such pan-specific extraction techniques are still unable to separate more general audio sources present in stereo audio signals, such as moving audio sources, sources with reverberation, or sources that are spatially dominant at various points in the time-frequency space.

[0010] It is therefore an object of the present disclosure to provide an improved method and audio processing system for performing enhanced audio source separation for stereo audio signals. Summary of the invention

[0011] According to a first aspect of the present invention, there is provided a method for extracting a target intermediate audio signal from a stereo audio signal, the stereo audio signal comprising a left audio signal and a right audio signal. The method comprises the following steps: obtaining a plurality of continuous time segments of the stereo audio signal, wherein each time segment comprises a representation of a portion of the stereo audio signal; and obtaining at least one of a target phase parameter and a target phase difference parameter for each of a plurality of frequency bands of each time segment of the stereo audio signal. The target phase parameter represents the distribution of the amplitude ratio between the left and right audio signals in the frequency band over the time segment, while the target phase difference parameter represents the distribution of the phase difference between the left and right audio signals of the stereo audio signal over the time segment.

[0012] The method also includes extracting a partial intermediate signal representation for each time segment and each frequency band, wherein the partial intermediate signal representation is based on a weighted sum of left and right audio signals, wherein the weight of each of the left and right audio signals is based on at least one of a target pan parameter (θ) and a target phase difference parameter for each frequency band and time segment, and forming a target intermediate audio signal by combining the partial intermediate signal representations for each frequency band and time segment.

[0013] Obtaining at least one of a target phase parameter and a target phase difference parameter may include receiving, accessing or determining a target phase parameter and / or a target phase difference parameter. For at least one time period and frequency band, at least one of the target phase parameter and / or the target phase difference parameter may be replaced with a default value.

[0014] A consecutive time segment refers to a segment describing a time portion of an audio signal, wherein a subsequent time segment describes a subsequent time portion of the audio signal and a preceding time segment describes a preceding time portion of the audio signal. The consecutive time segments may overlap in time or may not overlap in time.

[0015] The representation of a portion of a stereo audio signal may be any time domain representation or any frequency domain representation. The frequency domain representation may be any linear time-frequency domain, such as a Short Time Fourier Transform STFT representation or a Quadrature Mirror Filter QMF representation.

[0016] The present invention is based at least in part on the understanding that by obtaining a target pan parameter and / or a target phase difference parameter for each time segment and frequency band, a target intermediate audio signal can be extracted that is always aligned to a time- and / or frequency-variable audio source in a stereo audio signal. Furthermore, the method allows for the extraction of a target intermediate audio signal that is simultaneously aligned to two or more audio sources separated in frequency in the stereo audio signal, since the target intermediate audio signal is created using separate target pan parameters for each time segment and frequency band.

[0017] In some implementations, the weights of the left and right audio signals are based on a target pan parameter such that a left or right audio signal with a larger amplitude is assigned a larger weight.

[0018] That is, the left or right audio signal associated with a greater amplitude or power (indicated by the target pan parameter) will be assigned a greater weight and contribute more to the formation of the target intermediate audio signal.

[0019] In some embodiments, the method further comprises extracting a partial side signal representation for each time segment and frequency band, wherein the partial side signal representation is based on a weighted difference between left and right audio signals, wherein the weight of each of the left and right audio signals is based on at least one of a target pan parameter and a target phase difference parameter for each frequency band and time segment, and forming a target side audio signal by combining each partial side signal representation for each frequency band and time segment.

[0020] In other words, the target side audio signal can be formed in parallel with the target middle audio signal, and the target middle and side audio signals form a complete representation of the stereo audio signal. It should be understood that any implementation related to the formation or processing of the target middle audio signal can be performed similarly to the formation and processing of the target side audio signal, as will be described below.

[0021] If the stereo audio signal includes, for example, an ideal center phase audio source, or a single audio source panned under a constant power law, the target middle audio signal will ideally capture all signal energy of the audio source, while the target side audio signal will not capture such energy. However, for a general audio source, the target middle audio signal will capture one or more target audio sources, and may capture some other non-target sounds, while the target side audio signal will capture non-target sounds, while may eliminate or almost eliminate one or more target sources. In addition to the target middle audio signal, also including the target side audio signal ensures that all audio signals of the left and right audio signal pairs are present in the target middle and side audio signal pairs. For example, this allows lossless reconstruction of left and right audio signals.

[0022] Any functionality described with respect to the method may have a corresponding feature in the system or device, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The invention will now be described in more detail with reference to the accompanying drawings, which show currently preferred embodiments of the invention.

[0024] Figure 1 is a block diagram depicting an analysis device according to some embodiments.

[0025] Figure 2 is a flow chart illustrating a method according to one embodiment.

[0026] Figure 3 is a block diagram depicting an alternative analysis device according to some embodiments.

[0027] Figure 4a A set of target panning parameters and target phase difference parameters distributed over multiple time periods and frequency bands according to some embodiments are shown.

[0028] Figure 4b A set of partial intermediate signal representations forming a target intermediate audio signal according to some embodiments is shown.

[0029] Figure 4c A set of partial side signal representations forming a target side audio signal according to some embodiments is shown.

[0030] Figure 5a is a block diagram of an analysis device according to some embodiments.

[0031] Figure 5b is a block diagram of an analysis apparatus using alternative phase and phase difference parameters according to some embodiments.

[0032] Figure 6a is a block diagram of an audio processing system having a mono source separator and a mid / side processor according to some implementations.

[0033] Figure 6b is a block diagram of another audio processing system having a stereo processor that operates on alternate left and right audio signals according to some implementations.

[0034] Figure 6c is a block diagram of yet another audio processing system that applies an estimated stereo soft mask to a target middle audio signal to reconstruct processed left and right audio signals according to some embodiments. DETAILED DESCRIPTION

[0035] The systems and methods disclosed in this application can be implemented as software, firmware, hardware or a combination thereof. In hardware implementation, the division of tasks does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed by several physical components in cooperation.

[0036] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a network device, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by the computer hardware. In addition, the present disclosure will relate to any collection of computer hardware that executes instructions, alone or in combination, to perform any one or more of the concepts discussed herein.

[0037] Some or all components may be implemented by one or more processors that accept computer-readable (also referred to as machine-readable) codes containing a set of instructions that, when executed by one or more processors, implement at least one of the methods described herein. Any processor that is capable of executing (sequentially or otherwise) a set of instructions specifying an action to be taken is included. Thus, an example is a typical processing system (i.e., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may also include a memory subsystem that includes a hard drive, an SSD, RAM, and / or ROM. A bus subsystem for communication between components may be included. During execution by the computer system, the software may reside in the memory subsystem and / or in the processor.

[0038] The one or more processors may operate as independent devices or may be connected (e.g., networked) to other processors. Such a network may be based on a variety of network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0039] The software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transient media). As known to those skilled in the art, the term "computer storage media" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by a computer. In addition, as known to those skilled in the art, (transitory) communication media typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium.

[0040] A detailed description of the presently preferred embodiment is included below along with an introduction to basic mid and side audio signal extraction.

[0041] Extraction of basic mid and side audio signals

[0042] A stereo audio signal including a left L audio signal and a right R audio signal is optionally represented as a mid audio signal M and a side audio signal S. The basic mid and side audio signals M, S are constructed from the left and right audio signals L, R using the following analytical formula:

[0043] M=0.5L+0.5R Formula 1

[0044] S=0.5L-0.5R Formula 2

[0045] And, the original left and right audio signals L, R can be reconstructed from the mid and side audio signals M, S using the following synthesis formula:

[0046] L=M+S Formula 3

[0047] R=MS Formula 4

[0048] The middle audio signal M boosts audio signal features that are in phase and located at the center of the stereo mix, while the side audio signal S attenuates the in-phase audio signal features. For example, if the stereo audio signal includes a center-phase left and right audio signal pair L, R (equal amplitude, in-phase components in each of the left and right audio signals), the middle audio signal M will include the stereo audio signal, while the side audio signal S will cancel it; the basic middle / side audio signal is aligned with the center-phase audio signal included in the middle audio signal M and excluded from the side audio signal S.

[0049] Stereo signals in the time and time-frequency domains

[0050] The left, right, center and side audio signals L, R, M, S can be represented in the time domain or the frequency domain. The frequency domain representation can be, for example, a short-time Fourier transform STFT representation or a quadrature modulation filter QMF representation. For example, the left and right audio signals L, R can be represented in the frequency domain as

[0051] L=|L|e i∠L Formula 5

[0052] R=|R|e i∠R Formula 6

[0053] Wherein, |L| and |R| represent the signal amplitude of left and right audio signals L, R respectively, and wherein ∠L and ∠R represent the phase of left and right audio signals L, R respectively. In general, |L|, |R|, ∠L and ∠R are functions of frequency ω and time t, but for the sake of simplicity, all formulas do not consider time and frequency correlation. Therefore, each symbol |L|, |R|, ∠L and ∠R represents many values. Each such value corresponds to the audio content of the left or right signal, and does not necessarily characterize a single target source. These values ​​can usually be described as "detected", because in many situations they are considered to be detected from the stereo input signal.

[0054] The left and right audio signals L, R may alternatively be represented by a combination of a detected amplitude parameter U and a detected pan parameter θ

[0055]

[0056]

[0057] This means that the left and right audio signals L, R from equations 5 and 6 can be expressed by U, θ instead of |L| and |R|. For example, the left and right audio signals L, R can be expressed as

[0058] L=U cos(θ)e i∠L Formula 9

[0059] R=Usin(θ)e i∠R Formula 10

[0060] The detected amplitude parameter U and the detected phase parameter θ form a representation of the stereo audio signal, which is called a stereo polar magnitude (SPM) representation. Although the SPM representation can be replaced by any equivalent representation, the SPM representation of the stereo audio signal will be used in the following description.

[0061] The detected pan parameter θ ranges from 0 to Where 0 represents a stereo audio signal in which only the left audio signal L is non-zero, Represents a stereo audio signal in which only the right audio signal R is non-zero. The detected pan parameter value Indicates a centered stereo audio signal with equal amplitude for the left and right audio signals L, R. Although the detected pan parameter is derived from a stereo input signal, it also mathematically conforms to the model for source mixing. For a pan signal with amplitude |S x |, phase ψ x and an audio source S with known pan parameters Θx x , the left and right audio signals L and R can be expressed as

[0062]

[0063]

[0064] It can be seen that the detected phase parameter θ at the time-frequency point in the signal created using Equations 11 and 12 will be equal to the audio source S x The Θx is as follows:

[0065]

[0066] Mid and side audio signals with fixed panning

[0067] The basic mid and side audio signals M, S are extracted by equally weighting each of the left and right audio signals L, R to align an audio source (center audio source) assumed to be present in equal proportions in the left and right audio signals L, R. For a non-center audio source, the weights of the left and right audio signals L, R in equations 1 and 2 should be adjusted.

[0068] For example, the coefficients of the left and right audio signals L, R in equations 1 and 2 may be changed from 0.5 under the constraint that the sum of the coefficients should be equal to 1. This enables the coefficients of the left and right audio signals L, R in equation 1 to be, for example, 0.7 and 0.3 for an audio source that is primarily present in the left audio signal (the same is true for the side signal construction in equation 2). However, this leads to an undesirable situation in which the amplitudes of the mid and side audio signals M, S vary greatly when the audio source moves between the left and right audio signals L, R (assuming constant power law mixing).

[0069] To avoid this problem, for a sound with a known or estimated “target” pan parameter Θ x Audio source S x The middle and side audio signals M and S can be extracted from the left and right audio signals L and R according to the following formula:

[0070] M=cos(θ x )L+sin(Θ x )R Formula 14

[0071] S = sin(Θ x )L-cos(Θ x )R Formula 15

[0072] The left and right audio signals L, R can be reconstructed as

[0073] L = cos(θ x )M+sin(Θ x )S formula 16

[0074] R = sin(Θ x )M-cos(Θ x )S. Formula 17

[0075] For example, if Θ x = 0, indicating that the audio source only appears in the left audio signal, then the middle audio signal M becomes equal to the left audio signal L, and the side audio signal S is equal to the right audio signal R, for Similarly, for an audio source that appears in the left and right audio signals L, R with equal amplitude (i.e. ), the middle audio signal M equally weights the left and right audio signals L, R. In addition, audio sources located to the left and right of center will result in the middle audio signal M being equal to the left and right audio signals L, R, respectively. The range of Θ and arrive The left and right audio signals L, R are weighted within the range of θ. Therefore, the coefficients used to weight the left and right audio signals L, R to construct the mid and side audio signals M, S are based on the fixed target pan parameter θ x ,vice versa.

[0076] Extraction of target mid and side audio signals

[0077] One aspect of the present invention relates to creating a target mid and / or side audio signal M, S based on at least one of a target pan parameter θ and a target phase difference parameter φ obtained for each time segment and frequency band of a stereo audio signal. The target pan parameter θ represents an estimate or indication of an amplitude ratio between a left and right audio signal L, R in each time segment and frequency band for sounds corresponding to one or more target sources. The target phase difference parameter φ represents an estimate or indication of a phase difference between a left and right audio signal L, R in each time segment and frequency band for sounds corresponding to one or more target sources.

[0078] In some embodiments, the phase parameter θ and / or the phase difference parameter Φ for each time period and frequency band 111, 112 is the median, mean, mode, percentile, maximum or minimum value of the phase parameter θ and / or the phase difference parameter Φ for the time period. Typically, for each sample of the stereo audio signal, there is a detected phase parameter θ and / or a detected phase difference parameter φ. However, the target phase parameter θ and / or the phase difference parameter Φ for creating the target mid and side audio signals M, S does not necessarily have such a fine granularity, but may represent a statistical feature (e.g., an average) of a plurality of samples, such as an average of all samples in a predetermined time period. For example, although the detected phase parameter θ and / or the detected phase difference parameter φ may present different values ​​more than a thousand times per second, the target phase parameter θ and / or the phase difference parameter Φ may only change a few times per second.

[0079] Figure 1 An analysis device 10 is shown for extracting target mid M and target side S audio signals from a stereo signal comprising left and right audio signals L, R. The extraction of the mid and side audio signals M, S is performed in an extraction unit 12 of the analysis device 10, which obtains target pan parameters θ and / or target phase difference parameters Φ for the left and right audio signals L, R and a plurality of time segments 100 and frequency bands 111, 112.

[0080] The extraction unit 12 obtains the left and right audio signals L, R (stereo audio signals) as a plurality of continuous time segments, each time segment comprising a representation of a portion of the left and right audio signals R, L. Alternatively, the extraction unit 12 is configured to divide the left and right audio signals R, L into a plurality of continuous time segments. Each time segment comprises two or more frequency bands of the stereo audio signal, wherein each frequency band in each time segment is associated with at least one of the obtained target pan parameter θ and the obtained target phase difference parameter Φ.

[0081] exist Figure 1In the illustrated embodiment, the analysis device 10 receives the left and right audio signals L, R and the target phase parameter θ and / or the target phase difference parameter φ for each frequency band 111 in each time period, wherein the number of frequency bands is from frequency band 1 to frequency band B. θ (t,1) and Φ (t,1) represents the target phase parameter θ and the target phase difference parameter Φ of the first frequency band 111 in time period t, and θ (t,2) and Φ (t,2) θ represents the target phase parameter θ and the target phase difference parameter Φ of the second frequency band 112 for the time period t.

[0082] Using the left and right audio signals L, R and the target phase parameter θ and / or the target phase difference parameter Φ, the extraction unit 12 extracts the target middle audio signal M and the target side audio signal S. In some embodiments, the extraction unit 12 extracts only one of the target middle M and side S audio signals, for example, only extracts the middle audio signal M.

[0083] In order to allow the modeling of the inter-channel phase difference between the left and right audio signals L, R, the true source phase of the target audio source is defined to be proportionally closer to the left or right audio signal exhibiting a larger amplitude. That is, for a stereo audio signal dominated in power or amplitude by the left audio signal L, the phase of the left audio signal L will be closer to the true source phase of the audio source than the phase of the right audio signal R, and vice versa for a stereo audio signal dominated by the right audio signal R. Audio signal S x is modeled as occurring in the left and right audio signals L, R according to the following mixing equation:

[0084]

[0085]

[0086] where ψ x Is the audio source S x The phase of , and Ф x is the phase difference parameter of the target source.

[0087] Based on the hybrid model described in the above equations 17 and 18, the extraction device 12 can use the following analytical formula to align the source with the target acoustic phase parameter Θ and the target phase difference parameter Φ.

[0088]

[0089]

[0090] It should be understood that Θ and Φ may vary with time and frequency depending on the characteristics of the target source. The mid and side signals of equations 20 and 21 may be referred to as "target mid and side signals". Thus, M and S may be understood as signals composed of separate signal components corresponding to different values ​​of Θ and Φ for each time period and frequency band 111, 112. 2 Θ and cos 2 The weighting of the phase difference parameter Φ of the left and right audio signals L, R by θ is merely exemplary, and other weighting functions based on θ are also possible. 2 Θ x Can be used Instead, cos 2 Θ can also be expressed as 1- Instead. Similarly, using sin 2 Θ x And cos 2 Θ x The models of equations 18 and 19 for weighting the phase difference parameter Φ between the left and right audio signals L, R are also merely exemplary.

[0091] In equation 20, the target intermediate audio signal M is based on the weighted sum of the left and right audio signals L, R, where the weight of the left audio signal is The weight of the right audio signal is And wherein for each time period and frequency band 111 , 112 the phase and / or phase difference parameters θ and Φ are obtained.

[0092] Similarly, in Equation 20, the target side audio signal S is based on the weighted difference between the left and right audio signals L, R, where the weight of the left audio signal is The weight of the right audio signal is And wherein for each time period and frequency band 111 , 112 the phase and / or phase difference parameters θ and Φ are obtained.

[0093] The weights of the left and right audio signals L, R in the weighted sum and weighted difference of equations 20 and 21 for extracting the mid and side audio signals M, S are complex values ​​and include, for example, real-valued amplitude factors, such as cos(θ), and complex-valued phase factors, such as The obtained target pan and / or target phase difference parameters θ and Φ will influence the weights so that the target mid and / or side audio signals M, S can be extracted.

[0094] For example, the real-valued amplitude factor can be based on Θ, so that the left or right audio signal with a larger amplitude has a larger weight in the weighted sum. In addition, the complex-valued factor can be based on Φ, so that a larger phase difference Φ means a larger difference between the corresponding phases of the weights in the weighted sum. In some embodiments, the complex-valued factor of each weight in the weighted sum is based on both Φ and Θ, for example, one of the weights is The weight of the left or right audio signal L, R having a larger amplitude is set to a smaller phase.

[0095] Similarly, the real-valued amplitude factor in the weighted difference (for target side signal extraction) can be based on the target phase parameter θ, so that the left or right audio signal with a larger amplitude is set with a smaller weight in the weighted difference. In addition, the complex-valued factor can be based on Φ, so that a larger phase difference Φ means a larger difference between the corresponding phases of the weights in the weighted difference. In some implementations, the complex-valued factor of each weight in the weighted difference is based on both Φ and θ, for example, one of the weights is The weight of the left or right audio signal with a larger amplitude is set to a smaller phase.

[0096] In some embodiments, for at least one time period and frequency band 111, 112, only one of the target phase and phase difference parameters θ and Φ is obtained, and the other is assigned a default value. For example, if only the target phase parameter θ is obtained, the target phase difference parameter Φ is assumed to be Φ=0, and if only the target phase difference parameter Φ is obtained, the target phase parameter θ is assumed to be These default values ​​assume that the audio source is located in the center of the stereo audio signal with no inter-channel phase differences, but other default values ​​are possible.

[0097] The extraction device 12 combines the partial intermediate signal components or representations M of each frequency band in each time period. (t,n) To obtain the target intermediate audio signal M. Similarly, the extraction device 12 obtains the target auxiliary audio signal S by combining the partial side signal representations S(t,n) for each frequency band in each time segment. A partial intermediate signal component or representation M is created for each frequency band 1...B in time segment t. (t,n) , and by combining all partial intermediate signals of time period t, M (t,n) , create the target intermediate audio signal time segment M (t) , where the target intermediate audio signal time period M (t) The target middle audio signal M is formed by a sequence of . The target side audio signal S is formed in a similar manner. The partial middle signal of the combined frequency band 1 ... B represents M (t,n) It may include processing part of the target intermediate audio signal M with a synthesis filter bank. (t,n) .

[0098] Further references Figure 2 The flowchart in Figure 1 details the method performed by the analysis device 10. A stereo audio signal is obtained by the analysis device 10 at S1, and target pan and / or phase difference parameters Θ and Φ are obtained by the analysis device 10 for a plurality of time periods 100 and frequency bands 111, 112 at step S2.

[0099] In step S3, the mid / side extractor 12 extracts a portion of the mid signal representing M. (t,n) (e.g., according to the above formula 19), where the partial mid signal representation is based on the weighted sum of the left and right audio signals L, R. The weight of each of the left and right audio signals L, R is based on at least one of the target sound phase parameter θ and the target phase difference parameter Φ for each frequency band 111, 112 and time period. Similarly, in step S31, the partial side signal representation S is extracted. (t,n) (eg, according to equation 20 above), where the partial side signal represents a weighted difference based on the left and right audio signals L, R.

[0100] In some embodiments, the target side audio signal S is not calculated, and the extraction unit 12 is, for example, an extraction unit 12 configured to extract only the target middle audio signal M. Alternatively, before reconstructing the left and right audio signals L, R in step S6 (for example, using the following reconstruction equations 21 and 22), the target side audio signal S is calculated, but it is reduced to a small value or zero in step S51. Since the target middle audio signal M is expected to capture substantially all the energy of the target audio source in some cases, the target side audio signal S may mainly include undesirable noise or background audio, which can be ignored or attenuated.

[0101] After step S3 and step S31, the method proceeds to steps S4 and S41, including by representing the portion of the middle and side signals M (t,n) , S (t,n) The method continues with steps S5 and S51, including performing processing of the target mid and / or side audio signals M, S, and the processing example includes attenuating the target side audio signal S or providing the target mid audio signal M to a mono source separator, as will be described below in conjunction with Figure 6a , 6b and as described in 6c.

[0102] Finally, the method proceeds to step S6, comprising reconstructing the left and right audio signals L, R from the target mid and side audio signals M, S. The reconstruction of the left and right audio signals L, R from the target mid and side audio signals M, S is performed by a synthesis device, as will be described below in conjunction with Figure 5a and 5b Described in more detail.

[0103] As an example of the operation of the analysis device 10, consider an audio source having left and right audio signals L, R with time-varying phases. According to the constant power law, as time t advances from 0 seconds to 10 seconds, the audio source pans from the far right to the far left at a constant rate. For the basic mid and side audio signals extracted for this audio source, for example, according to the above equations 1 and 2, the mid audio signal M will capture substantially all of the energy of the audio source at t=5, and substantially no energy at t=0 and t=10, while the side audio signal S will capture substantially all of the energy of the audio source at t=0 and t=10, and substantially no energy at t=5. The target mid audio signal M extracted using the phase parameters and / or phase difference parameters θ and Φ that vary for each time period and frequency band will contain substantially all of the energy of the audio source for any t∈[0,10], while the target side audio signal S will contain substantially no energy of the time-varying audio source.

[0104] The constant power law ensures that the audio signal power (proportional to the sum of the squares of the amplitudes) of the left and right audio signals L, R remains constant for all pan angles. For example, the audio signal amplitudes of the left and right audio signals are scaled by a scaling factor that depends on the pan angle to ensure that the square of the sum of the left and right audio signals is equal for all pan angles. Unlike the linear panning law that ensures that the sum of the signal amplitudes is constant while allowing the total signal power to vary with the pan angle, the constant power panning law will ensure that the perceived audio signal power is constant for all pan angles. The constant power law is described, for example, in "Loudness Concepts & Pan Laws" by Anders Oland and Roger Dannenberg, Introduction to Computer Music, Carnegie Mellon University.

[0105] This demonstrates how the target mid and side audio signals M, S can align and separate audio sources that vary in time and frequency in a stereo audio signal. In addition, if a second audio source of a separate frequency moves between the left and right audio signals L, R at a different rate than the first audio source (or is stationary, for example, when center panned), the target mid audio signal M is aligned with the first and second audio sources in each frequency band, which means that even if the audio sources have different frequencies, substantially all of the energy from the two audio sources will still appear in the target mid audio signal M while also moving between the left and right audio signals L, R at different rates.

[0106] exist Figure 3, a combined analysis device 10' is depicted, which is configured to extract the target mid and side audio signals M, S according to the above, while also extracting or determining the target sound phase parameter θ and / or phase difference parameter Φ for each frequency band 111, 112 and time period of the left and right audio signals L, R using a parameter extraction unit 14. Therefore, the target sound phase parameter θ and / or phase difference parameter Φ are obtained together with the left and right audio signals L, R, or are determined from the left and right audio signals L, R. An example method of doing this will now be described.

[0107] The target phase difference parameter Φ can be calculated, for example, by the parameter extractor 14 as a typical or dominant phase difference between the left and right audio signals L, R in each time period and frequency band 111, 112. For example, in the time-frequency domain (e.g., in the STFT domain), the detected phase difference can be calculated for each STFT tile as φ=Arg(R / L), and by analyzing the distribution on φ, the value of Φ can be estimated to be the dominant value or typical value of the time period and frequency band. Similarly, the detected phase parameter θ can be calculated for each STFT tile as θ=arctan(|r| / |L|), and by analyzing the distribution thereon, the value of θ can be estimated to be the dominant value or typical value of the time period and frequency band, where L and R represent the left and right audio signals L, R in the time period and frequency band 111, 112. The typical value or dominant value estimated from the distribution can be the median, mean, mode, percentile, minimum or maximum value.

[0108] In "Dialog Enhancement via Spatial-Level Filtering and Classification" (Master, A et al., Audio Engineering Society 149th Conference Paper 10427), a method for obtaining indicators of the mean and spread of the phase difference between the channels, labeled thetaMid, thetaWidth, phiMid, and phiWidth, is proposed. The phase parameter Θ used by the analysis device 10 can be based on thetaMiddle proposed in the conference paper, and similarly, the target phase difference parameter Φ used by the analysis device 10 can be based on phiMiddle, a so-called "shift and squeeze" or "S&S" parameter proposed in the paper.

[0109] For each time segment and frequency band within it, the audio processing system can create a square of the amplitude U (i.e., U 2) weighted histogram of 51 bins on θ. The system does the same for the detected pan parameter φ and a version of φ that ranges from 0 to 2*π, called φ2. However, each of these histograms uses 102 bins. Each histogram is smoothed over its given dimension and with respect to the time period. For the smoothed θ histogram, the system detects the target pan parameter as the highest peak, called thetaMiddle, and the width around this peak required to capture 40% of the energy in the histogram, called thetaWidth. It does the same for φ and φ2, recording phiMiddle, phi2Middle, phiWidth, and phi2Width, but for the width requiring 80% energy capture. The system records the final values ​​of phiMiddle and phiWidth, based on which the final values ​​have a higher concentration in phi space, as indicated by a smaller phiWidth value.

[0110] For example, Figure 3 The parameter extractor 14 in the example creates a total signal power U from equation 7 for each time period and the frequency band therein. 2 A histogram of the weighted detected phase parameters θ is obtained, and the same operation is performed on the detected phase difference parameter φ. The histogram can have any number of bins, such as 10 or more bins, or 50 or more bins. In one implementation, each histogram has 51 bins. Each histogram is smoothed with a suitable smoothing function at least with respect to the time dimension and / or with respect to the time and θ and / or φ dimensions. For the smoothed θ histogram, the system detects the θ value of the highest peak, called thetaMiddle, and the width around this peak required to capture 40% of the energy in the histogram, called thetaWidth. The same operation is performed on φ, recording phiMiddle and phiWidth, but the width requires 80% energy capture. It should be understood that the parameter extractor 14 can be configured to obtain the detected phase parameter θ and / or the detected phase difference parameter φ and / or the detected amplitude U, or to determine the detected phase parameter θ and / or the detected phase difference parameter φ and / or the detected amplitude U from the left and right audio signals L, R (using, for example, Equation 7).

[0111] Details of time periods and frequency bands

[0112] refer to Figure 4a , depicts a plurality of time periods 110, 120 from a first time period to a time period t, wherein each time period 110, 120 comprises a plurality of frequency bands 111, 112, 121, 122 from a first frequency band to a frequency band B. The time periods 110, 120 and the frequency bands 111, 112, 121, 122 form a time and frequency slice representation, wherein each frequency band of each time period forms a corresponding time / frequency slice.

[0113] like Figure 4a As shown, each frequency band 111, 112, 121, 122 of each time segment 110, 120 is associated with a corresponding target sound phase parameter θ and / or a target phase difference parameter Φ. At least two frequency bands 111, 112, 121, 122 (or sub-bands) are defined for each time segment 110, 120, for example a high frequency band and a low frequency band including frequencies above and below a threshold frequency. That is, each time segment 110, 120 includes frequency bands 1...B, where B is at least 2. Experiments have shown that octave or quasi-octave frequency bands are very suitable for the alignment of dialogue audio sources, for example frequency bands with edges at 0 Hz, 400 Hz, 800 Hz, 1600 Hz, 3200 Hz, 6400 Hz, 13200 Hz and 24000 Hz, but different frequency band distributions are also possible.

[0114] The duration of each time segment may correspond to 10 ms to 1 s of a stereo audio signal. In some embodiments, each time segment corresponds to a plurality of frames of a stereo audio signal, such as 10 frames, wherein each frame represents, for example, 10 to 200 ms of a stereo audio signal, such as 50 to 100 ms of a stereo audio signal. Experiments have shown that using 10 overlapping frames to represent a time segment, wherein each frame is 50 to 100 ms long with 75% overlap, is a good compromise between fast response, parameter stability, parameter reliability, and computational cost for typical dialogue sources. However, in particular for audio sources other than dialogue, such as music, time segments of longer or shorter duration are also considered.

[0115] Figure 4b The partial intermediate audio signal representation M extracted for each time period 210, 220 and frequency band 211, 212, 221, 222 is shown. (t,b) As shown in the figure, the first time period 210 includes a portion of the intermediate audio signal of the first frequency band 211, which is represented by M (1,1) The second portion of the intermediate audio signal of the second frequency band 212 represents M (1,1) And so on, until the last middle audio signal of band B represents M (1,B) By combining the partial intermediate audio signal representation M of the first time period 210 (1,1) …M (1,B) , the first time segment M of the target intermediate audio signal M is created (1) Similarly, by combining the partial intermediate audio signal representations of the subsequent time period 220, (1,1) …M (1,B) , creates a subsequent time segment M of the target intermediate audio signal M (2) …M (t) Finally, by combining each time segment M in turn(1) …M (t) To create the target intermediate audio signal M.

[0116] Further references Figure 4c , showing the partial side audio signal representation S extracted for each time period 310, 320 and frequency band 311, 312, 321, 322 (t,b) . M is represented by a partial intermediate audio signal similar to that described above (t,b) The combined side audio signal represents S (t,b) To form the target side audio signal S. It should be understood that the portion of the middle audio signal represents M (t,b) and / or part of the side audio signal indicates S (t,b) The extraction and combination of may be performed in a single step by a single processing unit, or as two or more steps performed by two or more processing units.

[0117] Reconstructing left and right audio signals from target mid and side audio signals

[0118] Figure 5a A block diagram illustrating a synthesis device 20 for reconstructing left and right audio signals L, R from target mid and side audio signals M, S is shown. The synthesis unit 20 includes a reconstructor unit 22, which reconstructs the left and right audio signals L, R from the target mid and side audio signals M, S using a phase parameter θ and / or a phase difference parameter Φ. For example, the reconstructor unit 22 implements the following synthesis equation for the target mid and side audio signals M, S extracted by equations 20 and 21:

[0119]

[0120]

[0121] The target phase parameter θ and / or the target phase difference parameter Φ obtained by the synthesis device is equal to the target phase parameter θ and / or the target phase difference parameter Φ determined or obtained by the analysis device. Alternatively, the target phase parameter θ and / or the target phase difference parameter Φ used by the synthesis device is an optional parameter θ ALT , Ф ALT , and / or at least one of Θ and Φ is not specified and is replaced with a default value for at least one time period and frequency band.

[0122] As shown in equations 22 and 23, the left and right audio signals L and R are reconstructed based on the weighted sum and weighted difference of the target middle and side audio signals M and S. However, these equations can be modified to reduce the weight or eliminate the contribution of the side signal. For example, the left audio signal L and the right audio signal R can be reconstructed only from the target middle audio signal M, and the middle weighting factor for the left signal is The intermediate weighting factor for the right signal is Wherein the weighting factor is based on at least one of the target sound phase parameter θ and / or the target phase difference parameter Φ. In this example, the contribution of the side signal is zero. Other contributions between zero and the values ​​shown in equations 22 and 23 are also possible.

[0123] Similarly, the left and right audio signals L, R can be reconstructed by taking into account the target side audio signal S. In this case, the left audio signal L is based on a weighted sum of the target middle audio signal M and the target side audio signal S, where the weights and Based on at least one of a phase parameter and a phase difference parameter. The right audio signal R is based on a weighted difference between the target mid audio signal M and the target side audio signal S, wherein the weight and Based on at least one of the pan parameter Θ and / or the phase difference parameter Φ. The contribution from the intermediate signal may also be down-weighted or reduced to zero.

[0124] Figure 5b is a block diagram showing a synthesis device 20 that uses equations 22 and 23 to substitute the phase parameter θ ALT and / or alternative phase difference parameter Φ ALT To perform the reconstruction, this results in the replacement of the left and right audio signals L ALT , R ALT In the implementation shown, for all time periods and frequency bands, and Φ ALT =Ф CEN = 0 to create a centered left and right audio signal L with no inter-channel phase difference CEN , R CEN Center-panned stereo audio signal.

[0125] The alternative middle and side weighting factors can be defined as the middle and side weighting factors described in equations 22 and 23, where Θ, Φ are replaced by the alternative corresponding values ​​Θ obtained for each time period and frequency band. ALT , Ф ALT In some embodiments, only the target intermediate audio signal M and the alternative intermediate weighting factors (eg, coefficients of the target intermediate audio signal M in equations 22 and 23, where θ ALT , Ф ALT Substituting Θ, Φ) to create alternative left and right audio signals L ALT , R ALT Alternatively, the target mid and target side audio signals M, S and alternative mid and side weighting factors (e.g., coefficients of the target mid audio signal M and the target side audio signal S in equations 22 and 23, respectively, are used, where θ ALT , ФALT Substituting Θ, Φ) to create alternative left and right audio signals L ALT , R ALT .

[0126] Processing device for realizing target mid and side audio signals

[0127] In the following, Figure 6a , 6b 6c depicts a block diagram showing an audio processing system comprising an analysis device 10 and a synthesis device 20, in combination with additional audio processing units, such as a mono source separator 30, a mid / side processor 40, a stereo processor 50 and a stereo soft masking estimator 60, for performing a target mid and side audio signal M, S or an alternative left and right audio signal L CEN , R CEN Audio processing.

[0128] Figure 6a An audio processing system is shown for processing a stereo audio signal via target mid and side audio signals M, S extracted by an analysis device 10. As shown, the target mid audio signal M is used as input to a mono source separator 30. Furthermore, the target mid and side M, S audio signals are provided to a mid / side processor 40 for applying enhancement and / or attenuation to the audio signal.

[0129] The target intermediate audio signal M is provided to a mono source separation system 30, which is configured to separate the audio source in the target intermediate audio signal M and generate a source-separated intermediate audio signal M. sep For example, the mono source separation system 30 generates a source-separated intermediate audio signal M sep , wherein the intelligibility of at least one audio source present in the target intermediate audio signal M is enhanced.

[0130] The intermediate audio signal M after source separation sep The target side audio signal S is provided to the mid / side processor 40 to generate processed mid and side audio signals M', S'. The mid / side processor 40 may apply a gain to the source-separated mid audio signal M sep and / or attenuate the target side signal S. In some cases, the mid / side processor 40 sets the target side audio signal S to zero. The processed mid and side audio signals M', S' are then provided to the synthesis device 20, which includes a left and right audio signal reconstruction unit 22, which reconstructs the processed left and right audio signals L', R' based on the processed mid and side audio signals M', S' and the pan and / or phase difference parameters θ, φ obtained by the analysis device 10.

[0131] In some embodiments, the source-separated intermediate audio signal Msep is directly provided to the synthesis device 20, which reconstructs the processed left and right audio signals L', R' based at least on the source-separated intermediate audio signal Msep (and optionally the target side audio signal S) and the target phase and / or phase difference parameters Θ, Φ.

[0132] Optionally, the mono source separation system 30 is omitted and the target mid audio signal M and the target side audio signal S are directly provided to the mid and side audio processing unit 40, which extracts processed mid and side audio signals M', S' from the target mid and side audio signals M,S.

[0133] Figure 6b It shows how to extract alternative center left and right audio signals L using the target mid audio signal M and the target side audio signal S. CEN , R CEN , to enable the use of the stereo processing system 50 for a center-phase stereo audio signal.

[0134] The analyzing device 10a obtains any left and right audio signals L, R and target phase and / or phase difference parameters θ, φ for each time period and frequency band, and extracts a target middle audio signal M and a target side audio signal S. The target middle and side audio signals M, S are provided to the synthesizing device 20a, which also obtains a set of alternative center phase and / or phase difference parameters θ CEN , Ф CEN , indicating a center-phase stereo audio signal. Thus, the analyzing means 10a and the synthesizing means 20a create from any original stereo signal a left and right audio signal L including the centered CEN , R CEN The center panned stereo signal.

[0135] Alternative center panned left and right audio signals L CEN , R CEN is provided to a stereo processing system 50, which is configured to perform stereo source separation on the center phase stereo source to output processed center left and right audio signals L' CEN , R' CEN The processed center left and right audio signals L' CEN , R' CEN is characterized by, for example, a center left and right audio signal L CEN , R CEN The intelligibility of at least one audio source present in the image is enhanced.

[0136] Processed center left and right audio signals L' CEN , R'CEN It is then supplied to the second analysis device 10b, which uses the processed center left and right audio signals L' CEN , R' CEN and the center phase and / or phase difference parameter θ CEN , Ф CEN Finally, the second synthesis device 20b reconstructs the processed left and right audio signals L', R' using the original phase and / or phase difference parameters Θ, Φ obtained or determined by the first analysis device 10a.

[0137] It should be noted that the center phase and / or phase difference parameter θ indicating the center phase without inter-channel phase difference CEN , Ф CEN is the alternative pan and / or phase difference parameter Θ ALT , Ф ALT For example, the central parameter Θ is replaced by CEN , Ф CEN Any replacement parameter Θ can be used ALT , Ф ALT Instead, it indicates, for example, a strictly right or left stereo audio signal with or without a non-zero phase difference.

[0138] It is also noted that the target mid and side audio signals M, S fed between the first analyzing means 10a and the first synthesizing means 20a may be subjected to a mono signal separation process and / or a mid and side process as described above in connection with Figure 6a The same applies to the processed target mid and side audio signals M', S' fed between the second analysis device 10b and the second synthesis device 20b. That is, Figure 6a The mono source separator 30 and / or the mid / side processor 40 may be used to process the mid and side audio signals passed between at least one of the first analysis device 10a and the first synthesis device 20a and between the second analysis device 10b and the second synthesis device 20b. In addition, although the first analysis and synthesis devices 10a, 20a are described as separate units, they may be combined into a single unit, see equations 20, 21, 22, 23, it is clear that the left and right audio signals L for creating the substitute are used. CEN , R CEN The operations of can be performed as a single step. The same applies to the second analysis and synthesis apparatus 10b, 20b.

[0139] In some implementations, the alternate (eg, centered) left and right audio signals L CEN , R CENis created by the first synthesis means 20a using only the target intermediate audio signal M. Similarly, the second analysis means 10b may be configured to extract only the processed target intermediate audio signal M'.

[0140] Figure 6c An analysis device 10 is shown operating in conjunction with a filter estimator configured to receive left and right audio signals L, R and estimate filters for source separation. The filter estimator is, for example, a stereo soft masking estimator 60 configured to receive left and right audio signals L, R and estimate a soft masking F. The stereo soft masking estimator 60 may be any soft masking estimator and, for example, is configured to output a soft masking F intended for stereo source separation. Traditionally, the soft masking F is applied to the left and right audio signals L, R of a stereo audio signal.

[0141] exist Figure 6c In the implementation shown, the estimated stereo soft mask F is applied to the target intermediate audio signal M to form the source separated intermediate audio signal M sep The intermediate audio signal M after source separation sep The target side audio signal S is provided to the mid / side processor 40, which outputs a processed target middle audio signal M' and a processed target side audio signal S', wherein the processed side audio signal S' may be reduced in weight compared to S, including reduced in weight to zero. The processed target middle audio signal M' and the processed target side audio signal S' are provided to the synthesis unit 20, which reconstructs the processed left and right audio signals L', R'. Optionally, the mid / side processor is omitted, whereby the source separated middle audio signal M sep and the optional target side audio signal S are directly provided to the synthesis unit 20 for reconstruction.

[0142] The stereo soft masking estimator 60 is configured to determine or obtain the phase and / or phase difference parameters θ, φ and provide these parameters to the analysis device 10. Alternatively, the phase and / or phase difference parameters θ, φ are obtained from elsewhere by the analysis device 10 or determined by the analysis device 10 based on the left and right audio signals L, R.

[0143] Unless otherwise stated, it will be apparent from the following discussion that it should be understood that throughout the public discussion, terms such as "processing," "computing," "calculating," "determining," "analyzing," etc., are used to refer to the actions and / or processes of computer hardware or computing systems or similar electronic computing devices that manipulate and / or transform data represented as physical quantities (e.g., electronic quantities) into other data similarly represented as physical quantities.

[0144] It should be understood that in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes combined in a single embodiment, figure or description thereof for the purpose of simplifying the disclosure and helping to understand one or more various inventive aspects. However, this method of disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than the features explicitly stated in each claim. On the contrary, as reflected in the following claims, the inventive aspects are less than all the features of a single aforementioned disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, and each claim is independently a separate embodiment of the present invention. In addition, although some embodiments described herein include some but not other features included in other embodiments, the combination of features of different embodiments is meant to be within the scope of the present invention and to form different embodiments, as will be understood by those skilled in the art. For example, in the following claims, any claimed embodiment may be used in any combination.

[0145] In addition, some embodiments are described here as a combination of methods or elements of methods that can be implemented by a processor of a computer system or by other devices that perform the function. Therefore, a processor with necessary instructions for performing this method or method element forms a device for performing the method or method element. Note that when the method includes several elements, such as several steps, the order of these elements is not implied unless otherwise specified. In addition, the elements of the device embodiments described here are examples of devices for performing the functions performed by the elements, with the purpose of realizing the present invention. In the description provided herein, many specific details are set forth. However, it should be understood that embodiments of the present invention can be implemented without these specific details. In other cases, known methods, structures and techniques are not shown in detail to avoid blurring the understanding of this specification.

[0146] Those skilled in the art realize that the present invention is by no means limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, Figure 6c The stereo processing system in the embodiment may further include processing the intermediate audio signal M or the intermediate audio signal M after source separation. sepThe mono source separator 30 is operated. As another example, it is contemplated that all components or processes associated with forming target middle and side audio signals and processing the middle and side audio signals can be configured to form or process only the target middle audio signal, or alternatively, to form or process the target middle audio signal and the target side audio signal. Similarly, reconstructing the left and right audio signals (e.g., alternative or centered left and right audio signals) based on the target middle audio signal and the side audio signal can include using only the target middle audio signal, or alternatively using the target middle audio signal and the side audio signal, to reconstruct the left and right audio signals.

[0147] Various features and aspects will be understood from the following enumerated exemplary embodiments ("EEE"):

[0148] EEE 1. A method for extracting a target intermediate audio signal (M) from a stereo audio signal, the stereo audio signal comprising a left audio signal (L) and a right audio signal (R), the method comprising: obtaining (S1) a plurality of consecutive time segments of the stereo audio signal, wherein each time segment comprises a representation of a portion of the stereo audio signal; for each of a plurality of frequency bands of each time segment of the stereo audio signal, obtaining (S2) at least one of the following parameters: a target phase parameter (θ) representing a distribution of an amplitude ratio between left and right audio signals (L, R) in the frequency band over the time segment; and a target phase difference parameter (φ) representing Distribution of a phase difference between left and right audio signals (L, R) of a stereo audio signal over a time period; for each time period and each frequency band, extracting (S3) a partial intermediate signal representation (211, 212), wherein the partial intermediate signal representation (211, 212) is based on a weighted sum of the left and right audio signals (L, R), wherein the weight of each of the left and right audio signals (L, R) is based on at least one of a target phase parameter (θ) and a target phase difference parameter (Φ) for each frequency band; and forming (S4) a target intermediate audio signal (M) by combining the partial intermediate audio signal representations (211, 212) for each frequency band and time period.

[0149] EEE 2. The method according to EEE 1, wherein the weights of the left and right audio signals (L, R) are based on the target pan parameter (θ) such that a left or right audio signal with a larger amplitude is set with a larger weight.

[0150] EEE 3. The method according to EEE 1 or EEE 2, wherein the weights of the left and right audio signals (L, R) are complex-valued weights, and wherein the phases of the complex-valued weights are based on a target phase difference parameter (Φ). EEE 3. The method according to EEE 1 or EEE 2, wherein the weights of the left and right audio signals (L, R) are complex-valued weights.

[0151] EEE 4. The method according to EEE 3, wherein the complex-valued weight is based on the target phase difference parameter (Φ) and the target sound phase parameter (Θ), so that for a left or right audio signal with a larger amplitude, the phase difference produced by applying the complex weight is smaller.

[0152] EEE 5. The method according to any one of the preceding EEEs further comprises: providing a stereo audio signal to a stereo source separation system (60), the stereo source separation system (60) being configured to output a filter for stereo source separation; and applying the filter to the target intermediate audio signal to form a processed target intermediate audio signal (M sep ).

[0153] EEE 6. The method according to any one of the preceding EEEs further comprises: providing a target intermediate audio signal (M) to a mono source separation system (30), the mono source separation system (30) being configured to perform audio source separation on the mono audio signal and output a processed mono audio signal; and using the processed mono audio signal as a processed target intermediate audio signal (M sep ).

[0154] EEE 7. The method according to any one of EEE 5-6, further comprising reconstructing processed left and right audio signals (L', R') to form a processed stereo audio signal; wherein each of the processed left and right audio signals (L', R') is based on a processed target intermediate audio signal (M) weighted with corresponding left and right weighting factors. sep ), wherein the left and right weighting factors are based on at least one of a target sound phase parameter (θ) and a target phase difference parameter (Φ) for each time segment and frequency band.

[0155] EEE 8. The method according to any one of the preceding EEEs further comprises: reconstructing the alternative left and right audio signals (L cen , R cen ), forming a replacement stereo audio signal; and wherein the replacement left and right audio signals (L cen , R cen ) is based on a weighted contribution of at least the target intermediate audio signal, each contribution being based on a respective alternative intermediate weighting factor, wherein for each time segment and each frequency band, each intermediate weighting factor is based on an alternative pan parameter (θ cen ) and alternative phase difference parameters (Ф cen )

[0156] EEE 9. The method according to EEE 8 further comprises: performing stereo signal processing on the substitute stereo audio signal to form a substitute left and right audio signal (L') including the processedcen , R' cen ) of the processed substitute stereo audio signal; and reconstructing a processed target intermediate audio signal (M'); wherein the processed target intermediate audio signal (M') is based on the processed substitute left and right audio signals (L') weighted by the weighting factor cen , R' cen ), where the weighting factors are based on the alternative pan parameters (θ) for each frequency band and time segment. cen ) and alternative phase difference parameters (Ф cen )

[0157] EEE 10. A method according to EEE 8 or 9, wherein the replacement phase parameter (θ cen ) and / or alternative phase difference parameters (Ф cen ) indicates the center pan.

[0158] EEE 11. The method according to any one of the aforementioned EEEs also includes: extracting (S31) a partial side signal representation (311, 312) for each time period and frequency band, wherein the partial side signal representation (311, 312) is based on a weighted difference between left and right audio signals (L, R), wherein the weight of each of the left and right audio signals (L, R) is based on at least one of a target sound phase parameter (Θ) and a target phase difference parameter (Ф) for each frequency band and time period; and forming (S41) a target side audio signal by combining each partial side signal representation for each frequency band and time period.

[0159] EEE 12. The method according to EEE 11 also includes: performing processing (S5, S51) on at least one of the target middle and target side audio signals (M, S) to form processed target middle and target side audio signals (M', S'); reconstructing (S6) the processed left and right audio signals (L', R') to form a processed stereo audio signal; wherein the processed left audio signal (L') is based on a weighted sum of the processed target middle audio signal and the target side audio signal (M', S'), wherein the weight is based on at least one of the target sound phase parameter (θ) and the target phase difference parameter (Ф), and wherein the processed right audio signal (R') is based on a weighted difference between the processed target middle audio signal and the target side audio signal (M', S'), wherein the weight is based on at least one of the target sound phase parameter (θ) and the target phase difference parameter (Ф).

[0160] EEE 13. A method according to EEE 12, wherein performing processing on at least one of a target middle audio signal and a target side audio signal (M, S) comprises at least one of: attenuating the target side audio signal (S); applying a gain to the target middle audio signal (M); performing monaural signal source separation on the target middle audio signal (M); and applying a stereo source separation filter to the target middle audio signal (M).

[0161] EEE 14. A method according to any one of the aforementioned EEEs, wherein obtaining at least one of a target phase parameter (θ) and a target phase difference parameter (φ) comprises: obtaining at least one of a plurality of detected phase parameters (θ) and a plurality of detected phase difference parameters (φ); for each detected phase parameter (θ) and a detected phase difference parameter (φ), obtaining a detected amplitude parameter (U); and determining at least one of the target phase parameter (θ) and the target phase difference parameter (φ) by calculating an average value of at least one of the plurality of detected phase parameters (θ) and a plurality of detected phase difference parameters (φ), wherein the average value is a weighted average value weighted by the detected amplitude parameter (U).

[0162] EEE 15. The method according to EEE 14 also includes: classifying each of the at least one of the multiple detected phase parameters (θ) and the multiple detected phase difference parameters (φ) into a predetermined number of intervals; wherein each interval is associated with at least one of the detected phase parameter value and the detected phase difference parameter value, and wherein the average value of at least one of the multiple detected phase parameters and the multiple detected phase difference parameters (φ) is based on the detected phase parameter value and the detected phase difference parameter value.

[0163] EEE 16. An audio processing system, configured to extract a target intermediate audio signal (M) from a stereo audio signal, the stereo audio signal comprising a left audio signal (L) and a right audio signal (R), the audio processing system comprising an extraction unit (12), configured to: obtain a plurality of continuous time periods of the stereo audio signal, wherein each time period comprises a representation of a portion of the stereo audio signal; for each of a plurality of frequency bands of each time period of the stereo audio signal, obtain at least one of the following parameters: a target phase parameter (θ), which represents the distribution of the amplitude ratio between the left and right audio signals (L, R) in the frequency band over the time period; and a target phase difference parameter (Φ) representing the distribution of the phase difference between the left and right audio signals (L, R) of the stereo audio signal over a time period; for each time period and each frequency band, extracting a partial intermediate signal representation (211, 212), wherein the partial intermediate signal representation (211, 212) is based on a weighted sum of the left and right audio signals (L, R), wherein the weight of each of the left and right audio signals (L, R) is based on at least one of the target phase parameter (Θ) and the target phase difference parameter (Φ) for each frequency band; and forming a target intermediate audio signal (M) by combining the partial intermediate audio signal representations for each frequency band and time period.

[0164] EEE 17. The audio processing system according to EEE 16, wherein the weights of the left and right audio signals (L, R) are based on the target pan parameter (θ) such that a left or right audio signal with a larger amplitude is set with a larger weight. ...

[0165] EEE 18. An audio processing system according to EEE 16 or EEE 17, wherein the weights of the left and right audio signals (L, R) are complex-valued weights, and wherein the phase of the complex-valued weights is based on a target phase difference parameter (Φ). ...

[0166] EEE 19. An audio processing system according to EEE 18, wherein the complex-valued weight is based on the target phase difference parameter (Φ) and the target sound phase parameter (Θ), so that for a left or right audio signal with a larger amplitude, the phase difference produced by applying the complex weight is smaller.

[0167] EEE 20. The audio processing system according to any one of EEE 16 to 19, further comprising: a stereo source separation system (60) configured to output a filter for stereo source separation, wherein a stereo audio signal is provided to the stereo source separation system (60), and the filter for stereo source separation is applied to the target intermediate audio signal to form a processed target intermediate audio signal (M sep ).

[0168] EEE 21. The audio processing system according to any one of EEEs 16 to 20, further comprising a mono source separation system (30) configured to perform audio source separation on a mono audio signal and output a processed mono audio signal, wherein the target intermediate audio signal (M) is provided to the mono source separation system (30), and the processed mono audio signal is used as the processed target intermediate audio signal (M sep ).

[0169] EEE 22. The audio processing system according to any one of EEEs 20 to 21, further comprising a reconstruction unit (20) configured to reconstruct processed left and right audio signals (L', R') to form a processed stereo audio signal; wherein each of the processed left and right audio signals (L', R') is based on a processed target intermediate audio signal (M) weighted with corresponding left and right weighting factors. sep ), wherein the left and right weighting factors are based on at least one of a target sound phase parameter (θ) and a target phase difference parameter (Φ) for each time segment and frequency band.

[0170] EEE 23. The audio processing system according to any one of EEEs 16 to 22, further comprising a reconstruction unit (20) configured to reconstruct the substitute left and right audio signals (L cen , R cen ), forming a replacement stereo audio signal; and wherein the replacement left and right audio signals (L cen , R cen ) is based on a weighted contribution of at least the target intermediate audio signal, each contribution being based on a respective alternative intermediate weighting factor, wherein for each time segment and each frequency band, each intermediate weighting factor is based on an alternative pan parameter (θ cen ) and alternative phase difference parameters (Ф cen )

[0171] EEE 24. An audio processing system according to any one of EEE 16 to 23, further comprising a stereo processor (50) configured to perform stereo signal processing on the substitute stereo audio signal to form a substitute left and right audio signal (L') comprising processed cen , R' cen ) of the processed substitute stereo audio signal; and a second extraction unit configured to extract a processed target middle audio signal (M'), wherein the processed target middle audio signal (M') is based on the processed substitute left and right audio signals (L') weighted by the weighting factor cen , R' cen ), where the weighting factors are based on the alternative pan parameters (θ) for each frequency band and time segment. cen) and alternative phase difference parameters (Ф cen )

[0172] EEE 25. An audio processing system according to EEE 23 or 24, wherein the replacement pan parameter (θ cen ) and / or alternative phase difference parameters (Ф cen ) indicates the center pan.

[0173] EEE 26. An audio processing system according to any one of EEEs 16 to 25, wherein the extraction unit (12) is further configured to extract (S31) a partial side signal representation (311, 312) for each time period and frequency band, wherein the partial side signal representation (311, 312) is based on a weighted difference between left and right audio signals (L, R), wherein the weight of each of the left and right audio signals (L, R) is based on at least one of a target phase parameter (Θ) and a target phase difference parameter (Φ) for each frequency band and time period; and to form a target side audio signal by combining each partial side signal representation for each frequency band and time period.

[0174] EEE 27. The audio processing system according to EEE 16 to 26, further comprising a mid / side processor (40) configured to perform processing (S5, S51) on at least one of the target mid and target side audio signals (M, S) to form processed target mid and target side audio signals (M', S'), and a reconstruction unit (20) configured to reconstruct the processed left and right audio signals (L', R') to form a processed stereo audio signal; wherein the processed left audio signal (L') is based on a weighted sum of the processed target mid audio signal and the target side audio signal (M', S'), wherein the weight is based on at least one of a target sound phase parameter (θ) and a target phase difference parameter (Φ), and wherein the processed right audio signal (R') is based on a weighted difference between the processed target mid audio signal and the target side audio signal (M', S'), wherein the weight is based on at least one of the target sound phase parameter (θ) and the target phase difference parameter (Φ).

[0175] EEE 28. An audio processing system according to EEE 27, wherein the processing performed by the mid / side processor (40) includes at least one of: attenuating a target side audio signal (S); applying a gain to a target middle audio signal (M); performing mono signal source separation on the target middle audio signal (M); and applying a stereo source separation filter to the target middle audio signal (M).

[0176] EEE 29. The audio processing system according to any one of EEEs 16 to 28 further includes a parameter extraction unit (14) configured to obtain at least one of a plurality of detected phase parameters (θ) and a plurality of detected phase difference parameters (φ); for each detected phase parameter (θ) and detected phase difference parameter (φ), obtain a detected amplitude parameter (U); and determine at least one of a target phase parameter (θ) and a target phase difference parameter (φ) by calculating an average value of at least one of a plurality of detected phase parameters (θ) and a plurality of detected phase difference parameters (φ), wherein the average value is a weighted average value weighted by the detected amplitude parameter (U).

[0177] EEE 30. An audio processing system according to EEE 29, wherein the parameter extraction unit (14) is further configured to classify each of the at least one of the multiple detected phase parameters (θ) and the multiple detected phase difference parameters (φ) into a predetermined number of intervals; wherein each interval is associated with at least one of the detected phase parameter value and the detected phase difference parameter value, and wherein the average value of the at least one of the multiple detected phase parameters and the multiple detected phase difference parameters (φ) is based on the detected phase parameter value and the detected phase difference parameter value.

[0178] EEE 31. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to any one of EEE 1-15. EEE 32.

[0179] EEE 32. An audio processing device according to the method described in EEE 2 or according to any one of EEE 3-15 dependent on EEE 2, or according to EEE 17 or according to any one of EEE 18-30 dependent on EEE 17, wherein the amplitude of each of the left and right audio signals is determined, and in response to determining that the amplitude of the left audio signal is greater than the amplitude of the right audio signal, the weight of the left audio signal is determined to be greater than the weight of the right audio signal, and in response to determining that the amplitude of the right audio signal is greater than the amplitude of the left audio signal, the weight of the right audio signal is determined to be greater than the weight of the left audio signal.

[0180] EEE 33. An audio processing device according to EEE 4 or according to any one of EEE 5-15 depending on EEE 4, or according to any one of EEE 20-30 depending on EEE 19, wherein the amplitude of each of the left and right audio signals is determined, and wherein in response to determining that the amplitude of the left audio signal is greater than the amplitude of the right audio signal, a weight of the left audio signal that causes a first phase shift when applied to the left audio signal is determined, and a weight of the right audio signal that causes a second phase shift when applied to the right audio signal is determined, and in response to determining that the amplitude of the right audio signal is greater than the amplitude of the left audio signal, a weight of the right audio signal that causes a first phase shift when applied to the right audio signal is determined, and a weight of the left audio signal that causes a second phase shift when applied to the left audio signal is determined, and wherein the first phase shift is less than the second phase shift.

[0181] EEE 34. An audio processing device according to the method described in EEE 8 or according to any one of EEE 9-15 depending on EEE 8, or according to any one of EEE 24-30 depending on EEE 23 or according to EEE 23, wherein the replacement left and right audio signals are second left and right audio signals respectively, and / or wherein the replacement stereo signal is a second stereo signal.

[0182] EEE 35. A method according to EEE 9 or 10 or according to any one of EEE 11-15 depending on EEE 9 or 10, or wherein the replacement stereo audio signal (L cen ,R cen ) performs stereo signal processing including replacing the stereo audio signal (L cen ,R cen ) applies source separation to output processed alternative left and right audio signals (L' cen ,R' cen ).

Claims

1. A computer-implemented method for extracting a target intermediate audio signal (M) from a stereo audio signal, the stereo audio signal comprising a left audio signal (L) and a right audio signal (R), the method comprising: obtaining (S1) a plurality of consecutive time segments of a stereo audio signal, wherein each time segment comprises a representation of a portion of the stereo audio signal; For each frequency band of the plurality of frequency bands of each time segment of the stereo audio signal, at least one of the following parameters is obtained (S2): a target pan parameter (θ) representing the distribution of the amplitude ratio between the left and right audio signals (L, R) in the frequency band over a time period; as well as a target phase difference parameter (Φ) representing a distribution of a phase difference between left and right audio signals (L, R) of a stereo audio signal over a time period; extracting (S3) for each time segment and each frequency band a partial intermediate signal representation (211, 212), wherein the partial intermediate signal representation (211, 212) is based on a weighted sum of left and right audio signals (L, R), wherein a weight of each of the left and right audio signals (L, R) is based on at least one of a target pan parameter (θ) and a target phase difference parameter (Φ) for each frequency band; as well as A target intermediate audio signal (M) is formed (S4) by combining the partial intermediate audio signal representations (211, 212) for each frequency band and time segment.

2. The method according to claim 1, wherein the weights of the left and right audio signals (L, R) are based on the target pan parameter (θ) such that a left or right audio signal with a larger amplitude is assigned a larger weight.

3. The method according to claim 1 or 2, wherein the weights of the left and right audio signals (L, R) are complex-valued weights, and wherein the phase of the complex-valued weights is based on a target phase difference parameter (Φ).

4. The method of claim 3, wherein the complex-valued weight is based on the target phase difference parameter (Φ) and the target pan parameter (θ) such that for a left or right audio signal with a larger amplitude, the phase difference produced by applying the complex weight is smaller.

5. The method according to any one of the preceding claims, further comprising: providing a stereo audio signal to a stereo source separation system (60), the stereo source separation system (60) being configured to output a filter for stereo source separation; and The filter is applied to the target intermediate audio signal to form a processed target intermediate audio signal (M sep ).

6. The method according to any one of the preceding claims, further comprising: providing a target intermediate audio signal (M) to a mono source separation system (30), the mono source separation system (30) being configured to perform audio source separation on the mono audio signal and output a processed mono audio signal; and The processed mono audio signal is used as the processed target intermediate audio signal (M sep ).

7. The method according to any one of claims 5-6, further comprising: reconstructing the processed left and right audio signals (L', R') to form a processed stereo audio signal; wherein each of the processed left and right audio signals (L', R') is based on a processed target intermediate audio signal (M) weighted with respective left and right weighting factors sep ), wherein the left and right weighting factors are based on at least one of a target sound phase parameter (θ) and a target phase difference parameter (Φ) for each time segment and frequency band.

8. The method according to any one of the preceding claims, further comprising: Reconstruct the replacement left and right audio signals (L cen , R cen ), forming a substitute stereo audio signal; and The replaced left and right audio signals (L cen , R cen ) is based on a weighted contribution of at least the target intermediate audio signal, each contribution being based on a respective alternative intermediate weighting factor, wherein for each time segment and each frequency band, each intermediate weighting factor is based on an alternative pan parameter (θ cen ) and alternative phase difference parameters (Ф cen ) 9. The method according to claim 8, further comprising: Performing stereo signal processing on the substitute stereo audio signal to form a signal including processed substitute left and right audio signals (L' cen , R' cen ) a processed substitute stereo audio signal; and reconstructing the target intermediate audio signal (M”); where the processed target intermediate audio signal (M') is based on the processed substitute left and right audio signals (L') weighted by weighting factors cen , R' cen ), where the weighting factors are based on alternative pan parameters (θ) for each frequency band and time segment cen ) and alternative phase difference parameters (Ф cen ) 10. The method according to claim 8 or 9, wherein the replacement pan parameter (θ cen ) and / or alternative phase difference parameters (Ф cen ) indicates the center pan.

11. The method according to claim 9 or 10, further comprising: For alternative stereo audio signals (L cen ,R cen ) performs stereo signal processing including replacing the stereo audio signal (L cen ,R cen ) applies source separation to output processed alternative left and right audio signals (L' cen ,R' cen ).

12. The method according to any one of the preceding claims, further comprising: extracting (S31) for each time segment and frequency band a partial side signal representation (311, 312), wherein the partial side signal representation (311, 312) is based on a weighted difference between left and right audio signals (L, R), wherein the weight of each of the left and right audio signals (L, R) is based on at least one of a target pan parameter (θ) and a target phase difference parameter (Φ) for each frequency band and time segment; and A target side audio signal is formed (S41) by combining each partial side signal representation of each frequency band and time segment.

13. The method according to claim 12, further comprising: performing processing (S5, S51) on at least one of the target mid and target side audio signals (M, S) to form processed target mid and target side audio signals (M', S'); reconstructing (S6) the processed left and right audio signals (L', R') to form a processed stereo audio signal; wherein the processed left audio signal (L') is based on a weighted sum of processed target mid audio signals and target side audio signals (M', S'), wherein the weights are based on at least one of a target pan parameter (Θ) and a target phase difference parameter (Φ), and The processed right audio signal (R') is based on a weighted difference between processed target mid audio signals and target side audio signals (M', S'), wherein the weight is based on at least one of a target pan parameter (Θ) and a target phase difference parameter (Φ).

14. The method of claim 13 , wherein performing processing of at least one of the target mid audio signal and the target side audio signal (M, S) comprises at least one of: Attenuate the target side audio signal (S); applying a gain to a target intermediate audio signal (M); performing mono source separation on the target intermediate audio signal (M); and A stereo source separation filter is applied to the target intermediate audio signal (M).

15. The method according to any one of the preceding claims, wherein obtaining at least one of a target acoustic phase parameter (θ) and a target phase difference parameter (Φ) comprises: Obtain multiple detected sound phase parameters (θ) and multiple detected phase difference parameters At least one of; For each detected phase parameter (θ) and detected phase difference parameter obtaining a detected amplitude parameter (U); and By calculating multiple detected sound phase parameters (θ) and multiple detected phase difference parameters The target phase parameter (θ) and the target phase difference parameter are determined by taking the average value of at least one of At least one of the above, the average value is a weighted average value weighted by the detected amplitude parameter (U).

16. The method according to claim 15, further comprising: The multiple detected sound phase parameters (θ) and the multiple detected phase difference parameters into a predetermined number of intervals; Each interval is associated with at least one of a detected phase parameter value and a detected phase difference parameter value, and wherein a plurality of detected phase parameters and a plurality of detected phase difference parameters are The average value of at least one of is based on the detected sound phase parameter value and the detected phase difference parameter value.

Citation Information

Cited By

  • Audio processing method and device, electronic equipment and computer readable storage medium

    CN121148414A

  • Audio processing method and device, electronic equipment and computer readable storage medium

    CN121148414B