Separating Panned Sources from Generalized Stereo Backgrounds with Minimal Training

Abstract

In an embodiment, a spatial level filter (SLF) is generated by obtaining a first sample set from a plurality of target source levels and spatial distributions within frequency subbands in the frequency domain, obtaining a second sample set from a plurality of background levels and spatial distributions within the frequency subbands in the frequency domain, adding the first sample set and the second sample set to create a combined sample set, detecting levels and spatial parameters of each sample in the combined sample set for each subband within the frequency domain, weighting the detected levels and spatial parameters by the levels and spatial distributions of the target sources and backgrounds, storing the weighted levels, spatial parameters, and signal-to-noise ratios (SNRs) within the subbands of each sample in the combined sample set in a table, and re-indexing the table by the weighted levels, spatial parameters, and subbands.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 038,046, filed June 11, 2020, and European Patent Application No. 20179449.2, filed June 11, 2020, which applications are incorporated herein by reference.

[0002] [Technical field] The present disclosure relates generally to audio signal processing, and more particularly to audio source separation techniques. [Background technology]

[0003] A two-channel audio mix (e.g., a stereo mix) is created by mixing multiple audio sources together. There are several examples where it may be desirable to detect and extract individual audio sources from a two-channel mix, including but not limited to: remix applications where audio sources are rearranged into a two-channel mix, upmixing applications where audio sources are placed or rearranged into a surround sound mix, audio source expansion applications where certain audio sources (e.g., speech / dialogue) are boosted back into the two-channel or surround sound mix, etc. Summary of the Invention

[0004] Details of the disclosed implementations are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims.

[0005] In an embodiment, the method comprises: obtaining, with one or more processors, a frequency domain representation of the first sample set from a plurality of target levels and spatial distributions of a plurality of frequency subbands; obtaining, with the one or more processors, a frequency domain representation of a second sample set from a plurality of background levels and spatial distributions of the plurality of frequency subbands; adding, with the one or more processors, the first sample set and the second sample set to generate a combined sample set; detecting, with the one or more processors, level and spatial parameters for each sample of the combined sample set for each subband of the plurality of frequency subbands; weighting, within each of said plurality of frequency subbands, the detected level and spatial parameters according to their respective levels and spatial distributions of the target source and the background; storing, using the one or more processors, a table of weighted levels and spatial parameters and a signal-to-noise ratio (SNR) within the plurality of frequency subbands for each sample of the combined sample set; using the one or more processors, re-indexing the table with the weighted level and spatial parameters and sub-bands, so that the table contains target percentile SNRs for the weighted level and spatial parameters and sub-bands, and for a given input of quantized detected spatial and level parameters and sub-bands, an estimated SNR associated with the quantized detected spatial and level parameters and sub-bands is obtained from the table; Includes.

[0006] In an embodiment, the method further comprises smoothing the data indexed to one or more detected levels, one or more spatial parameters or frequency sub-bands.

[0007] In an embodiment, the frequency domain representation is a Short Time Fourier Transform (STFT) domain representation.

[0008] In an embodiment, the spatial parameters include panning and phase difference between two channels of a mixed audio signal.

[0009] In an embodiment, the target source is amplitude panned using a constant power law.

[0010] In an embodiment, the target percentile SNR is the 25th percentile.

[0011] In an embodiment, the method comprises: using one or more processors to convert one or more frames of a two-channel time domain audio signal into a time-frequency domain representation comprising a plurality of time-frequency tiles, the frequency domain of the time-frequency domain representation comprising a plurality of frequency bins grouped into a plurality of subbands; For each time-frequency tile, calculating, using the one or more processors, spatial parameters and levels of time-frequency tiles; generating, using the one or more processors, a percentile signal-to-noise ratio (SNR) for each frequency bin of a time-frequency tile; generating, using the one or more processors, a fractional value for the bin based on the SNR for the bin; applying, using the one or more processors, the fractional values ​​of the bins of the time-frequency tiles to generate modified time-frequency tiles of the estimated audio source; Includes.

[0012] In an embodiment, the frames of the time-frequency tiles are assembled into chunks, each chunk including a plurality of subbands, and the method comprises, for each subband in each chunk: calculating, using one or more processors, spatial parameters and levels for each time-frequency tile in the chunk; generating, using the one or more processors, a percentile signal-to-noise ratio (SNR) for each frequency bin of a time-frequency tile; generating, using the one or more processors, a fractional value for the bin based on the SNR for the bin; applying, using the one or more processors, the fractional values ​​of the bins of the time-frequency tiles to generate modified time-frequency tiles of the estimated audio source; Includes.

[0013] In an embodiment, the method includes converting, using the one or more processors, the modified time-frequency tiles into a plurality of time-domain audio source signals.

[0014] In an embodiment, the spatial parameters include panning and phase difference between channels for each time-frequency tile.

[0015] In an embodiment, the fractional value is obtained from a look-up table or function of a spatial level filtering (SLF) system that is trained on the panned target source.

[0016] In an embodiment, transforming one or more frames of a two-channel time domain audio signal into a frequency domain signal comprises applying a short-time frequency transform (STFT) to said two-channel time domain audio signal.

[0017] In an embodiment, multiple frequency bins are grouped into octave or near-octave sub-bands.

[0018] Certain embodiments disclosed herein provide one or more of the following advantages: The disclosed embodiments allow for the extraction of a target source from a recording of a mix consisting of a source and some background (source separation). More specifically, the disclosed embodiments allow for the extraction of sources that have been mixed (purely or mostly) using amplitude panning, which is the most common way dialogue is mixed in television and movies. The ability to extract such sources allows for dialogue enhancement (extracting and then boosting the dialogue in the mix) or upmixing. Furthermore, high-quality estimates of the sources may be extracted with little to no training data or delay, a distinct feature from most other approaches to source separation. [Brief description of the drawings]

[0019] In the accompanying drawings referenced below, various embodiments are illustrated in block diagrams, flow charts, and other diagrams. Each block in a flow chart or block may represent a module, program, or portion of code that includes one or more executable instructions for performing a specified logical function. Although these blocks are shown in a specific order for performing steps of a method, they do not necessarily have to be performed in the exact order shown. For example, they may be performed in reverse order or simultaneously, depending on the characteristics of each operation. It should be noted that each block in the block diagrams and / or flow charts, and combinations thereof, may be implemented by a dedicated software-based or hardware-based system that performs the specified functions / operations, or by a combination of dedicated hardware and computer instructions.

[0020] [Figure 1] 1 illustrates a signal model for source separation that accounts for time-domain mixing, according to an embodiment.

[0021] [Diagram 2]FIG. 1 is a block diagram of a system for generating a spatial level filter (SLF) lookup table trained to extract panned sources according to an embodiment.

[0022] [Diagram 3] 1 is a visual representation of the inputs and outputs of an SLF lookup table trained to extract panned sources according to an embodiment.

[0023] [Figure 4] FIG. 1 is a block diagram of a system for detecting and extracting spatially distinct sub-band audio sources from a two-channel mix using an SLF trained to extract panned sources, according to an embodiment.

[0024] [Diagram 5] FIG. 1 is a flow diagram of a process for generating an SLF lookup table trained to extract panned sources according to an embodiment.

[0025] [Figure 6] FIG. 1 is a flow diagram of a process for detecting and extracting spatially distinct sub-band audio sources from a two-channel mix using an SLF trained to extract panned sources, according to an embodiment.

[0026] [Figure 7] FIG. 7 is a block diagram of an apparatus architecture for implementing the systems and processes described with reference to FIGS. 1-6 according to an embodiment.

[0027] The same reference numbers used in the various drawings indicate similar elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Signal Models and Assumptions FIG. 1 shows a signal model 100 for source separation that accounts for time-domain mixing, according to an embodiment. The signal model 100 separates the target source s1 and the background b into the “left channel” (x1 or X2) depending on the context. L ) and "Right Channel" (x2 or X R Consider a fundamental time domain mixing into two channels, called , which are input to a source separation system 101 that estimates:

number

[0029] The target source s1 is assumed to be amplitude panned using a constant power law. Other panning laws can be converted to a constant power law, so the use of a constant power law in the signal model 100 is not limited. In constant power law panning, the source s1, mixed into the left / right (L / R) channels is described as:

number

number

[0030] [Continuing in the STFT domain, the addition of background B to each channel is expressed as:

number

[0031] For background B, additional parameters ∠B and φ BThese parameters, which include , respectively describe the phase difference between the left channel phase of S1 and B, and the inter-channel phase difference between the phase of B in the left and right channels of the STFT space. Note that the inter-channel phase difference of a panned source is zero by definition, so φ is not included in equations [5] and [6]. S1 No parameters need to be included. The target S1 and background B are assumed to share no particular topological relationship in the STFT space, so the distribution of ∠B is modeled as uniform.

[0032] There is an important spatial difference between the target source and the background. Spatially, Θ1 is treated as a specific single value (the "panning parameter" of the target source S1), but Θ B and Φ B Since each of the vectors has a statistical distribution, a statistical model (e.g., a Bayesian model) can be used to perform source separation.

[0033] To examine this case, the "target source" is assumed to be panned, meaning that it can be characterized by Θ1. The interchannel phase difference of the target source is assumed to be 0. Also, its level L S = |S1| is also assumed to have a distribution and to be known over at least approximately the octave subbands. The spatial information is assumed to be fully specified by the panning parameters of the sources.

[0034] Background B is Θ B The inter-channel phase difference φ B It is also characterized as being distributed above the background level L B =|B| also has a distribution and is assumed to be known at least over approximately the octave subbands.

[0035] For the purposes of this model, the source and background shall only be modeled at the point where both are assumed to be "active". In this sense, the current source of interest and the background shall always be assumed to be "on" or "off", and the separation shall assume that both the target source and the background are "on". It can be shown that extraction is nearly perfect even when the target source is active but the background is not. If the target source and panning parameters are unknown, they can be estimated using techniques known to those skilled in the art. In some cases, such as mostly music, there may be a harmonic relationship between the target source and the background. Such a relationship is not modeled separately in the signal model 100. The distribution is assumed to include some degree of overlap of harmonics that is appropriate for the particular application.

[0036] Training Process 2 is a block diagram of a system 200 for generating an SLF lookup table trained to extract panned sources, according to an embodiment. An SLF is a system trained to extract a target source with a given level distribution and specified spatial parameters from a mix that includes a background with the given level distribution and spatial parameters.

[0037] The system 200 includes a target source parameter database 201, a target source distribution sampler 202, a transform unit 203, a parameter detector 204, a reindex unit 205, a target SNR selector 206, a trained SLF lookup table 207, a background parameter database 208, a background distribution sampler 209, and a transform unit 210. Although the distribution samplers 202, 209 and the transform units 203, 210 are shown as separate blocks in Figure 2, in practice the samplers 202, 209 and the transform units 203, 210 may be combined into a single module (e.g., a software module) that operates on both the target source and background databases 201, 208.

[0038] The goal of the training procedure implemented by system 200 is to create a Bayesian model that predicts the relative fraction of energy that belongs to a target source in each STFT domain bin or tile, given a two-channel input (e.g., an L / R stereo input). To achieve this goal, four parameters are used that are detectable for the two-channel input in the STFT domain.

[0039] The first parameter is b, which represents a frequency subband of roughly an octave. This parameter is obtained by a trivial mapping from a given frequency bin ω to the subband b it belongs to. Examples of subband boundaries are shown below:

[0040] The second parameter is the detected “panning” for each (ω,t) tile, defined as:

number

[0041] The third parameter is the "phase difference" detected for each tile, which is defined as:

number

[0042] The fourth parameter is the "level" detected for each tile, defined as:

number

[0043] Each frequency bin ω is understood to represent a particular frequency. However, data may also be grouped into subbands, which are collections of contiguous bins, with each frequency bin ω belonging to a subband. Grouping data into subbands is particularly useful for certain estimation tasks performed by the system. In an embodiment, octave or near-octave subbands are used, although other subband definitions may be used. An example of banding includes defining band edges as follows, with values ​​shown in Hz:

number

[0044] Note that if the definition of "octave" is followed strictly, there could be an infinite number of bands where the lowest band approaches infinitesimal width, so some choice must be made to allow a finite number of subbands. In an embodiment, the lowest band is chosen to be the same size as the second band, but other conventions may be used in other embodiments. Herein, the terms "subband" and "band" may be used interchangeably.

[0045] To understand how to build a Bayesian system based on these four parameters, first recall Bayes' rule:

number

[0046] In this case, the goal of the training process is to be able to estimate the distribution over the SNR of each spectrogram tile, given several observations, b, Θ, φ, and U, mentioned above. Bayes' rule is given by:

number

[0047] Our goal is to train a Bayesian system that can generate all quantities on the right hand side of equation

[11] , and be able to estimate the quantities on the left hand side of equation

[11] . To do so, we estimate p(SNR) by considering the level distribution over the target source over the background.

[0048] The conditional probability of p(b,Θ,φ,U|SNR) is estimated from the distribution of parameters (Θ,φ,U) in each frequency band b when target and background are mixed at various SNRs. The procedure for generating this data involves generating many samples of data from databases 201, 208 for each target source and background by sampling from known or assumed spatial and level distributions using distribution samplers 202, 209. Transformers 203, 210 create STFT domain values ​​with the properties of the samples.

[0049] Recall that the training procedure described here explicitly specifies the panning parameters of the target source to be extracted later, since the target source is assumed to have specific panning parameters. In the example embodiment described here, the target source is assumed to have Θ1=π / 4, which corresponds to a central panned source. When generating the training data, it is assumed that there is a random phase relationship between the target and the background, as mentioned above. In practice, this can be implemented by setting one phase value to 0 and the other to various samples on the unit circle.

[0050] To create the training data, the frequency domain representations output by the transform modules 203, 210 are summed (as shown in the signal model 100 of FIG. 1) to create a joint frequency domain representation. Note that when performing Bayesian training, there will be a very large number of combinations of target and background data items. However, this large number of combinations translates into a much smaller number of evenly quantized target-to-background ratios.

[0051] To efficiently exploit this reality, the training process creates separate sets of uniformly sampled data for each of the target and background SNR (0 to 37 dB, but a larger range can be chosen), target and background phase difference (0 to 2π), background Θ (0 to π / 2), and background φ magnitude (0 to π). For every possible combination of this data, the training process computes the detected (Θ,φ,U) values ​​and stores them in storeThetaHat, storePhiHat, and storeUdBHat, respectively. Note that these calculations do not yet take into account the specific spatial and level distributions of each of the target and background. These are merely lookup tables that map from all possible combinations of the relevant input attributes to the detected Θ, φ, and U. Using these tables improves efficiency later in the training process.

[0052] Next, the specific spatial and level data of the target and background are incorporated. Recall that the goal is to obtain p(b, Θ, φ, U|SNR). In practice, the distribution on each variable (Θ, φ, U) can be represented by a quantized probability density function (pdf), and the SNR can also be quantized. In an embodiment, a quantization of 51 levels for the magnitude of φ (0 to π), a quantization of 51 levels for Θ (0 to π / 2), 1 dB increments for U (e.g. range 0 to 127 dB), and 1 dB increments for DNR (e.g. range -40 dB to +60 dB) is used. Given such a quantization, the information p(b, Θ, φ, U|SNR) can be stored in a multidimensional array "storePopularity" of size: 7 bands, 101 trained SNRs (-40 to 60), 51 Θ bins, 51 φ bins, 128 dB levels (e.g., 0 to 127). For each item, the value stored in the array represents the probability (or, equivalently, "popularity") of that particular combination relative to other combinations in the array. For example, the array elements (4, 49, 26, 26, 90) represent how "popular" it is that, for frequency band 4 and +8 dB DNR (49th value), a detected Θ value of π / 4 (26th value), a φ magnitude value of π / 2 (26th value), and a level U of 89 dB (90th value).

[0053] To obtain p(b, Θ, φ, U|SNR), the training process cycles exhaustively (or via sampling) through all possible combinations of target and source spatial and level data. Then, when a particular SNR, phase difference, background Θ, and background φ are observed in the training data, the data previously stored in storeThetaHat, storePhiHat, and storeUdBHat are used to search for the resulting Θ, φ, and U, respectively, to reduce the training computations. This search is also called "parameter discovery" and is performed by block 204 in FIG. 2. Importantly, the popularity of each such combination, as indicated by the values ​​of the spatial and level distributions on the target and background, is also used. These weight their contribution to the storePopularity array, incorporating p(SNR) as appropriate. By cycling through all such combinations and recording their popularity, the storePopularity array described above is created. This array may be sparse or noisy, so it shall be smoothed using techniques familiar to those skilled in the art. An example technique is smoothing over one or more dimensions of the table.

[0054] At this stage, the data required for Bayesian analysis is obtained, but not provided in the desired lookup table or function form. The final step in the training process is to obtain a usable p(SNR|b,Θ,φ,U) from the data p(b,Θ,φ,U|SNR)p(SNR) in storePopularity of the following size: 7 bands, 101 trained SNRs (-40 to 60), 51 Θ bins, 51 φ bins, 128 dB levels (e.g., 0 to 128). To understand how this corresponds to p(b,Θ,φ,U|SNR)p(SNR), recall that p(b,Θ,φ,U|SNR)p(SNR) can be expressed equally as p(b,Θ,φ,U,SNR) or equivalently as p(b,SNR,Θ,φ,U). The five indices are the same as those in storePopularity.

[0055] This reindexing or remapping is done by blocks 205 and 206 in FIG. 2. Note that the objective p(SNR|b, Θ, φ, U) is not a single set of values, but a set of distributions over the SNR given some detected (Θ, φ, U) for each frequency band b. A decision is made on how to succinctly describe these distributions to keep the representation size manageable. Typical ways include taking the mean, median, or other parameters. Considering the needs of the practical application for which this system is designed, the 25th and 50th percentiles of each SNR distribution are used in the embodiment.

[0056] To obtain p(SNR|b, Θ, φ, U), the training process operates to perform reindexing (block 205) and target SNR selection (block 206). The basic goal is to assemble and characterize all SNR data from storePopularity that correspond to a given (Θ, φ, U) triplet detected in frequency band b. Since frequency bands are treated independently, this is equivalent to considering the goal to perform N separate exercises to find p(SNR|Θ, φ, U) for each of the N frequency bands. Block 205 performs this task by cycling through all frequency bands and all sampled distribution levels for the variables detected Θ, detected φ, and detected level. For each such value, a buffer is created from storePopularity that contains all SNRs and how popular they are given a particular combination of detected Θ, φ, and U values. More specifically, the buffer is a subset of storePopularity as follows: storePopularitySmoothed(band index, (all data), Θ index, φ index, U index). The next block 206 analyzes the buffer of values ​​and, in an embodiment, finds and records the 25th and 50th percentile values ​​of the trained SLF lookup table (207). Specifically, these values ​​are each recorded in new arrays, percentile25SNRvalues ​​and percentile50SNRvalues, respectively, each indexed by (band index, find Θ index, find φ index, find U index), which is in fact the desired representation for p(SNR|b, Θ, φ, U).

[0057] Since the training data may be sparse, some buffers where the percentile SNR is calculated may have too few data points to generate a reliable percentile SNR value. To address this, two exemplary approaches can be used, although other approaches can also be used. One approach is to share data from adjacent frequency bands, Θ values, φ values, or U values ​​(with a preference for sharing frequency bands and U levels) before calculating the percentile SNR. Another approach is to calculate the percentile SNR even from sparse data, and if it appears unstable, replace or smooth the percentile SNR value with the SNR value from the adjacent U value, or the SNR value from the frequency band as appropriate.

[0058] At this stage, the reindexing is complete and we now turn to the application of the trained system. The system has a reindexed table, whose indices represent the quantized values ​​of Θ, φ, U, as well as the index b of the frequency band of interest. To obtain the soft mask values ​​using such a table, the function quantizes the input Θ, φ, U values ​​to 51, 51, and 128 levels, respectively. The conversion from the detected Θ, φ, U values ​​to their indices is trivial and follows the same quantization used when creating the quantized distributions above. The function accesses the values ​​in the table that correspond to these quantized index levels (and the index of the frequency band b that corresponds to the frequency bin ω of interest).

[0059] Note that in this case, although the percentile25SNRvalues ​​and percentile50SNRvalues ​​are obtained from a table with a particular index, the SNR values ​​may in fact be given by a more general function that takes any (not necessarily quantized) values ​​of Θ, φ, U, b. In fact, a function that seeks to obtain softmask values ​​from Θ, φ, U, b does not need to access a table to output the softmask values. It can directly calculate the softmask values ​​using curves or generic functions (including trained neural networks) that approximate and / or interpolate the values ​​in the table. It can be readily seen from inspection of FIG. 3 (a representation of a 25th percentile SNR system) that a curve can be fit to the data represented in the table. In embodiments that use a table, it is understood that the table is not the limiting method for obtaining the softmask values, but rather a recommended efficient method for doing so. Functions such as those resulting from least squares curve fitting or neural networks that approximate or interpolate tables can be constructed using techniques familiar to those skilled in the art.

[0060] FIG. 3 is a visual representation of the inputs and outputs of an SLF lookup table trained to extract panned sources according to an embodiment. More specifically, FIG. 3 shows a visual representation of a 25th percentile four-dimensional (4D) SLF lookup table trained on a central panned target source as described with reference to FIG. 2. The SLF lookup table is large, but also highly repetitive. Techniques familiar to those skilled in the art can be used to reduce the lookup time and memory required to store information in this table (e.g., entropy coding) or to convert the information in the table into a continuous function, as described above.

[0061] As mentioned before, the visual representation in Figure 3 is in 4D. The four input variables are the modified left-right Θ and input-output φ axes for each subplot, as well as the vertical (subband b) and horizontal (level U) subplot indices. Note that for practical reasons, the horizontal subplot dimension (level U) does not display all levels stored in the SLF lookup table. In this case, 1 dB increments are used over the 128 dB range in the table, thus requiring 128 left-right subplots. In practice, finer or coarser increments could be used to increase precision or improve search efficiency, respectively. Looking at Figure 3, note that there are many "not displayed" subplots from left to right.

[0062] The output variable of an SLF lookup table is a softmask value between 0 and 1 inclusive, displayed on the vertical axis of each subplot. The softmask value represents the fraction of the corresponding input STFT that is passed to the output. Since there is one (four-dimensional) input per STFT tile, there is also one output per STFT tile. The result of applying an SLF table / function is an STFT-sized representation consisting of values ​​between 0 and 1.

[0063] As mentioned above, the soft mask values ​​generated by percentile25SNRvalues ​​or percentile50SNRvalues ​​can be used, although other percentiles can also be used. In general, using percentile25SNRvalues ​​results in a source separation solution that balances including some background in the source estimate and introducing some artifacts. Using percentile50SNRvalues ​​leads to a solution with fewer artifacts but more background. The application of the soft mask parameters is illustrated in block 404 of FIG. 4.

[0064] In an embodiment, the soft mask values ​​and / or the signal values ​​are smoothed over time and frequency using techniques familiar to those skilled in the art. Assuming a 4096 point FFT, a smoothing vs. frequency using a smoother [0.17 0.33 1.0 0.33 0.17] / sum([0.17 0.33 1.0 0.33 0.17]) can be used. For higher or lower FFT sizes, some reasonable scaling of the smoothing range and coefficients needs to be performed. Assuming a sample hop size of 1024, a smoothing vs. time of approximately [0.1 0.55 1.0 0.55 0.1] / sum([0.1 0.55 1.0 0.55 0.1]) can be used. Changing the hop size or frame length can adjust the smoothing appropriately.

[0065] Exemplary Applications FIG. 4 is a block diagram of a system for detecting and extracting spatially distinct sub-band audio sources from a two-channel mix using SLF, according to an embodiment. The system 400 includes a transform unit 401, a parameter calculator 402, a table lookup unit 403, a soft mask applicator 404, and an inverse transform unit 405. The table lookup unit 403 operates on a database 406 that stores SLF lookup tables trained to detect panned sources, as described with reference to FIG. 2. In this example application, it is assumed that the target source to be extracted has known panning parameters, or that detection of such parameters is performed using any number of techniques known to those skilled in the art. One example technique for detecting panning parameters is peak picking from a level-weighted histogram of theta values.

[0066] Referring to FIG. 4, a transform unit 401 is applied to a two-channel input signal (e.g., a stereo mix signal). In an embodiment, the system 400 uses STFT parameters including window type and hop size that are known to those skilled in the art to be relatively optimal for the source separation problem. However, other STFT parameters may be used. A parameter calculator 402 calculates the values ​​of parameters (Θ, φ, U) for each octave subband b from the STFT representation. These values ​​are used by a table lookup unit 403 to perform a table lookup against an SLF lookup table stored in a database 406. The table lookup generates a percentile SNR (e.g., 25th percentile) for each STFT tile or bin. From the SNR, the system 400 calculates the percentage of STFT inputs to be output as a Bayesian estimate. For example, if the estimated percentile SNR is 0 dB, the percentage of inputs passed will be 0.5 or 50% since the target source and background are estimated to have the same level U. The general formula follows the Wiener filter assumption: Input fraction = 10^(SNR / 20) / (10^(SNR / 20)+1). The soft mask applicator 404 then multiplies the input STFT of each channel by this fractional value, which ranges from 0 to 1 for each STFT tile. The inverse transform unit 405 then inverts the STFT representation to obtain a two-channel time-domain signal representing the estimated target source.

[0067] Although the above example embodiment used a STFT time-frequency representation (eg, tiles), any suitable time-frequency representation may be used.

[0068] Although the above example source separation application used an SLF lookup table, in other embodiments an SLF function can be used in place of the lookup table.

[0069] Exemplary Processing 5 is a flow diagram of a process 500 for generating an SLF lookup table trained to extract panned sources according to an embodiment. The process 500 can be implemented, for example, by the device architecture 700 described with reference to FIG.

[0070] The process 500 first obtains a frequency domain representation of samples from the target source level and spatial distribution in the subband (501), obtains a frequency domain representation of samples from the background level and spatial distribution (502), and adds the first and second sample sets to create a combined sample set (503), as described with reference to FIG. 2.

[0071] The process 500 then detects (504) the level and spatial parameters of each sample in the combined sample set for each subband, and weights (505) the detected level and spatial parameters by the level and spatial distribution of the target source and background, respectively, within each subband, as described with reference to FIG.

[0072] The process 500 continues by storing (506) each sample in the combined sample set, its weighted level and spatial parameters, and its SNR along with the subband in a table, as described with reference to FIGS.

[0073] The process 500 then re-indexes the stored parameters and SNRS so that the table contains the weighted level and spatial parameters and the target percentile SNR for the subband, and for a given input of the quantized detected spatial and level parameters and subband, the estimated SNR associated with the quantized detected spatial and level parameters and subband (507), as described with reference to Figures 2 and 3. The SLF look table is then stored in a database for use in source separation applications such as those described with reference to Figures 4 and 6.

[0074] 6 is a flow diagram of a process 600 for detecting and extracting spatially distinct sub-band audio sources from a two-channel mix using an SLF trained to detect panned sources, according to an embodiment. The process 600 can be implemented, for example, by the device architecture 700 described with reference to FIG.

[0075] The process 600 first converts a two-channel time-domain audio signal into a frequency-domain representation that includes time-frequency tiles with multiple frequency bins grouped into subbands (601). For example, an STFT can be used to create an STFT representation of each channel of the two-channel time-domain audio signal.

[0076] The process 600 continues by calculating the spatial and level parameters for each frequency bin 602. For example, the parameters (Θ, φ, U) can be calculated using equations [7]-[9].

[0077] The process 600 continues by generating, for each tile, a percentile SNR for each frequency bin in the tile (603), generating a fractional value for the frequency bin based on the SNR of the frequency bin (604), and applying the fractional value to each frequency bin in the tile to generate a modified tile of the estimated audio source (605), as described with reference to FIG. 4. The SLF lookup table / function is trained to detect panned sources, as described with reference to FIGS. 2 and 5. The fractional values, also referred to herein as soft mask values, are real numbers between 0 and 1 inclusive, and represent a fraction of the corresponding input STFT that is passed to the output. The result of applying the SLF table / function is a representation of the STFT size, comprised of values ​​between 0 and 1. In an embodiment, the soft mask values ​​and / or the SNR values ​​are smoothed over time and frequency using techniques familiar to those skilled in the art.

[0078] The process 600 optionally inverse transforms (606) the time-frequency tiles of the estimated target audio source into a two-channel time-domain estimate of the target audio source, as described with reference to Figure 4. Note that in some embodiments, the time-frequency tiles of the estimated audio source may be utilized in the frequency domain, while in other embodiments, a two-channel time-domain estimate of the estimated audio source may be utilized.

[0079] Device Architecture Example FIG. 7 is a block diagram of an apparatus architecture 700 for implementing the systems and processes described with reference to FIGS. 1-6, according to an embodiment. The device architecture 700 can be used with any computer or electronic device capable of performing the mathematical calculations described above.

[0080] In the illustrated example, the device architecture 700 includes one or more processors 701 (e.g., CPU, DSP chip, ASIC), one or more input devices 702 (e.g., keyboard, mouse, touch surface), one or more output devices (e.g., LED / LCD display), memory 704 (e.g., RAM, ROM, Flash), and an audio subsystem 706 (e.g., media player, audio amplifier, support circuits) coupled to speakers 706. Each of these components is coupled to one or more buses 707 (e.g., system, power, peripherals, etc.). In an embodiment, the features and processes described herein may be implemented as software instructions stored in memory 704 or other computer-readable medium and executed by the one or more processors 701. Other architectures having more or fewer components are possible, such as architectures that use a combination of software and hardware to implement the features and processes described herein.

[0081] Although the present specification includes many specific implementation details, these should not be considered as limitations on the scope of what may be claimed, but rather as descriptions of features specific to a particular implementation. Certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately or in any suitable subcombination in multiple embodiments. Furthermore, although features may be described above as operating in a particular combination and initially claimed as such, one or more features from the claimed combination may in some cases be separated from the combination, and the claimed combination may be directed to a subcombination or variation of the subcombination. The logical flow depicted in the figures does not require the particular order or sequential order depicted to achieve desirable results. Furthermore, other steps may be provided or steps may be removed from the described flow, and other components may be added or removed from the described system. Accordingly, other implementations are within the scope of the following claims.

Claims

1. 1. A method comprising: obtaining, with one or more processors, a frequency domain representation of a first sample set from levels and spatial distributions of a plurality of target sources in a plurality of frequency subbands; obtaining, with the one or more processors, a frequency domain representation of a second sample set from a plurality of background levels and spatial distributions of the plurality of frequency subbands; adding, with the one or more processors, the first sample set and the second sample set to generate a combined sample set; detecting, with the one or more processors, level and spatial parameters for each sample of the combined sample set for each subband of the plurality of frequency subbands; weighting, within each of said plurality of frequency subbands, the detected level and spatial parameters according to their respective levels and spatial distributions of said target source and background; storing, using the one or more processors, a table of weighted levels and spatial parameters and a signal-to-noise ratio (SNR) within the plurality of frequency subbands for each sample of the combined sample set; using the one or more processors, re-indexing the table with the weighted level parameters, spatial parameters and sub-bands, so that the table contains target percentile SNRs for the weighted level and spatial parameters and sub-bands, and for a given input of quantized detected spatial and level parameters and sub-bands, an estimated SNR associated with the quantized detected spatial and level parameters and sub-bands is obtained from the table; The method includes:

2. The method of claim 1 , further comprising smoothing the indexed data with respect to one or more detected levels, one or more spatial parameters, or frequency subbands.

3. The method of claim 1 or 2, wherein the frequency domain representation is a Short-Time Fourier Transform (STFT) domain representation.

4. The method according to any of claims 1 to 3, wherein said spatial parameters include panning and phase difference between two channels of a mixed audio signal.

5. The method of any preceding claim, wherein the target source is amplitude panned using a constant power law.

6. The method of any of claims 1 to 5, wherein the target percentile SNR is the 25th percentile.

7. 1. A method comprising: using one or more processors to convert one or more frames of a two-channel time domain audio signal into a time-frequency domain representation comprising a plurality of time-frequency tiles, the frequency domain of the time-frequency domain representation comprising a plurality of frequency bins grouped into a plurality of subbands; For each time-frequency tile, calculating, using the one or more processors, spatial and level parameters for time-frequency tiles; using the one or more processors to obtain percentile signal-to-noise ratios (SNRs) for each frequency bin of a time-frequency tile associated with the calculated spatial and level parameters from a spatial level filtering (SLF) look-up table trained on the panned target source, the SLF look-up table being a table obtained after the re-indexing step in the method according to any one of claims 1 to 6; generating, using the one or more processors, a fractional value for the bin based on the percentile SNR for the bin; applying, using the one or more processors, the fractional values ​​of the bins of the time-frequency tiles to generate modified time-frequency tiles of the estimated audio source; The method includes:

8. The method of claim 7 , further comprising: converting, using the one or more processors, the modified time-frequency tiles into a plurality of time-domain audio source signals.

9. 9. The method of claim 7 or 8, wherein transforming one or more frames of a two-channel time domain audio signal into a frequency domain signal comprises applying a short-time frequency transform (STFT) to the two-channel time domain audio signal.

10. The method according to any of claims 7 to 9, wherein a number of frequency bins are grouped into octave or near-octave sub-bands.

11. An apparatus comprising: one or more processors; a memory storing instructions which, when executed by said one or more processors, cause said one or more processors to perform a method according to any one of claims 1 to 10; Including equipment.

12. A non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 10.

Images