Generative neural network model for processing audio samples in a filter bank region

Abstract

To provide a generative neural network model for processing samples in a filter-bank domain.SOLUTION: A neural network system implementing a generative model for autoregressively generating a distribution for a plurality of current filter-bank samples of an audio signal is provided. The current samples correspond to a current time slot, and each current sample corresponds to a channel of a filter-bank. The system includes: a hierarchy of a plurality of neural network processing tiers ordered from a top tier to a bottom tier, each tier having been trained to generate conditioning information based on previous filter-bank samples and, each tier at least excluding the top tier having been trained, also based on the conditioning information on a higher tier in the hierarchy; and an output stage having been trained to generate a probability distribution based on previous samples of one or more previous time slots and the conditioning information on the lowest processing tier.SELECTED DRAWING: Figure 3B

Description

[Technical Field]

[0001] [Related applications] This application claims priority to the following priority applications: U.S. Provisional Application 63 / 092,754 (reference number D20037USP1) filed on 16 October 2020 and European Application 20207272.4 (reference number D20037EP) filed on 12 November 2020.

[0002] [Technical field] This disclosure relates to the intersection of machine learning and audio signal processing. In particular, this disclosure relates to a generative neural network model for processing samples in a filter bank region. [Background technology]

[0003] Generative neural network models may be trained to learn at least approximately the true distribution of a training dataset, and the model may generate new data by sampling from such a learned distribution. Thus, generative neural network models have proven useful in a variety of signal synthesis schemes, including both speech and audio synthesis, audio coding, and audio augmentation. Such generative models are known to operate either in the time domain or in the amplitude spectrum (i.e., on a spectrogram) of the frequency representation of a signal.

[0004] However, generative models that operate in the time domain (such as WaveNet and sampleRNN) do not always readily integrate with other signal processing tools that have frequency domain interfaces, such as tools used for equalization, and often use regression networks, which may have limited parallelization potential. Furthermore, modern generative models that operate on spectrograms (e.g., MelNet) do not reconstruct the phase of the audio signal during synthesis, but instead rely on phase reconstruction algorithms (e.g., Griffin-Lim) as a post-processing step to properly reconstruct the audio.

[0005] Considering the above, an improved generative model is needed for audio signal processing. [Overview of the project]

[0006] This disclosure aims to satisfy, at least partially, the identified needs described above.

[0007] According to a first aspect of this disclosure, a neural network system (hereinafter referred to as "the System") is provided for autoregressively generating probability distributions of multiple current samples for a filter bank representation of an audio signal. The System may be, for example, a computer implementation system.

[0008] To the extent of this disclosure, the current sample corresponds to the current time slot, and each current sample corresponds to each channel in the filter bank.

[0009] The system includes a hierarchy of multiple neural network processing layers (hereinafter referred to as "layers" or "tiers") ordered from the top layer to the bottom layer, and each layer is trained to generate conditioning information based on previous samples for the filter bank representation and, for at least each processing layer other than the top layer, on conditioning information generated by processing layers higher up in the hierarchy (e.g., directly above the layer in the hierarchy).

[0010] The system further includes previous samples corresponding to one or more previous time slots for a filter bank representation, and an output stage trained to generate a probability distribution based on the conditioning information generated from the lowest processing layer.

[0011] According to a second aspect of this disclosure, a method is provided for autoregressively generating probability distributions of multiple current samples for a filter bank representation of an audio signal. Each current sample corresponds to a current slot, and each current sample corresponds to each channel of the filter bank. Such a method can, for example, use a system (implemented on a computer) according to the first aspect to achieve such a goal.

[0012] A third aspect of this disclosure provides a non-temporary computer-readable medium (hereinafter referred to as the "medium"). The medium stores instructions that, when executed by at least one computer processor belonging to computer hardware, are operable to use computer hardware to implement the system of the first aspect and / or perform the method of the second aspect.

[0013] This disclosure improves existing technologies in several ways. By operating directly in the filter bank domain, the generative models according to this disclosure (as implemented / executed, for example, in the system of the first embodiment, in the method of the second embodiment, and / or using the media of the third embodiment) can enable easier integration with other signal processing tools that have a frequency domain interface, such as tools used for equalization. The model can learn how to cancel aliasing inherent in real-valued filter banks, for example. Because audio signals are separated into dedicated frequency bands, the model can also learn to suppress quiet or empty frequency bands, for example, or to process general audio (e.g., music) more satisfactorily than models operating in the time domain. From another perspective, the model operates on a filter bank representation, which is equivalent to essentially processing both the magnitude and phase of the audio signal. The synthesis process does not require various spectrogram inversion methods (e.g., the Griffin-Lim method) to approximately recover phase information, for example. In some embodiments, as will be described in more detail below, the model also provides increased parallel processing capability during audio generation, and can generate up to the entire time slot of the filter bank in each step.

[0014] Other purposes and advantages of this disclosure will become apparent from the following description, drawings and claims. Within the scope of this disclosure, all features and advantages of the generative models described, for example, with reference to the system of the first aspect, are also relevant to the methods of the second aspect and / or the media of the third aspect, and can be used in combination with them, and vice versa. [Brief explanation of the drawing]

[0015] The following describes an embodiment with reference to the attached drawings.

[0016] [Figure 1] The generated filter bank is shown in general terms.

[0017] [Figure 2] The use of a generative model according to one or more embodiments of this disclosure in a signal processing scheme is schematically shown.

[0018] [Figure 3A] Two or more embodiments of a system implementing the generative model described herein are schematically shown. [Figure 3B] Two or more embodiments of a system implementing the generative model described herein are schematically shown.

[0019] [Figure 4A] Two or more embodiments of a system implementing the generative model described herein are schematically shown. [Figure 4B] Two or more embodiments of a system implementing the generative model described herein are schematically shown.

[0020] [Figure 5] A schematic flow diagram of one or more embodiments of the method described herein is shown.

[0021] In the drawings, similar elements are given the same reference numeral unless otherwise specified. Unless explicitly stated otherwise, the drawings show only the elements necessary to illustrate the exemplary embodiment, and other elements are omitted or merely suggested for clarity. [Modes for carrying out the invention]

[0022] A K-dimensional vector random variable can be represented by the symbol X, and its probability density function q X Assume we have (x). In this disclosure, the realization of such a random variable is represented by x, which can, for example, represent a vector of continuous samples of an audio signal. The dimension K can be arbitrarily large, and conversely, unless otherwise stated, it is assumed that it does not need to be explicitly specified in subsequent statements.

[0023] distribution q X (x) is assumed to be unknown in principle and described only by the training data. The generative model (implemented by the system described here) has a probability density function p X (x) represents the generative model q X (x) and p X The model is trained to maximize the distributional agreement between (x). Several distributional agreement metrics can be used to achieve this. For example, the (unknown) function q can be trained according to the following equation. X (x) and the (trainable) function p X One approach is to train the model to minimize the Kullback-Leibler (KL) divergence between (x):

number

[0024] Since only the second term of equation (1) above can be affected by model training, for example, by minimizing the negative log likelihood (NLL) defined as follows, D KLIt can be assumed to minimize: [Number]

[0025] However, q X (x) is unknown, and since the expected value of logp X (x) cannot be calculated analytically in general, practical problems may arise. To address such problems, data-driven approximations can be used. For example, if a set of N realizations of a random variable X with probability density q X (x) (i.e., a set of N vectors x) is available as training data, and such a set is denoted as Q, the following approximation is assumed to be used: [Number] This is thought to be accurate when N is large enough (thus resembling the form of Monte Carlo sampling). In practice, the set Q constitutes a smaller part of the training data and is sometimes called a "mini-batch".

[0026] The main feature of a trained generative model is that it can reconstruct signals, such as by random sampling from the trained (or learned) distribution function p X . In practice, the function p X is parameterized by a (trainable) neural network model (i.e., instead of directly providing a large set of output values of the function p X for a large set of input values, the network instead tries to find a few parameters, such as the mean, standard deviation, and / or additional moments, that can fully describe, for example, a Gaussian distribution or the like).

[0027] For example, when dealing with media signals in the form of audio signals, to capture the statistical dependencies often seen in such signals, p XIt is expected that the function p will need to be made more complex. X The associated neural network used to learn this would need to be large. To reduce the required size of the neural network, a regression form of the model can be used. As a first step towards such a regression model, the signal samples are blocked into frames. Here, x n The notation used is that it represents all samples of vector x belonging to the nth such frame. Typically, in previous state-of-the-art models, x n is a scalar (containing samples of an audio signal). The next step is the function p X It can be recursively approximated by the following equation:

number

[0028] According to the above equation, the unconditional probability density p X Alternatively, it may be possible to construct a model with a conditional probability density p. This may allow the use of a relatively small number of model parameters (i.e., a smaller neural network) compared to an unconditional model. During the training of such a model, adjustments can be made to previously available samples. During the generation process, the model can generate one frame at a time and make adjustments to previously generated samples.

[0029] The adjustment is usually extended with additional side information represented by θ, which modifies equation (4) so ​​that the following equation can be read:

number

[0030] For some tasks that use a generative model, additional side information θ may represent auxiliary information related to the task. For example, in a coding task, θ can include, for example, quantized parameters corresponding to the frame to be reconstructed at the current regression step of the model (i.e., in the case of frame n, depending on one or more previous frames <n). In another example, in a signal enhancement task, θ can include, for example, samples of a distorted signal or features extracted from samples of a distorted signal.

[0031] For simplicity, θ is omitted in the following description. However, it should be understood that when the generative model is applied to a specific problem, θ (i.e., additional side information) may be added for conditioning.

[0032] To make the model trainable, it is assumed that p has an analytical form. This can be achieved by selecting a prototype distribution of p. For example, simple parametric distributions such as the logistic distribution, Laplace distribution, Gaussian distribution, or similar distributions can be used. As an example, the case of the Gaussian distribution is described below.

[0033] It can be assumed as follows:

Equation

[0034] However, in practice, using a mixture model may result in improved modeling capabilities. In such a situation, when the prototype distribution is a Gaussian distribution, the following equation may be assumed instead:

Equation

[0035] For example, in the scalar case, it might be conceivable to use other prototype distributions to create, for instance, logistic, Laplace, or a similar mixture. In the case of vectors (M-dimensional), a mixture of components can be generated using an M-scalar distribution and an M×M linear transformation to introduce dependencies between the M dimensions.

[0036] As discussed earlier, previously known generative models for audio, for example, operate in either the time domain or (irreversibly due to the inherent need for approximate phase reconstruction) the spectrogram, which can complicate integration with other signal processing components for audio that provide only frequency domain interfaces. Therefore, to overcome such problems, this disclosure provides a generative model that operates in a filter bank representation of a signal. As a result, x n In the following, this corresponds to a multidimensional time slot that represents a sample of the signal within the filter bank region.

[0037] For illustrative purposes, a typical filter bank will be explained with reference to Figure 1.

[0038] Figure 1 schematically shows an example of a generating filter bank 100. In the filter bank 100, a sample x[n] of the signal (indicating a specific time step here) is passed through the analysis stage 110, and each sample is filtered through its respective analysis filters H0(z), H1(z), ..., H M-1(z) is provided to multiple channels (where M is the total number of such analysis filters and channels). Each analysis filter can, for example, correspond to a specific frequency band. In the smallest filter bank containing only two channels, H0(z) can correspond to, for example, a low-pass filter, and H1(z) can correspond to, for example, a high-pass filter. When using two or more channels, the filter between the first and last filters may be, for example, a well-tuned band-pass filter. The output from each analysis filter is then downsampled by a coefficient M, and the output from the analysis stage 110 is obtained by sampling the samples x0[m], x1[m], ..., x from the multiple filter banks. M-1 [m] is where all correspond to the current filter bank time slot m. Here, samples x0[m], x1[m], ..., x M-1 [m] is either described as being in the "filter bank region" or constitutes the "filter bank representation" of the input signal x[n].

[0039] Subsequently, before being supplied to the synthesis stage 120 of the filter bank 100, sample x j Various operations are performed on [m] (e.g., additional filtering, extraction of codependent features between different channels, estimation of energy within each band / channel, etc.). Here, the sample is first upsampled by coefficient M, and then each of the composite filters F0(z), F1(z), ..., F of each channel in filter bank 100 is applied. M-1It is passed to (z). The output from the synthesis stage 120 may then be added to produce an output sample x'[n] which can, for example, represent a time-delayed version of the input sample x[n]. Depending on the exact configuration of the various analysis and synthesis filters and the final processing performed between the analysis and synthesis stages 110 and 120, the output signal x'[n] may or may not be a complete reconstruction of the input signal x[n]. In many situations, such as encoding / decoding audio signals, the analysis section of the filter bank can be used on the encoder side to extract various samples from the filter bank region and apply various processing to extract features that can be used, for example, to reduce the number of bits required to fully reconstruct the signal in the synthesis stage on the decoder side. For example, the information extracted from various samples in the filter bank region can be provided as additional side information, and the samples in the filter bank region themselves can be quantized and / or compressed and then transferred to the decoder side along with the additional side information. In another example, the samples in the filter bank themselves can be omitted, and only the additional side information can be transferred to the decoder side. The decoder can then reconstruct a signal x'[n] that is sufficiently similar to the original input signal x[n] based on the compressed / quantized samples (if available) of the filter bank, along with the additional side information provided. Filter bank 100 may be, for example, a quadrature mirror filter (QMF) filter bank, but it is also assumed that other suitable types of filter banks may be used. The filter bank may be, for example, a critical sampling filter bank, but other variations are also assumed. The filter bank may be a real-valued operation such as a cosine modulation filter bank, but other variations such as a complex exponential modulation filter bank are also assumed.

[0040] The use of the model according to this disclosure in a signal processing scheme will be explained in more detail with reference to Figure 2.

[0041] Figure 2 schematically illustrates the processing scheme 200. In the preprocessing phase 210, it is assumed that the time-domain dataset 211 provides, for example, a large number of time samples of audio. For example, the time-domain dataset 211 may contain various recordings of various sounds sampled at a particular sampling rate, such that, for example, a vector 212 of time-domain samples of one or more audio signals is extracted from the dataset 211. These vectors 212 can be thought of as containing what are commonly called "ground truth" samples. Each such sample may, for example, represent the amplitude of a time-domain audio signal at a particular sampling time. The time-domain dataset 211 may also contain various features (or additional side information) 213 related to the time-domain samples 212, such as, for example, a time-domain quantized waveform (decoded by, for example, a legacy codec), quantized spectral data converted from the time domain (reconstructed by, for example, a legacy codec decoder), spectral envelope data, a parameter description of the signal, or other information describing a frame. Such features 213 are not necessarily updated for every sample 212; instead, they may be updated once for every frame containing multiple time-domain samples 212.

[0042] The time-domain sample 212 is then supplied to at least the analysis stage 214 of the filter bank, where the signal represented by the time-domain sample (as previously described with reference to Figure 1) is divided into multiple filter bank bands / channels, which may be grouped together, for example, for the same time slot m, and in this way, multiple filter bank samples, each corresponding to a different filter bank channel, are vector x m =[x0[m],x1[m],…,x M-1 [m]] is constructed, where M is the total number of filter bank channels, as previously mentioned. Additional side information 213' is also expected to be extracted using the filter bank and provided together with (or as a complement to) the additional side information 213'.

[0043] Subsequently, the filter bank samples 215 provided by the filter bank analysis stage 214 and the additional side information 213 and / or 213' are provided to the filter bank dataset 221. The filter bank dataset 221 defines both a training set of data (from which the model learns) and an interference set of data (which can be used for the model to make predictions based on what it has learned from the training set of data). Usually, the data is split so that the interference set does not contain the exact same audio signal as the training dataset, so that the model is forced to learn to extract more general features of the audio rather than just learning ways to copy audio signals it has already experienced. The filter bank samples 215 may be referred to as "filter bank ground truth" samples.

[0044] During the training stage 220, the filter bank ground truth samples 222 belonging to the training dataset are provided to the system 224 according to the present disclosure. The system 224 can include, for example, computer hardware for implementing a generative model. Additional side information 223 may also be provided to the system 224. Based on the provided samples 222 (and optionally the provided additional side information 223), the system 224 is repeatedly trained to predict the filter bank samples for the current time slot m using the filter bank samples generated previously for one or more previous time slots <m>. During the training stage 220, such "previously generated filter bank samples" may be, for example, previous ground truth samples. In the most common embodiments, the system learns a way to estimate the probability distribution of the filter bank samples belonging to the current time slot and can obtain actual samples by sampling from such a distribution. <{

[0045] For each current (filter bank) time slot m, the model of the system 224 is p(x m |x <m), therefore p X We continuously learn how to estimate (x). As already explained here, this can be achieved by using backpropagation, for example by referring to one or more of equations (2) to (7) together with the loss function lNLL mentioned above, in order to try to minimize the loss function.

[0046] After successful training, the model of system 224 is defined by several optimized model parameters 225 (including, for example, various weights and biases of the system). Once the training stage 220 is complete, the processing scheme 200 can move on to the estimation stage 220'. In the estimation stage, the trained model 224 may generalize and operate on unseen data. In the estimation stage 220', the model of system 224 may use the optimized model parameters 225 and does not require access to filter bank ground truth samples. In some situations, the model of system 224 is expected to be allowed at least access to additional side information 223', which may correspond to features such as audio signals that the system 224 reconstructs by iteratively predicting the probability distribution of filter bank samples for each time slot. Since the model can generalize and, when expanded, may operate in the estimation stage 220', the additional side information 223' is not the same as the additional side information 223 provided to the model of system 224 during the training stage 220. The model of system 224 is intended to be generalized, and it can use additional side information 223' to generate audio samples (which are not visible in training).

[0047] In the post-processing stage 230, the filter bank samples 226, reconstructed by sampling from the probability distribution generated by the system 224 (model), can pass through at least the filter bank synthesis stage 231 so that, for example (in the time domain), an output signal 232 is generated.

[0048] In the following, unless explicitly stated otherwise, "system" and "model of the system" are not distinguished. In other words, when a system is referred to as "a system that is trained to do something" or "a system that learns to do something," such references should be interpreted as a model of the system implemented using, for example, the computer hardware included in the system, and that the system has been trained / learned.

[0049] As can be seen from Figure 2, System 224 according to this disclosure, once trained, can be used, for example, in an encoding / decoding scheme. For example, as already described here, System 224 forms part of the decoder side and is given the task of predicting the samples in the current filter bank based only on its own previously generated samples and on additional side information provided, for example, from the encoder. Thus, it may be assumed that a lower bitrate is required to stream sufficient information over the channel between the encoder and the decoder, because System 224 can learn on its own how to "fill in the gaps" of the given information in order to adequately reconstruct the audio signal on the decoder side, for example. As mentioned earlier here, once trained, System 224 may also be used for other tasks, such as signal augmentation.

[0050] Two or more embodiments of a system in accordance with this disclosure (for example, system 224 described with reference to Figure 2) will be described with reference to Figures 3A and 3B.

[0051] Figure 3A schematically shows a system 300 that is implemented or can be implemented on one or more computers. System 300 includes a (neural network processing) layer T N-1 ,T N-2 ..., including layer 310 of T0. Overall, layer 300 contains a total of N layers. Figure 3A shows that there are at least three such layers, but it is also possible that there are fewer than three layers, such as just two, T1 and T0.

[0052] The layers are hierarchically arranged from the upper layer to the lower layer. In the configuration shown in FIG. 3A, the upper layer is layer T N-1 and the lower layer is layer T0. As will be described later, each layer T j (where j is an integer from 0 to N-1) is trained to generate conditional information c j to be inherited by the layer next below in the hierarchy. For example, the conditional information generated by layer T N-1 is inherited by the next layer T N-2 and so on. It is desirable for each layer to provide conditional information only to the next layer within the hierarchy, but it is also assumed that, if possible, one or more layers provide conditional information to a further lower layer within the hierarchy.

[0053] Each layer T j is trained to generate its conditional information c <m} j based on the previous filter bank samples {x j} generated by the system 300 during the previous time slot <m. As shown in the figure, the sets of previous filter bank samples provided to each layer are not necessarily equal. In some embodiments, each layer or at least some of the layers can receive a different number of previous samples (e.g., in the case of different sets of filter bank channels) and / or a different set of previous time slots. In some embodiments, each layer can receive the same set of previously generated filter bank samples.

[0054] In some embodiments, each layer T j can also use additional side information {a} j to generate its conditional information c jThe model is trained to generate the following. As shown in the figure, the content of such additional side information is not necessarily the same across all layers. In some embodiments, the additional side information may differ from layer to layer, or at least in some layers, while in other embodiments, the additional side information provided to each layer may be the same. As can be seen from the absence of an m-index in the additional side information in Figure 1, it is assumed that the additional side information does not necessarily change from time to time. In some embodiments, the additional side information may be constant over two or more consecutive time periods, while in other embodiments, the additional side information may change from time to time. Here, "different" additional side information includes, for example, that the side information in each layer belongs to the same category of side information, but, for example, that the side information in one layer is updated more frequently or is similar to that of another layer. Similarly, "different" may include, for example, that the side information provided to one layer does not belong to the same category as the side information provided to another layer. Here, "category" may include, for example, data related to quantized waveforms, spectral envelope (energy) data, quantized filter bank coefficients, parameter signal descriptions (e.g., vocoder parameters), and / or other additional side information described herein.

[0055] Furthermore, generative models can still produce useful results even without conditioning them with additional side information. For example, when generating various noises, wind sounds, background noises, or unknown songs, the sounds may lack "meaning" (i.e., they do not contain speech, lyrics, known songs, or similar elements). For instance, a generative model could be exposed to various recordings of wind sounds during training and then learn how to independently reproduce similar "wind sound" noises (during the estimation phase) without requiring additional side information. Such noises can be constructed in a non-repetitive manner by randomly sampling from the generated probability distribution.

[0056] Below the lowest layer T0, the system 300 includes an additional neural network 380, which is, for example, a multilayer perceptron (MLP) network or similar. In some embodiments, the MLP is fully connected or configured in a manner desirable to operate as intended. The neural network 380 receives conditioning information c0 from the lowest layer T0. In some embodiments, the neural network 380 receives a set of samples {x} from a previously generated filter bank. <m} * It can also receive additional side information {a} which may or may not be equal to other sets of such samples provided to the layer (this may or may not be equal to other sets of such additional side information provided to the layer). Also, in some embodiments, network 380 may or may not be equal to additional side information {a} * It can also receive. The neural network 380 forms part of the output stage 320 of system 300, and from there it estimates the probability distribution p(x m |x <m ) is generated. As shown in Figure 3A, in some embodiments the output stage 320 may include, for example, part or all of the bottom layer T0, but in other embodiments the output stage does not include any layer. Further embodiments of system 300 in which the output stage includes one or more layers are shown below.

[0057] As described here, to generate / obtain a probability distribution, the output stage can use a model containing a single prototype distribution, or a mixture model containing a sum of several such prototype distributions (for example, estimating the corresponding parameters using a multilayer perceptron network).

[0058] In some embodiments, the system 300 may further include means for generating a plurality of current samples (i.e., samples of the current filter bank) for a filter bank representation by sampling from the generated probability distribution.

[0059] In some embodiments, each layer Tj This may include one or more convolutional networks or modules configured to receive previously generated filter bank samples. Each such convolutional network / module can learn to extract features from the filter bank samples. In some embodiments, such convolutional networks are configured to use kernels whose size decreases with decreasing layer order in the hierarchy, i.e., from the top layer to the bottom layer. The kernel size can, for example, reduce the time dimension of the lower layers, thus enabling an increase in time resolution. In some embodiments, the kernel size does not change in the frequency dimension, but such variations may also be envisioned. One such embodiment in which the kernel size of the lower layers is envisioned to be small is described here, for example, with reference to Figure 4A.

[0060] In some embodiments, various layers T j It is configured to operate recursively, obtained by including one or more recurrent neural networks in each layer. For example, each layer may include at least one recurrent unit (or module) configured to receive the sum of outputs from the convolutional network as input. At least each layer except the lowest / lowest layer receives the output from at least one recurrent unit as input, and its output is conditional information c j It may also include at least one (trained) upsampling module configured to generate the output c0. In some embodiments, the lowest layer may also include at least one such (trained) upsampling module. For example, if the lowest layer does not include an upsampling module, the output c0 from the lowest layer may be, for example, the output from at least one regression unit of the lowest layer T0.

[0061] In such a recurrent network, since the internal (latent) state of the network is (to a high or low degree) memorized, a new such latent state can be calculated based on one or more of the previous states. The use of such "memory" is beneficial, for example, when processing sequential data such as filter bank (audio) samples following each other in a sequence of time slots.

[0062] FIG. 3B schematically shows another embodiment of the system 300, where the output stage 320 includes a bottom layer and the output stage is divided into several sub-layers 390-0, ..., 390-(L-1) (L is the total number of sub-layers). In the output stage 320, each sub-layer 390-j layer T0 of the sub-layer T 0,j is included. The sub-layers are executed sequentially, and each sub-layer may be trained to generate a probability distribution of one or more current samples corresponding to a proper subset (i.e., not all but at least one) of the channels of the filter bank. For example, the proper subset is different for each sub-layer. The proper subsets may overlap (i.e., at least one subset includes channels that are also included in another subset), or may not overlap (each channel is included in only one of the subsets). For all sub-layers except at least the first executed sub-layer, each sub-layer may be trained to generate a probability distribution based also on the current samples generated by one or more of the previously executed sub-layers. Each sub-layer T 0,j is provided with a set of previously generated samples {x ≦m} 0,j which may also include one or more filter bank samples currently generated for the channels processed by the previous sub-layer 390-<j (as will be described later). The same applies to the neural network 380-j, which is provided with such a set of previously generated filter bank samples {x ≦m} * jIt may have a filter bank sample that can include the sample generated during the current step, but for channels lower (frequency) than the channel being processed by the sublayer in question. Considering the current sample for the lower channel, it can be obtained using the masked kernel of the convolutional layer contained in the various sublayers, for example, as will be discussed later, also with reference to Figure 4A. Also, as mentioned above, each sublayer T 0,j And each neural network 380-j provides additional side information {a} 0,j and {a} * j It is also expected that each set of these will be received, which may or may not be identical to the corresponding additional side information provided to the upper layers of tier 310.

[0063] In some embodiments, the first sublayer 390-0 may generate one or more current samples corresponding to at least the lowest channel of the filter bank, and the last sublayer may generate one or more current samples corresponding to at least the highest channel of the filter bank. In each sublayer 390-j, the corresponding sublayer T 0,j This is used to condition information c in subsequent neural networks such as MLP 380-j. 0,j To provide.

[0064] A more detailed example of using such subdivision of the output stage 320 in multiple sublayers / sublayers is shown below with reference to Figures 4A and 4B.

[0065] Figure 4A schematically shows system 400, which includes a hierarchy 410. In this example of system 400, hierarchy 410 includes three layers T2, T1, and T0. In other examples, the hierarchy is expected to include fewer than three layers or more layers.

[0066] Each layer uses a sample from the previously generated filter bank {x <m} is an additional side information component a m and b mIt is received together with the side information component a. m This includes envelope energy and side information component b m This includes a quantized representation of the sample (e.g., obtained as a reconstruction from a legacy codec). Here, additional side information is updated for each time slot (therefore the index "m"), which may be the same for all layers, for example. In some embodiments, System 400 described herein and other systems may also be assumed to use "look-ahead" with respect to the additional side information (if provided), meaning that additional side information for one or more "future" time slots > m is also used and provided to various layers. In some embodiments, it may also be assumed that additional side information is provided to layers for only one or more future frames.

[0067] For illustrative purposes, we may assume that the sample in the filter bank corresponds to 16 channels. For example, a 16-channel QMF filter bank might provide the sample in the filter bank used for training the filter bank training dataset used for training system 400. Thus, the sample vector x of each filter bank... m Assume that it contains 16 elements, each corresponding to one of the 16 QMF filter bank channels.

[0068] In each layer, the previously received sample {x <m The set of {x} can include multiple recently generated sample vectors. Such multiples could be, for example, the most recent Z generated sample vectors, i.e., {x}. <m}={x m-Z ,x m-Z+1 ,…,x m-1} can be included. Overall, if each sample vector represents 16 filter bank channels, then the previous set of samples {x <m} includes 16 * Z channel elements. In this example, it can be assumed that each layer receives the same set of previous samples. It can also be assumed that each layer receives a different set of previous samples. For example, since the temporal resolution increases as the layers progress downwards in layer 410, some embodiments of system 400 may include providing fewer previous samples for lower layers. Here, "fewer previous" may, for example, include only the last Z' samples (where Z' < Z), while higher layers (such as layer T2, etc.) may receive all Z available previous samples. It should also be noted that lower layers may, for example, be run / executed more frequently than higher layers.

[0069] The top (i.e., highest) layer T2 includes a convolutional network 432 that takes as input a set of previous samples {x <m}. The convolutional network 432 can include, for example, 16 channels and can use a kernel size of 15. Layer T2 further includes another convolutional network 442 that takes as input the envelope energy a m . The convolutional network 442 can include, for example, 19 channels. Layer T2 further includes a convolutional network 452 that takes as input the quantized sample b m . The convolutional network 452 can include, for example, 16 channels. The kernel size and stride of each of the convolutional networks 442 and 452 can be adapted, for example, to the temporal resolution of the provided additional side information components a m and b m . In general, the exact kernel size and stride of the various convolutional networks 432, 442, 452 (and the corresponding convolutional networks of other lower layers) can be adjusted based on a plurality of factors including, for example, the provided frame size of the previous samples {x <m}, the temporal resolution of the additional side information a m and b m , etc. The same layer T jIf the number of time-direction (sequence) output samples from various convolutional networks differs (considering the choice of kernel sizes and stride for networks 432, 442, and 452), it is assumed that one or more upsampling units are also provided so that the outputs from the various convolutional networks are summed in the desired manner. For certain combinations of quantized samples and envelope conditions, the quantized samples are a in layer T2. m and b m This may be localized through separate initial processing. Such localization can be further improved in the layer 410 hierarchy as the kernel size decreases, for example, if quantized samples are provided to lower layers.

[0070] Here, it should be noted that, for example, a given kernel size is merely for explanatory purposes and may be obtained through limited experimentation depending on, for example, the type of audio being processed (e.g., speech, general audio, music, single instrument, etc.). Various convolutional networks can use a nominal stride equal to the frame size of each layer, i.e., depending on the exact number of previous sample vectors provided to each layer. The number of output channels of various convolutional networks can correspond, for example, to the number of hidden dimensions used in the model, and these numbers can be found, for example, based on the type of audio being processed.

[0071] Subsequently, the outputs from all of the convolutional networks 432, 442, and 452 are added together and provided as input to the recurrent neural network (RNN) 462. The RNN 462 is implemented using one or more stateful network units, such as a gated recurrent unit (GRU), a long short-term memory unit (LSTM), a quasi-recurrent neural network, an Elman network, or similar. A key characteristic of such an RNN is that it remembers (at least to some extent) the hidden latent states between each time slot.

[0072] In general, at least a portion of a convolutional network (e.g., 432) is assumed to use as many groups as possible to keep all filter bank channels isolated, for example, up to the summation stage which is assumed to be provided at the end of the convolutional network. In other words, each channel can be convolved with its own set of filters. This can improve the system's ability to model / learn inter-channel dependencies, as disclosed here. Although not explicitly shown in Figure 4A, for example, convolutional network 432 is assumed to include both convolutional elements and subsequent summation elements.

[0073] The output from RNN462 is provided as input to an upsampling stage 472, which is implemented, for example, using a transposed convolutional network. It is assumed that the network itself may learn exactly how to perform such upsampling. That is, the upsampling provided by stage 472 may be "trained upsampling".

[0074] The output from the upsampling stage 472 is provided to the next layer T1 as conditioning information c2. Layer T1 is assumed to contain the same type of components as the top layer T2, and all of the above regarding layer T2 applies to layer T1 and the lowest layer T2 unless otherwise stated below. 0,0 ,…,T 0,L-1 It is also expected that this will apply to the following.

[0075] One difference between layers is that at least some convolutional networks operate using a smaller kernel size than the corresponding convolutional network in the layer above. For example, convolutional network 431, which contains 16 channels, can use a kernel size of, for example, 5. Similarly, convolutional network 451, which contains 16 channels, can use a kernel size of, for example, 15. In some embodiments, convolutional network 441 may be assumed to be different from the corresponding component of the top layer T2, for example, depending on what additional side information is provided.

[0076] After summing the outputs of convolutional networks 431, 441, and 451, and then processing them further through RNN 461 and a trained upsampling stage 471, layer T1 outputs conditional information c1 that is inherited within the hierarchy to the next layer.

[0077] In system 400, the lowest layer is divided into multiple sublayers 490-j that are executed sequentially (where j = 0, ..., (L-1), and L is the total number of such sublayers). Each sublayer 490-j is executed by its respective sublayer T 0,j and includes sub-output stage 480-j. In Figure 4A, it is assumed that the conditioning information c1 is the same in all sublayers 490-j. In other embodiments, the conditioning information from layer T1 may differ in some or all sublayers, for example, in sublayer 490-j c 1,j It is assumed that it is defined as follows. The same assumed possibilities also apply to the system described later with reference to Figure 4B.

[0078] Here, "sequentially executed" means that the processing is first performed in the sublayer 490-0, and then in the next sublayer, and so on until the processing in the last sublayer 490-(L-1) is included. Sub-layer T 0,j In sub-layer T, in order for the layer to be able to access the samples that have been calculated (or generated) so far for time slot m, and for each sublayer to be able to calculate a single or a small number of the total number of channels, it is assumed that the kernels of each convolutional network 430-j are "masked" accordingly. The first sublayer 490-0 estimates one or more probability distributions of the current sample associated with the lowest channel or a subset of channels, and similarly for each subsequent sublayer, up to and including the last sublayer 490-(L-1) which has the responsibility of estimating one or more probability distributions of the current sample associated with the highest channel or a subset of channels.

[0079] For example, the sublayer 490-j can be configured such that the same number of channels are processed by each sublayer. For example, if there are a total of 16 channels and, for example, there are sublayers with L = 4, the first sublayer can be responsible for channels 0 to 3, the next sublayer for channels 4 to 7, and so on, with the last sublayer being responsible for channels 12 to 15. Of course, other divisions of responsibility for the various channels between the sublayers 490-j are also possible. By using the masked kernels in the convolutional network 430-j, even if there are placeholders for the current samples of channels that have not yet been calculated (the placeholder values are close to "true" values or at least not zero), for example, the first sub-layer T 0,0 the convolution performed by the convolutional network 430-0 of the sub-layer can be configured to ignore the placeholder values. Thus, for example, the calculation of the filter bank samples for the 14th channel, channels 0 to 3, depends only on the samples of one or more previously generated time slots. The next sub-layer T 0,1The corresponding convolutional network 430-1 (not shown in Figure 4a) has a kernel which can consider samples of channels generated from the probability distribution generated by the first sublayer 490-0, and layer T 0,L-1 The convolutional network 430-(L-1) continues to the final sublayer, which allows for the consideration of all previously generated samples for all channels, as well as currently generated samples for all channels lower than those processed by sublayer 490-(L-1).

[0080] Here, if we mean that a convolutional network has a "masked kernel," it should be noted that such a feature may only be important during training, and the entire system may have access to ground truth samples across the entire range of the channel, but it is assumed that, for example, a sublayer does not "see" such samples other than a particular proper subset of the channel. However, during estimation (i.e., after the generative model has already been trained), it is assumed that such "samples belonging to bands other than those associated with a particular sublayer" do not exist (due to the sequential execution of the sublayers), or at least there are zero. Therefore, a masked kernel may not be necessary during estimation.

[0081] As an example, using the above configuration of samples associated with a total of 16 channels, where there are four sublayers and each sublayer processes four channels, the calculation of the filter bank samples for the current time slot m may be as follows:

[0082] Pre-generated sample x <m The set (including samples from all channels) is the placeholder x of the current sample, which is calculated in the current step. mis provided to the convolutional network 430-0. The convolutional network 430-0 has masked kernels, and all current sample placeholders are ignored, but all previously generated samples provided for all channels are considered. Thereby, the sublayer T 0,0 outputs the conditional data c 0,0 which is provided to the sub-output stage 480-0 (which may be a multi-layer perceptron as previously described here). The sub-output stage 480-0 generates a probability distribution. Here, "x c,m " means a vector containing a sample at time slot m and some set "c" of channels, and "x :,<m " means a vector containing samples at one or more previous time slots < m for all (:) channels. In the case of the first sublayer 490-0, c = 0, 1, 2, 3. The probability distribution is conditioned on the previously generated samples for all previous time slots < m and all 16 channels. After sampling from such a distribution, samples x 0,m , x 1,m , x 2,m , and x 3,m for the first 4 channels at the current time slot m can be generated. The placeholders for these samples are replaced with the actually generated values.

[0083] In the next sublayer 490-1 (not shown), the kernels of the convolutional network 430-1 (not shown) are masked, and access to all previous samples provided for previous time slots < m is permitted, but access is permitted only to the 4 first channels at the current time slot m most recently generated from the first sublayer 490-0. After generating the conditional information c 0,1 provided to the sub-output stage of this layer (not shown), a probability distribution p(x c,m |x <c,m ;x :,<m) is generated. In other words, this probability distribution is valid for the current samples belonging to channels 4 to 7, but is not only conditioned on the samples previously generated during the previous time period <m, but also on the samples generated for the channels most recently processed by the first sublayer 490-0. Next, the current samples of channels 4 to 7 are obtained by sampling from this distribution and inserted to replace the corresponding placeholders. Similarly, each sublayer is executed in order until the last sublayer generates the probability distribution for the last four channels (i.e., c = 12, 13, 14, 15, etc.). This is conditioned on the previous samples generated for the previous time slot <m, but also on all the samples of the previously generated lower channels <c = 0,..., 11 for the current time slot m. After all the sublayers 490-j are executed in order, the probability distribution of all the current samples of all the channels is obtained, and all such samples may be generated by sampling from each of the generated probability distributions or may already be generated.

[0084] In the various sublayers 490-0 of the output stage 420, the components of the sublayer T 0,j are basically the same as those of the upper layer. For example, the convolutional network 430-j is similar to that of layer T1, except for the masked kernel of 430-j. Similarly, for example, the kernel sizes used in the convolutional networks 430-j and 450-j are also smaller than those of the upper layer (layer T1). As before, the convolutional network 440-j may be the same as or different from that of the upper layer. The conditioning information c 0,j is the same for the RNN 460-j and the (trained) upsampling stage 470-j that output it. In some embodiments, for example, it can be assumed that the sublayer T 0,j and the corresponding sub-output stage 480-j operate at the same (e.g., the highest possible) temporal resolution. In such a situation, for example, the corresponding upsampling stage 470-j may not be necessary.

[0085] For example, as shown in Figure 4A, various sub-output stages 480-j are used to generate previously generated samples {x ≦m} * j It is also possible to receive each set of these. In other embodiments, it may be assumed that such sets of previously generated samples are not needed for various sub-output stages 480-j.

[0086] Figure 4B shows another embodiment of the system according to this disclosure, which includes an additional regression unit 464. The regression unit 464 may be, for example, a GRU or LSTM or other type already described herein, and is common to all sublayers 490-j. The regression unit 464 may be further useful in predicting samples from lower filter bank channels to higher filter bank channels. In contrast to the various regression units 460-j (which may be assumed to operate in the "time direction"), the regression unit 464 operates in the "layer direction". For each sublayer 490-j, the sum of the outputs from the various convolutional layers 430-j, 440-j, 450-j and the conditioning information c1 from the layer above are divided into two parts. One part is mixed with the output of the regression unit 460-j and input d to the regression unit 464. j It is provided as follows. The other part is provided as a direct input to regression unit 460-j, as already described here. Regression unit 464 receives such input d from each sublayer 490-j. j It can be assumed that such an input d is received. j The internal state is updated each time. Next, output d from regression unit 464. * In other words, the current state is input to each sub-output stage 480-j as additional side information. In some embodiments, output d from regression unit 464 * Use conditional information output c 0,j It can be replaced.

[0087] In most commonly known modern models, as described at the beginning, we often operate with scalar sample vectors, where only a single value is computed for each time slot (e.g., samples of a time-domain monaural audio signal). This allows the use of simple scalar prototype distributions (e.g., Gaussian, logistic, Laplace, etc.) to construct mixed models as described above by referring to equation (7). However, this disclosure suggests operating a generative model in the filter bank domain where the vector is multidimensional for each time slot m, and the dimension is governed by the number of filter bank channels. For example, as mentioned above, in this disclosure, the vector x m This is such that each vector element sequentially contains multiple components, each corresponding to one filter bank channel. In other words, the disclosure can rely on multidimensional time slots, each time slot containing multiple frequency bands / slots.

[0088] Therefore, the generative models of the present disclosure can be configured to output multiple samples at once (from the same time slot). For example, in the embodiments described with reference to Figures 3B and 4A / 4B, if there are as many sublayers as channels, it can be assumed that each layer relates to only one channel and reconstruction occurs sequentially from 490-0 to 490-(L-1). However, there are often cases where multiple bands need to be processed in a single step (e.g., when there are fewer sublayers than channels, or when there is only a single layer / MLP in the output stage, as described with reference to Figure 3A). In such situations, the generative model may need to rely on a multivariate prototype distribution to reconstruct interband / interchannel dependencies. This may allow the reconstruction of these bands to be performed in a single step. This can provide several computational advantages because the MLP sublayers of the model can be executed in parallel (or parallel sublayers can be combined into a single sublayer, so only one sublayer needs to be executed). For example, the model can be configured to reconstruct all bands in a single step, eliminating the need to execute the MLP sublayers sequentially. The model can also be configured to output less bandwidth than the total number of channels at that point in time. In that case, multiple sequentially operating MLP sublayers are required.

[0089] Other examples can be considered, but we can first consider the multivariate Gaussian case. The generative model can be assumed to output parameters corresponding to an M-dimensional frame for a single time slot m, where M is the number of filter bank channels. In some cases, M may contain all available filter bank channels. However, in other cases, we can consider that M used below contains not all, but at least two or more such channels. The Gaussian mixture model in this case may include a J component and is written as follows:

number

number

[0090] However, a possible drawback of this approach is that the number of model parameters is O(M) 2 This can increase as shown above, which may be undesirable. To address such potential problems, the disclosure proposes using a non-correlated property, which is included in common filter banks (QMF, MDCT, DCT, etc.), meaning that as M increases, the individual dimensions of the frames (time slots) become increasingly uncorrelated (e.g., due to energy concentration occurring in these filter banks). This allows Σ j A structure can be imposed on it, and as a result U j A structure can also be imposed on it.

[0091] For illustrative purposes, we will again use the 16-dimensional case (corresponding to, for example, a 16-channel filter bank). In such a case, this disclosure is Σ j This suggests that we assume at least some of the diagonals are 0. In other words, it becomes:

number

number

[0092] Parameter U j It can be further decomposed into the following equation:

number

number

number

[0093] In the case of scalars, it is often true that, due to the characteristics of the associated training process, using distributions such as Laplace or logistic distributions yields better results than using Gaussian distributions. This disclosure suggests a method for generalizing the above method to distributions other than Gaussian distributions and is also valid for multidimensional time slots.

[0094] In the first step, the scalar parameter μ provided by the system is b and s b Using the M scalar distribution F of the mixed component j b (μ b ,s bIt has been proposed to define a linear transformation L that takes the form of a triangular matrix with a unit principal diagonal and a small number of non-zero diagonals. j It has been proposed to define a matrix. In general, this matrix can be lower triangular or upper triangular. For the sake of clarity, we will assume it is lower triangular, but it is understood that the upper triangular case can also be considered. Here, "a small number of non-zero diagonals" refers to the number of upper diagonals (in the case of an upper triangular matrix) or the number of lower diagonals (in the case of a lower triangular matrix). Furthermore, as an example, consider the case of Gauss:

number

number

[0095] The transformation described above can be used in a system such as system 300, as described with reference to Figure 3A, but here the output stage 320 is not further divided into multiple sequentially executed sublayers; instead, a single bottom layer T0 and a single output stage neural network 380 (e.g., MLP) are used to generate the probability distribution of the current sample for all channels in a single step. By (mostly) eliminating the in-frame / time slot regression present in the examples described with reference to Figures 3B and 4A / 4B, the system in Figure 3A can provide a more suitable alternative for parallelization on appropriate hardware. In the output stage 320 and neural network 380, the linear transformation L j and Fb The update of the parameters can be provided for each of the mixture components, where, as described above, L j is a lower triangular matrix with 1s on the main diagonal and b non-zero diagonals, where 1 < b < M. In some embodiments, for example, for reconstructing a signal or generating filter bank samples for the current time slot m, the sampling procedure can perform (random) sampling when it includes the transformation by L j .

[0096] The present disclosure further contemplates a method of autoregressively generating a probability distribution for a plurality of current samples for a filter bank representation of an audio signal, where the current samples correspond to a current time slot and each current sample corresponds to each channel of the filter bank. Such a method is, of course, assumed to use the generative model of the present disclosure implemented in any of the systems described herein to generate such a probability distribution. Such a method will be briefly described with reference to FIG. 5.

[0097] FIG. 5 schematically shows the flow of a method 500 according to one or more embodiments of the present disclosure. Step S501 is a step of generating conditioning information 10 (e.g., c0 as described above) using a hierarchy of a plurality of neural network processing layers, where the layers are ordered from the topmost processing layer to the bottommost processing layer, and each processing layer is trained to generate conditioning information based on previous samples for the filter bank representation and, for at least each processing layer other than the topmost layer, further based on the conditioning information generated by the processing layer above in the hierarchy. In step S501, "generating conditioning information" means using the conditioning information generated by the bottommost processing layer.

[0098] In step S502, the conditioning information 510 provided / generated in step S501 is based on previous samples corresponding to one or more previous time slots of the filter bank representation and the conditioning information 510 generated in step S501, e.g., p(xm |x <m It is used with an output stage trained to generate a probability distribution 520 such as the one shown above.

[0099] In some embodiments of Method 500, step S503 includes the step of generating a plurality of current samples for a filter bank representation by sampling from the generated probability distribution 520. The generated samples 530 are then provided, for example, as previously generated samples for one or both of steps S501 and S502.

[0100] In Method 500, steps S501 and S502 can, of course, be combined into a single step (not shown), simply corresponding to using the system disclosed herein to generate the probability distribution 520.

[0101] Method 500 is assumed to be modifiable in accordance with any embodiment of the System disclosed herein that has been described and / or discussed. For example, the System (and thereby steps S501 and S502) may use additional side information, the layers may be configured in the manner previously described herein, the steps may include the use of regression units as previously described herein, the output stage used in step S502 may be configured as previously described herein, and so on. In different representations, the flow of Method 500 is assumed to implement the generative model as described herein by using any embodiment of the System as described herein, for example.

[0102] The disclosure also envisions the provision of a non-temporary computer-readable medium that stores instructions that, when executed by at least one computer processor belonging to computer hardware, are operable to implement a generative model using computer hardware (i.e., by implementing the system described herein and / or by performing the methods described above).

[0103] The generative models implemented in the systems described herein or performed in the manner described herein are intended for use, for example, in encoding schemes, preferably decoders. Instead of sending a complete audio signal to the decoder, the generative model learns how to generate current samples based on previously generated samples (i.e., "fill in the blanks"), and by providing additional side information (such as samples from an eq-quantized filter bank, or other coding data), the generative model can learn how to generate samples from the filter bank so that a signal that adequately recalls the original signal is reconstructed in a later synthesis stage. As mentioned above, other tasks such as signal augmentation may also be suitable for generative models. The generative model may learn how to remove such noise by receiving a noisy signal as additional side information and adapting the generated probability distribution to which samples are generated accordingly.

[0104] The neural network systems of this disclosure can be implemented using computer hardware, for example, using a computer that includes a processor (a neural network system central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more high-frequency integrated circuits (RFICs), or any combination thereof) and memory coupled to the processor, as described in the exemplary embodiments above. As stated above, the processor can also be adapted to perform some or all of the steps of the methods described throughout the disclosure.

[0105] Computer hardware may include, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify the actions that the computer hardware should take. Furthermore, this disclosure must relate to a set of computer hardware that individually or collectively execute instructions that perform one or more of the concepts discussed herein.

[0106] As used herein, the term “computer-readable medium” includes, but is not limited to, data repositories in the form of solid memory, optical media, and magnetic media, for example.

[0107] Unless otherwise specified, as will be apparent from the following discussion, any discussion throughout this disclosure using terms such as “processing,” “computing, calculating,” “determining,” and “analyzing” refers to the operation and / or processing of computer hardware or computing systems, or similar electronic computing devices that manipulate and / or convert data presented as physical, for example, electronic quantities into other data similarly presented as physical quantities.

[0108] Similarly, the term “computer processor” may refer to any device or part of a device that processes electronic data from, for example, registers and / or memory and converts said electronic data into other electronic data that can be stored, for example, in registers and / or memory. “Computer,” “computing device,” “computing platform,” or “computer hardware” may include one or more processors.

[0109] The concepts described in this specification are executable by one or more processors that accept computer-readable (also called machine-readable) code, which includes an instruction set that, when executed by one or more of the processors, performs at least one of the methods described in this specification. This includes any processor capable of executing an instruction set (sequential or otherwise) that specifies the operation to be performed. Thus, one example is a standard processing system (i.e., computer hardware) comprising one or more processors. Each processor may include one or more of the following: a CPU, an image processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem, which includes main RAM and / or static RAM and / or ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system having processors connected by a network. If the processing system requires a display, such a display may include, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device, such as one or more of the following: an alphanumeric input unit, such as a keyboard, a pointing control device, such as a mouse, etc. The processing system may also include a storage system such as a disk drive unit. In some configurations, the processing system may include an audio output device and a network interface device. The memory subsystem therefore includes a computer-readable carrying medium that carries computer-readable code (e.g., software) containing an instruction set that, when executed by one or more processors, causes one or more of the methods described herein to be performed. Note that, when the method includes several elements, e.g., several steps, the order of these elements is not implied unless otherwise specified. The software may reside in a hard disk, or may reside entirely or partially in RAM and / or in a processor during execution by a computer system. Thus, the memory and processor also constitute a computer-readable medium that carries computer-readable code.Furthermore, the computer-readable carrying medium may form or be included in a computer program product.

[0110] In alternative exemplary embodiments, one or more processors may operate as standalone devices or be connected, for example, to other processors in a networked deployment; one or more processors may operate as a server or user device in a server-user network environment, or as a peer device in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web device, network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions to be taken by the machine.

[0111] The term “machine” should be understood to include any set of machines that individually or collectively perform a set of instructions (or sets of instructions) for performing any one or more of the methods described herein.

[0112] Accordingly, each exemplary embodiment of the method described herein is in the form of a computer-readable carrying medium that carries an instruction set, a computer program intended for execution on one or more processors, for example, one or more processors which are part of a web server configuration. Accordingly, as will be understood by those skilled in the art, exemplary embodiments of the disclosure may be embodied as a method, equipment such as a dedicated device, equipment such as a data processing system, or a computer-readable carrying medium, for example, a computer program product. The computer-readable carrying medium carries computer-readable code which, when executed on one or more processors, causes the processor or the number of processors to implement the method. Accordingly, embodiments of the disclosure may take the form of a method, an exemplary embodiment entirely of hardware, an exemplary embodiment entirely of software, or an exemplary embodiment which combines embodiments of software and hardware. Furthermore, the disclosure may take the form of a carrying medium that carries computer-readable program code which is embodied in the medium (for example, a computer program product on a computer-readable storage medium).

[0113] The software may further be transmitted or received over a network by a network interface device. The carrying medium is a signaling medium in an exemplary embodiment, but the term “carrying medium” should be interpreted to include signaling mediums or multiple mediums (e.g., central or distributed databases, and / or associated caches and servers) that store one or more instruction sets. The term “carrying medium” should also be interpreted to include any medium capable of storing, encoding, or carrying instruction sets for execution by one or more processors, and causing one or more processors to execute any one or more of the methods of this disclosure. The carrying medium may take many forms, but is not limited to non-volatile mediums, volatile mediums, and transmission mediums. Non-volatile mediums include, for example, optical, magnetic disks, or magneto-optical disks. Volatile mediums include dynamic memory such as main memory. Transmission mediums include coaxial cables, copper wires, and optical fibers, including wires that include bus subsystems. Transmission mediums may also take the form of acoustic or optical waves, such as those generated between radio waves and infrared data communications. For example, the term “carrying medium” should therefore be interpreted to include, but not limited to, solid memory, computer products embodied in optical and magnetic media, media that carry propagating signals representing instruction sets that, when detected and executed by at least one or more processors, perform a method, and transmission media in a network that carry propagating signals and represent instruction sets that, when executed, are detectable by at least one of the one or more processors.

[0114] The steps of the method discussed are, in one exemplary embodiment, performed by a suitable processor (or more processors) of a system / hardware that executes instructions (computer-readable code) stored in storage. It is also understood that this disclosure is not limited to any particular implementation or programming technique, and that this disclosure may be implemented using any suitable technique to perform the functions described in this specification. This disclosure is not limited to any particular programming language or operating system.

[0115] Throughout this disclosure, any reference to, for example, “one embodiment,” “several embodiments,” or “exemplary embodiments” means that certain features, structures, or characteristics described in relation to an exemplary embodiment are included in at least one exemplary embodiment of this disclosure. Accordingly, the appearance of phrases such as “in one exemplary embodiment,” “in several exemplary embodiments,” or “in an exemplary embodiment” in various places throughout this disclosure does not necessarily all refer to the same exemplary embodiment. Furthermore, certain features, structures, or characteristics may be combined in any suitable manner as will be apparent to those skilled in the art from this disclosure in one or more exemplary embodiments.

[0116] When used in this specification, unless otherwise specified, the use of ordinal numbers such as "first," "second," "third," etc., to describe a common object merely indicates that different instances of similar objects are being referred to, and does not imply that the objects being described must be in a given sequence in time, space, rank, or any other way.

[0117] In the following claims and descriptions in this specification, any one of the terms: includes, has, constitutes, or constitutes is broad in meaning, including at least the elements / features that follow it, but not excluding others. Accordingly, the term: includes, when used in a claim, should not be construed as being limited to the means, elements, or steps enumerated thereafter. For example, the expression "the apparatus includes A and B" should not be limited to the apparatus being composed solely of elements A and B. The term: has, when used in this specification, is also broad in meaning, including at least the elements / features that follow it, but not excluding others. Accordingly, has is synonymous with including and means including.

[0118] It should be understood that in the above description of exemplary embodiments of the Disclosure, various features of the Disclosure are sometimes grouped together in a single exemplary embodiment, figure, or description for the purpose of streamlining the Disclosure and aiding in the understanding of one or more of the various aspects of the Invention. However, this method of the Disclosure should not be construed as reflecting an intention that the claims require more features than expressly described in each claim. Rather, as reflected in the following claims, the aspects of the Invention are fewer than all the features of a single, aforementioned exemplary embodiment; therefore, the claims following the description are expressly incorporated into this description, and each claim stands independently as a separate exemplary embodiment of the Disclosure.

[0119] Furthermore, while some exemplary embodiments described herein include some features but not others included in other exemplary embodiments, as will be understood by those skilled in the art, combinations of features from different exemplary embodiments are within the scope of the disclosure and constitute different exemplary embodiments. For example, any of the claimed exemplary embodiments may be used in any combination within the following claims.

[0120] Numerous specific details are described in the description provided in this specification. However, it is understood that exemplary embodiments of this disclosure may be carried out without these specific details. In other examples, well-known methods, structures, and techniques have not been shown so as not to obscure the understanding of the description of the invention.

[0121] Accordingly, while what is believed to be the best mode of this disclosure is described, those skilled in the art will understand that other and further modifications may be made without departing from the spirit of this disclosure, and that all such changes and modifications are intended to be within the scope of this disclosure. For example, any formulas described above are merely expressions of procedures that may be used. Functions may be added to or removed from block diagrams, and operations may be interchanged between function blocks. Steps may be added to or removed from methods described within the scope of this disclosure.

[0122] Various aspects of the present invention may be apparent from the following enumerated example embodiments (EEE). (EEE1) A neural network system (300) for autoregressively generating a probability distribution of a plurality of current samples (x m ) for a filter bank representation of an audio signal, wherein the current samples correspond to a current time slot (m), and each current sample corresponds to each channel of the filter bank, A plurality of neural network processing layers (T N-1 ) ordered from the topmost processing layer (T N-1 ) to the bottommost processing layer (T0), wherein each processing layer (T N-2 , T j ) is trained to generate conditional information (c <m ) based on previous samples (x j+1 ) for the filter bank representation and, for at least each processing layer except the topmost layer, further based on conditional information (c j+1 ) generated by the upper processing layer (T j ) of the hierarchy (310), Previous samples (x <m ) corresponding to one or more previous time slots (<m) of the filter bank representation, and an output stage (320) trained to generate the probability distribution based on the previous samples and the conditional information generated from the bottommost processing layer, A system comprising. (EEE2) The system of EEE1, wherein each processing layer is trained to generate the conditional information further based on additional side information provided at the current time slot. (EEE3) The system according to EEE1 or 2, further comprising means configured to generate the plurality of current samples for the filter bank representation by sampling from the generated probability distribution. (EEE4) The system according to any one of EEE1 to 3, wherein each processing layer includes a convolutional module configured to receive the previous samples for the filter bank representation, each convolutional module has the same number of input channels as the number of channels in the filter bank, and the kernel size of the convolutional module decreases from the highest processing layer to the lowest processing layer in the hierarchy. (EEE5) Each processing layer includes at least one regression unit configured to receive the sum of the outputs from the convolution module as input, For at least each processing layer other than the minimum processing layer, there is at least one learning upsampling module configured to receive the output from at least one regression unit as input and generate conditioning information as output, Systems as described in EEE4, including those listed above. (EEE6) The system according to any one of EEE1 to 5, wherein the output stage includes the lowest processing layer, the lowest processing layer is subdivided into a plurality of sequentially executed sublayers, each sublayer is trained to generate a probability distribution of one or more current samples corresponding to a proper subset of channels in the filter bank, and at least all sublayers except the first executed sublayer are trained to generate the probability distribution based on the current samples generated by one or more previously executed sublayers. (EEE7) The system according to EEE6, wherein the first sublayer performed generates one or more current samples corresponding to at least the lowest channel of the filter bank, and the last sublayer performed generates one or more current samples corresponding to at least the highest channel of the filter bank. (EEE8) The system described in any one of EEE1 to 7, wherein the probability distribution of the current sample is obtained using a mixture model. (EEE9) The generation of the aforementioned probability distribution is a linear transformation (L) of the mixing coefficient (j) of the mixing model.j The system described in EEE8 includes providing updates to the linear transformation, wherein the linear transformation is defined by a triangular matrix with 1 on its main diagonal, and the triangular matrix has a number of non-zero diagonals greater than 1 and less than the number of channels in the filter bank. (EEE10) The sampling is a system according to EEE9, which is subordinate to EEE3, and includes the transformation by the linear transformation. (EEE11) The system according to EEE6, which is subordinate to EEE5, further comprising an additional regression unit (464) that is common to all sublayers of the lowest layer processing layer and is configured to receive as input a mixture of i) the sum of the outputs from the convolution module and ii) the output (460-j) of at least one regression unit, and based thereon generate additional side information (d*) for each sub-output stage (480-j) of each sublayer (490-j). (EEE12) A method for autoregressively generating a probability distribution for a plurality of current samples for a filter bank representation of an audio signal, wherein the current samples correspond to current time slots, and each current sample corresponds to each channel of the filter bank, and the method comprises the step of generating the probability function using a system described in any one of EEE1 to EEE11. (EEE13) A non-temporary computer-readable medium that stores instructions that, when executed by at least one computer processor belonging to computer hardware, are capable of using said computer hardware to implement the system described in any one of EEE1 to 11 and / or perform the method described in EEE12.

Claims

1. A neural network system that generates one or more filter bank samples of a filter bank representation of an audio signal, A neural network processing layer configured to generate conditioning information based on one or more previous filter bank samples and previous conditioning information of the aforementioned filter bank representation, It is the output stage, Based on the one or more previous filter bank samples and the generated conditioning information, a mixed model is used to generate a probability distribution of the one or more filter bank samples. The probability distribution is sampled to generate one or more filter bank samples. The output stage is configured as follows: A system that includes this.

2. The system according to claim 1, wherein the one or more filter bank samples correspond to a time slot and each channel of the filter bank.

3. A hierarchical structure of multiple neural network processing layers, ordered from the highest processing layer to the lowest processing layer, wherein each processing layer is configured to generate conditioning information based on one or more previous filter bank samples and / or previous conditioning information of the filter bank representation. The system according to claim 1 or 2, further comprising:

4. The system according to any one of claims 1 to 3, wherein the neural network processing layer includes at least one of a convolutional neural network, a regressive neural network, a gated regression unit, a long-term short-term memory unit, a quasi-regressive neural network, and / or an Elman network.

5. The system according to any one of claims 1 to 4, wherein the filter bank representation includes a QMF filter bank representation, and / or the one or more filter bank samples include QMF samples.

6. The generated conditional information is generated from the lowest processing layer, according to the system according to claim 3 or any one of claims 4 to 5 dependent on claim 3.

7. A decoder for decoding an audio signal, wherein the decoder is A generation model configured to receive a filter bank representation of an audio signal and generate one or more filter bank samples corresponding to the filter bank representation of the audio signal, wherein the generation model is A generative model including a neural network processing layer configured to generate conditioning information based on one or more previous filter bank samples and previous conditioning information of the filter bank representation, It is the output stage, Based on the one or more previous filter bank samples and the generated conditioning information, a mixed model is used to generate a probability distribution of the one or more filter bank samples. The probability distribution is sampled to generate one or more filter bank samples. The output stage is configured as follows: A decoder that includes this.

8. The decoder according to claim 7, wherein the generation model is further configured to receive side information.

9. The decoder according to claim 8, wherein the side information includes at least one of quantized waveform, spectral data, spectral envelope data, and / or parameter data.

10. The decoder according to claim 8 or 9, wherein the output stage is further configured to generate the probability distribution of one or more filter bank samples based on the side information.

11. The decoder according to any one of claims 7 to 10, wherein the one or more filter bank samples correspond to a time slot and each channel of the filter bank.

12. A hierarchical structure of multiple neural network processing layers, ordered from the highest processing layer to the lowest processing layer, wherein each processing layer is configured to generate conditioning information based on one or more previous filter bank samples and / or previous conditioning information of the filter bank representation. The decoder according to any one of claims 7 to 11, further comprising:

13. The decoder according to any one of claims 7 to 12, wherein the neural network processing layer includes at least one of a convolutional neural network, a regressive neural network, a gated regression unit, a long-term short-term memory unit, a quasi-regressive neural network, and / or an Elman network.

14. The decoder according to any one of claims 7 to 13, wherein the filter bank representation includes a QMF filter bank representation, and / or the one or more filter bank samples include QMF samples.

15. The generated conditional information is generated from the lowest processing layer, using the decoder according to claim 12 or any one of claims 13 to 14 dependent on claim 12.

16. A computer implementation method for generating one or more filter bank samples of a filter bank representation of an audio signal, wherein the method is performed by a computer, A step of generating conditional information based on one or more previous filter bank samples and previous conditional information of the filter bank representation, The steps include generating a probability distribution of the one or more filter bank samples using a mixture model based on the one or more previous filter bank samples and the generated conditioning information, The steps include sampling the probability distribution to generate one or more filter bank samples, A method that includes this.

17. The method according to claim 16, wherein the one or more filter bank samples correspond to a time slot and each channel of the filter bank.

18. The method according to claim 16 or 17, wherein the filter bank representation includes a QMF filter bank representation, and / or the one or more filter bank samples include QMF samples.

Images