Electronic device, method and computer program

Abstract

To provide an apparatus, a method and a computer program for music source isolation.SOLUTION: An electronic apparatus includes a circuit which is constituted: to input an audio mixed signal containing multiple sources into a deep neural network; to acquire equalizing parameters; to perform audio equalization to the audio mixed signal based on the equalizing parameters obtained from the deep neural network; and to obtain a separated source.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates generally to the field of audio processing, and more particularly to an apparatus, method, and computer program for music source separation. [Background technology]

[0002] Generally, Music Source Separation (MSS) refers to separating music into individual instrument tracks.

[0003] Traditionally, separation of musical sources is performed in the frequency domain by applying a short-time Fourier transform (STFT) to the audio signal. For example, Uhlich, Stefan & Giron, Franck & Mitsufuji, Yuki, "Deep Neural Network-Based Instrument Extraction from Music", 2015, 10.1109 / ICASSP.2015.7178348, discloses the extraction of musical instruments from music using deep neural networks. Furthermore, Uhlich, Stefan & Porcu, Marcello & Giron, Franck & Enenkl, Michael & Kemp, Thomas & Takahashi, Naoya & Mitsufuji, and Yuki in "Improving Music Source Separation Based on Deep Neural Networks by Data Augmentation and Network Blending", 2017, 10.1109 / ICASSP.2017.7952158, describe two different deep neural network architectures for this task: feedforward and recurrent architectures.

[0004] The above methods incur delays (latencies) due to their reliance on block-wise processing of the STFT (Short-Time Fourier Transform). For example, a typical value for the STFT frame size chosen according to the above approach is 2048 samples, which corresponds to an algorithmic latency of 2048 / 44100 s = 46.44 ms at a sampling rate of 44.1 kHz.

[0005] Time-domain approaches to music source separation that rely on deep neural networks (DNNs) have been described. For example, Stoller, Daniel et al., in "Wave-U-Net: A Multiscale Neural Network for End-to-End Audio Source Separation," ISMIR (2018), study end-to-end source separation in the time domain, which allows modeling of phase information and avoids fixed spectral transformations. Furthermore, Venkataramani, Shrikant & Smaragdis present an autoencoder neural network that can perform comparably to short-time front-end transforms in "End-to-End Source Separation with Adaptive Front-Ends" in Paris, 2017. However, these approaches incur significant computational complexity.

[0006] Although techniques exist for music source separation processing, it is generally desirable to improve apparatus and methods for music source separation. Summary of the Invention

[0007] According to a first aspect, the present disclosure provides an electronic device comprising: a circuit configured to input an audio mixture signal having a plurality of sources to a deep neural network to obtain equalization parameters, and perform audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources.

[0008] According to a second aspect, the present disclosure provides a method including: inputting an audio mixture signal including a plurality of sources into a deep neural network to obtain equalization parameters; and performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources.

[0009] According to a third aspect, the present disclosure provides a computer program comprising instructions that, when executed on a processor, cause the processor to input an audio mixture signal including a plurality of sources to a deep neural network to obtain equalization parameters, and perform audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources.

[0010] Further aspects are set out in the dependent claims, the following description and the drawings. [Brief description of the drawings]

[0011] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates the process of time-domain music source separation (MSS) using a deep neural network 2 (DNN) controlling a parametric equalizer 3 (EQ) to perform music source separation. [Diagram 2] 1 shows in more detail the process performed by the parametric equalizer 3 (EQ) of FIG. [Diagram 3] For each parametric filter i, we show the process of an artificial neural network, here a trained DNN, to estimate the gain parameters gi(x(n);θ), bandwidth parameters bwi(x(n);θ) and frequency parameters fi(x(n);θ) to perform parametric equalization of the mixed audio signal. [Figure 4] The artificial neural network, here the trained DNN process, used to estimate the gain parameters gi to perform graphic equalization of a mixed audio signal. [Diagram 5] 1 illustrates an embodiment of a DNN configured to control a music source separation process, where the DNN is implemented with a long short-term memory (LSTM) layer. [Figure 6] The process performed by LSTM51 is shown in more detail in Figure 5. [Figure 7] 1 shows a flow diagram visualizing a method for separating a specific source from a mixed signal containing multiple sources (or instruments). [Figure 8] FIG. 1 shows a flow diagram visualizing a method for training a deep recurrent neural network, i.e., LSTM, to estimate pre-determined equalizing parameters. [Figure 9] 1 is a schematic diagram illustrating an embodiment of an electronic device capable of implementing a target instrument estimation process based on a parametric / graphic EQ. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Before describing the embodiments in detail with reference to Figures 1 to 9, some general remarks will be made.

[0013] The present embodiment discloses an electronic device including a circuit configured to input an audio mixture signal including multiple sources into a deep neural network to obtain equalization parameters, and perform audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources.

[0014] The circuitry of an electronic device may include a processor (CPU), memory (RAM, ROM, etc.), memory and / or storage, interfaces, audio receivers, upmixing systems, etc. The circuitry may comprise or be connected to input means (mouse, keyboard, camera, etc.), output means (displays (e.g., liquid crystal, (organic) light emitting diodes, etc.)), speakers, etc., (wireless) interfaces, etc., which are commonly known as electronic devices (computers, smartphones, etc.). Also, the circuitry may be a single device, multiple devices, a chipset, etc. Additionally, the circuitry may implement a deep neural network (DNN), which may perform signal analysis on the received audio input signal to estimate at least one optimal signal parameter.

[0015] The audio mixture signal may be any kind of audio signal, for example, a sound signal including multiple sound sources, etc. It may be in the form of an analog signal, a digital signal, may originate from a hard disk, a voice recorder, a compact disc, a digital video disc, etc., or may be a data file, such as a wave file, an mp3 file, etc., and the present disclosure is not limited to any particular format of the input audio content. The input audio content may be, for example, a stereo audio signal having a first channel input audio signal and a second channel input audio signal, and the present disclosure is not limited to input audio content having two audio channels.

[0016] The equalizing parameters may be optimal signal parameters, such as any kind of parameters related to the audio mixture signal. The optimal signal parameters may be, for example, gain parameters, bandwidth parameters, frequency parameters, etc. The optimal signal parameters may be estimated using trained deep neural networks (DNNs), such as trained recurrent neural networks (RNNs), long short-term memory networks (LSTMs), gated recurrent units (GRUs), convolutional neural networks (CNNs), etc., which are trained to analyze the audio mixture input signal. Deep neural networks may be implemented, for example, in software or in hardware, such as by field programmable gate arrays (FPGAs), or using application specific integrated circuits (ASICs) customized for a particular implementation of the DNN as described in the embodiments below.

[0017] Performing the audio equalization may include performing filtering of the audio mixture input signal using optimal signal parameters estimated from the trained neural network. The equalization process may be a time domain process and may also be a process of estimating a target instrument, i.e. a separated source of a mixed signal, for example to estimate vocals, drums, bass, etc. The audio equalization process may be performed using algorithms as described in the following embodiments.

[0018] Audio equalization may be achieved, for example, by a graphic equalizer, a parametric equalizer, etc.

[0019] In some embodiments, a deep neural network may be trained to estimate optimal parameters for audio source separation, which may be equalizer parameters, etc., and these parameters may be estimated by analyzing the audio mixture signal to estimate desired equalizer settings, such as estimating optimal predetermined parameters. The signal analysis may be performed, for example, by a trained deep neural network, or a trained recurrent neural network, such as a long short-term memory model.

[0020] In some embodiments, the separated source may be vocals. The separated source may be other types of separated sources such as drums, bass, etc. The separated source may be a mix of separated sources, for example, in some embodiments, the separated source may be a mix in which vocals are suppressed.

[0021] In some embodiments, performing audio equalization may include performing a parametric equalizer, or performing a graphic equalizer, etc. Performing a parametric equalizer or performing a graphic equalizer may be implemented, for example, using an algorithm as described in the following embodiments. Audio equalization may be performed by graphic or parametric equalizers with parallel or cascade structures. In graphic audio equalizers with parallel structure, it is enough to control the gain parameter. In cascade structures, the gain parameter affects all filter coefficients, so in cascade structures, every gain change may require recalculating all coefficients.

[0022] In some embodiments, the at least one estimated optimal signal parameter may be a gain parameter, a frequency parameter, or a bandwidth parameter, or the like.

[0023] In some embodiments, implementing a parametric equalizer may include performing band-pass filtering on the received audio input signal based on predetermined fixed parameters, such as, for example, a peaking equalizer. Performing bandpass filtering on the received audio input signal may be implemented using, for example, an algorithm as described in the following embodiments. Alternatively, the bandpass filtering may be, for example, band peak filtering or shelving filtering. Also, the API (Application-Programming-Interface) can set the number of bands of the filter bank.

[0024] In some embodiments, implementing the parametric equalizer may further include allocating the received audio input signal to a plurality of bandpass filters to obtain a plurality of bandpass filtered signals.

[0025] In some embodiments, implementing the parametric equalizer may further include multiplying each bandpass filtered signal by a corresponding gain parameter to obtain a plurality of amplified signals.

[0026] In some embodiments, performing graphic equalization may include performing band notch filtering, band peak filtering, shelving filtering, or the like.

[0027] Some embodiments relate to a method that may be performed by an electronic device described herein or any other electronic device, processor, or other computing means, etc., comprising: inputting an audio mixture signal including a plurality of sources into a deep neural network to obtain equalization parameters; and performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources.

[0028] Some embodiments relate to a computer program comprising instructions that, when executed on a processor, cause the processor to input an audio mixture signal including a plurality of sources to a deep neural network to obtain equalization parameters, and perform audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources.

[0029] Hereinafter, the present embodiment will be described with reference to the drawings.

[0030] Figure 1 shows the process of time-domain music source separation (MSS) using a deep neural network 2 (DNN) controlling a parametric equalizer 3 (EQ). The DNN 2 receives a mixed signal x(n), which is a time-domain stereo audio signal obtained from an audio interface, and contains multiple audio sources, such as drums, bass, and vocals. A deep neural network 2 is trained to estimate optimal parameters for controlling a parametric equalizer 3 (EQ) to separate a particular target instrument from a mixture signal x(n), e.g. vocals. DNN2 analyzes the mixture signal x(n) and estimates optimal values ​​of given equalizing parameters for performing parametric equalization of the mixture signal x(n), namely, the gain parameter gi(x(n);θ) (also called "gain"), the bandwidth parameter bwi(x(n);θ) and the frequency parameter fi(x(n);θ), where θ represents the weights of DNN2 and thus represents the dependency of the equalizing parameters resulting from the network configuration and the input mixture. The index i represents a particular filter of the parametric equalizer 3 (EQ), as explained in more detail with respect to FIG. 2 below.

[0031] Simultaneously with the transmission to the DNN 2, the mixed signal x(n) is also sent to a parametric equalizer 3. The parametric equalizer 3 filters the mixed signal x(n) based on the equalizing parameters gi(x(n);θ), bwi(x(n);θ) and fi(x(n);θ) obtained from the DNN 2 to obtain an estimate s(caret)(n) of the target instrument (hereafter called the “separated source”). This is a stereo audio signal indicative of the instrument (here, for example, vocals) that the DNN was trained to extract from the mixed signal x(n).

[0032] DNN 2 in FIG. 1 can be, for example, a recurrent long short-term memory (LSTM) model, a gated recurrent unit (GRU), a convolutional neural network (CNN), or any other type of neural network that generates an output for each input sample or generates an output after processing a block of samples. In addition to the magnitude change caused by the equalizer 3, you can also use a group delay equalizer to change the group delay of a specific frequency band. Group delay is the delay of a signal caused by a filter on a narrowband signal. For example, you can use a group delay equalizer to reduce reverberation or add reverberation to a signal.

[0033] FIG. 2 shows in more detail the process performed by the parametric equalizer 3 of FIG. The parametric equalizer 3 comprises a plurality of bandpass filters 31-1 to 31-N, here parallel biquad filters (or "biquad" filters). The mixed signal x(n) is transmitted to each of the plurality of bandpass filters 31-1 to 31-N. Each of the bandpass filters 31-1 to 31-N passes particular frequencies of the mixed signal x(n) as defined by a respective bandwidth parameter bwi(x(n);θ) and frequency parameter fi(x(n);θ), where index i∈{1, ...,N} denotes a specific bandpass filter in the setting of the bandpass filters 31-1 to 31-N. Each bandpass filtered signal obtained from the multiple parallel bandpass filters 31 is amplified according to a corresponding gain parameter gi(x(n);θ) to obtain an amplified signal. The mixer 33 mixes the multiple amplified signals to obtain the target instrument estimate s(caret)(n). Accordingly, bandpass filters 31-1 to 31-N filter the mixed signal x(n) under the control of the DNN (2 in FIG. 1) to isolate a predetermined frequency range that corresponds to the desired target instrument, here vocals.

[0034] 2, the parametric equalizer 3 comprises a number of bandpass filters, or the parametric equalizer can be realized by, for example, a cascade structure of band-notch, band-peak and shelving filters.

[0035] FIG. 3 illustrates the process of an artificial neural network, here a trained DNN, which estimates the gain parameter gi(x(n);θ), bandwidth parameter bwi(x(n);θ) and frequency parameter fi(x(n);θ) for each parametric filter i to perform parametric equalization of the mixed audio signal, as detailed in FIG. 1.

[0036] Given a continuous-time input signal x(t) and an output y(t), the transfer function H(s) is the Laplace transform of the input

number

number

number

[0037] The transfer function of a digital biquad filter (or any discrete-time system) is expressed in the z-domain, while the transfer function of a continuous-time system is expressed in the s-domain, as discussed above. Therefore, the relationship between the input signal and the output is transformed from the s-domain to the z-domain using the following transformation equation:

number

[0038] A digital biquad filter (or "biquad" filter) as used in the signal processing embodiment described in Figure 2 above is a second order recursive linear filter containing two poles and two zeros. The biquad aspect refers to the fact that the transfer function of the filter is the ratio of two quadratic functions. The biquad transfer function is given by:

number

[0039] The correlation between the successive coefficients [b0, b1, b2, a1, a2] and the parameters for performing parametric equalization of the mixed audio signal, namely the gain parameter gi(x(n);θ), the bandwidth parameter bwi(x(n);θ) and the frequency parameter fi(x(n);θ), is given for the case of a chain of biquad filters as follows: Equation 5 can be expressed in the following form:

number

[0040] In the embodiment of FIG. 2, the mixed signal x(n) is sent to a number of parallel biquad filters 31-1 to 31-N, each of which is connected to a respective setting of filter parameters [b0 (i) , b1 (i) , b2 (i) , a1 (i) , a2 (i) ], where index i denotes a particular biquad filter. The DNN is configured to control the bandwidth parameter bwi(x(n);θ) and center frequency fi(x(n);θ) of each biquad, and the DNN also controls the respective parameters gi(x(n);θ).

[0041] Typically, to perform parametric equalization, an intermediate parameter α is introduced that connects the successive coefficients with these parameters. The mathematical function of the intermediate parameter α is given by the following equation:

number

number

[0042] In the above formula (7), the frequency fi(x(n);θ) is generally called the "shelf midpoint frequency" for the lowest and highest bands in which a shelving filter is used. In other cases of bandpass filters, the frequency fi(x(n);θ) is called the "center frequency."

[0043] In particular, we start with the transfer function of a second-order bandpass filter in the S-domain:

number

number

number

number

[0044] Finally, for band-pass / low-pass / high-pass filters, the gain H o (i) It is also possible to design a fixed filter with = 1, i.e., the filter coefficients [b0 (i) , b1 (i) , b2 (i) , a1 (i) , a2 (i) Note that ] is fixed and precomputed, and here we simply multiply the output of the filter by a gain, as shown in Figure 2.

[0045] FIG. 4 shows another embodiment of the process of an artificial neural network, here a trained DNN, used to control the equalization of a mixed audio signal to perform music source separation. According to this embodiment, the DNN controls the gain parameter g of the equalizer to perform the graphic equalization of the mixed audio signal. i Control only. That is, the equalizer according to this embodiment operating as a "graphic" equalizer comprises several equalizer units i whose gains gi(x(n);θ) are controlled by a DNN (where index i∈{1,...,N} denotes a particular equalizer unit in the set of equalizer units). The equalizer unit i of the graphic equalizer can be realized, for example, as a biquad that filters a mixed signal and whose gain parameters gi(x(n);θ) are controlled by a DNN, and whose bandwidth and frequency parameters bwi(x(n);θ) and center frequency fi(x(n);θ) are fixed parameters that can be selected according to design choices. For example, most graphic equalizers have between 7 and 31 bands. Each band has a fixed center frequency. Professional sound reinforcement graphic equalizers typically have 31 bands, with the center frequency of each band being 1 / 3 octave apart from the center frequency of the adjacent band. Thus, three bands (three sliders on the front panel) cover a combined bandwidth of one octave. Graphic equalizers with half a band per octave are commonly used when less precision is required. As in the embodiment of FIG. 4, the graphic equalizer is controlled by a DNN rather than a human, however, the expert must rely on the general equalizer design, but can select predefined center frequencies and bandwidths according to the expert's needs.

[0046] The above-mentioned embodiments of Figures 1 to 4 show the process of time-domain music source separation (MSS) using a deep neural network (DNN) (see reference number 2 in Figure 1) that controls a parametric equalizer (see reference number 3 in Figure 1 and Figure 3) or a graphic equalizer (see Figure 4) to perform music source separation (MSS).

[0047] FIG. 5 illustrates in more detail an embodiment of a DNN configured to control the music source separation process described in FIGS. 1-4 above, where the DNN is implemented as a long short-term memory (LSTM). Long Short-Term Memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. RNNs are a type of artificial neural network designed to recognize patterns in data sequences, such as numerical time series data emanating from a sensor, e.g. audio samples obtained by sampling an audio signal. By taking time and sequences into account, RNN algorithms have a temporal dimension. Unlike typical feedforward neural networks, LSTMs have feedback connections that allow for the learning of long-term dependencies.

[0048] LSTMs have the ability to remove or add information to so-called cell states, regulated by structures called gates. In particular, LSTMs contain multiple LSTM units, where a typical LSTM unit includes an input gate, an output gate, a forget gate, and a cell. The cell state stores values ​​over any time interval, thus implementing the long-term memory aspect of LSTMs. The three gates regulate the flow of information into and out of the cell.

[0049] More specifically, in this case the DNN, realized as a long short-term memory (LSTM), operates in the time domain, on windows of samples, or alternatively, on processed versions of the samples, i.e. features. For example, the last 512 samples [x(n-511),..., x(n)] can be used as the input vector to the DNN / LSTM. The output is the predicted gain parameters (gain) when using a graphical cascade equalizer, or (to not introduce latency) the gain, bandwidth and center frequency to be used at the current time step (instance) are the output of the DNN / LSTM when using a parametric parallel equalizer. Additionally, future gain, bandwidth and frequency parameters can be predicted, allowing larger values ​​to be used without requiring significant latency.

[0050] The output of the DNN / LSTM is then used to calculate the filter coefficients or gains as described above. In particular, when using the parametric parallel equalizer, the output of the DNN / LSTM was used to calculate the filter coefficients and adjust the gains. Furthermore, to avoid musical artifacts, linear interpolation is performed for smooth transitions between the different filter coefficients calculated every 512 samples.

[0051] In this embodiment, pre-processing of the mixed signal x(n) is performed. The mixed signal x(n) can be raw audio mixed signal data with any sampling rate, for example, 44.1 kHz, 48 kHz, 96 kHz for high resolution (Hi-Res) audio, 2.8224 MHz (DSD), etc.

[0052] As a preliminary step, a windowing process 50 is applied to the mixed signal x(n) to define audio windows x1, x2, . . . , x T The sequence of audio windows x1, x2, ---, x T The sequence is obtained, for example, based on a window size of 128 audio samples and a hopping distance of 64 audio samples. This sequence of audio windows x1, x2, ---, x T is fed to the LSTM 51, and x for t = 1, 2, ..., T t Output for each t We obtain the set of audio windows x1, x2, ---, x over which the LSTM iterates. Tmay have an audio window length of T = 8 for example. The output o of the LSTM 51 is fed to a mapping unit 52 which calculates, at each time step t, optimal values ​​of certain parameters, namely a gain parameter gi(x(n);θ), a bandwidth parameter bwi(x(n);θ) and a frequency parameter fi(x(n);θ) based on the output of the LSTM 51.

[0053] As previously mentioned, a series of audio windows may be derived based on a window size of 128 audio samples and a hopping distance of 64 audio samples, ie, a typical value of fifty percent (50%) overlap. Alternatively, one could use a higher hopping distance and linearly interpolate the filter coefficients instead of overlapping in the input domain and making more forward passes through the network, thus achieving a smaller time resolution with fewer forward passes and therefore calculations.

[0054] Figure 6 shows in more detail the process performed by the LSTM51 of Figure 5. The LSTM51 is input with a window of the mixed signal x(n).

[0055] An audio window is a set of T feature vectors x1, x2, ---, x that are fed into the LSTM iteratively. T LSTM constructs the input feature vector x1, x2, ---, x T Iterate over the set of hidden state vectors h1, h2,..., h T At time step t, 1 < t < T, h T is calculated using the following formula:

number

[0056] A plurality of RNN hidden layers, that is, layers 1 to L are stacked. Assuming that each layer contains a plurality of recurrent units as described in FIG. 5 above and the deep RNN has a total of L layers, the hidden state sequence of the lower layer is treated as the input sequence of the upper layer. Therefore, the hidden state sequence is given by the following formula.

Equation

[0057] That is, each layer, here each recurrent unit from layer 1 to layer L, receives the hidden state from the previous unit and generates an output as well as its own hidden state. Then, the output o t of the deep RNN is determined from the hidden state h t of the last layer L for each x L t Here, W

Equation

[0058] As mentioned above, the output of the DNN / LSTM can be the three parameters for all biquads. During training, the mean squared error between the DNN output and the optimal filter value (target) is minimized.

[0059] Next, the optimal values ​​of the equalization parameters, i.e., the gain parameter gi(x(n);θ), the bandwidth parameter bwi(x(n);θ) and the frequency parameter fi(x(n);θ), are calculated by applying the t is mapped (see calculation unit 52 in FIG. 5).

[0060] As mentioned before, the network output o t is a function of each audio window x1, --, x input to the LSTM51 such that the algorithmic latency is reduced. T is the output of the Lth layer for In such a case, you can stop the process at any time step t and output o up to that time step. t can be used to obtain the equalizing parameters, but this may lead to inaccuracies in the resulting equalizing parameters.

[0061] Or, use a wide window to input a full song and route the output to a single o t You can also get the equalizing parameters after obtaining the signal.

[0062] FIG. 7 shows a flow diagram visualizing a method for separating a specific source from a mixed signal containing multiple sources (or instruments). In step 70, a mixed signal (see x(n) in FIG. 1) containing multiple sources is received via an audio interface. In step 71, a DNN (see FIG. 1, reference number 2) operates on the mixed signal and estimates optimal parameters for controlling a parametric equalizer (see FIG. 1). In step 72, parameter equalization is performed on the mixed signal based on the equalization parameters obtained by the DNN in step 71 to obtain an estimate of the equalized signal, here the target source s(caret)(n), for example vocals.

[0063] FIG. 8 shows a flow diagram visualizing the method for training a deep recurrent neural network, i.e., LSTM, to estimate pre-determined equalizing parameters. In step 80, the separated sound sources, e.g. vocals and instruments, i.e. vocals, bass, drums, and other multiple sound signals are independently recorded or independently acquired from a database and stored in a database, thus obtaining a ground truth separation of the mixture. In step 81, a mixture of audio signals, such as a song (see FIG. 1), is obtained by mixing the audio signals of the separated sources obtained in step 80 or directly by a mixture, such as a song, stored in a database, for example. In step 82, a training data set is generated based on the obtained mixture and the ground truth separated sources. In step 83, an artificial neural network, here a DNN / LSTM, is trained using the training dataset. Any DNN training procedure, for example stochastic gradient descent (SGD), can be used to learn optimal weights for the DNN / LSTM such that a selected loss function is minimized. For example, the loss function can be the mean squared error (MSE) between the current s(caret)(n) and the ground truth s(n) in FIG. 1.

[0064] FIG. 9 illustrates a schematic diagram of an embodiment of an electronic device capable of implementing the process of target instrument estimation based on parametric / graphic EQ as described above. The electronic device 700 includes a CPU 701 as a processor. The electronic device 700 further includes a microphone 711, a speaker 710, and a convolutional neural network (or RNN) unit 708 connected to the processor 701. The processor 701 can, for example, implement a parametric and / or graphic EQ that realizes the processes described in more detail with respect to Figures 1 and 4. The RNN unit may for example be an artificial neural network in hardware, for example a neural network on a GPU, or any other hardware dedicated to the purpose of implementing an artificial neural network such as the LSTM 2 of Figure 1. The loudspeaker 710 consists of one or more loudspeakers distributed over a given space and configured to render audio. The electronic device 700 further comprises an audio interface 706 connected to the processor 701. The audio interface 706 functions as an input interface through which a user can input an audio signal. Furthermore, the electronic device 700 further comprises a user interface 709 connected to the processor 701. The user interface 709 functions as a man-machine interface and allows interaction between an administrator and the electronic system. For example, an administrator can configure the system using this user interface 709. The electronic device 701 further includes an Ethernet interface 707, a Bluetooth interface 704, and a WLAN interface 705. These units 704, 705 function as input / output interfaces for data communication with external devices. Additional loudspeakers, microphones and video cameras, for example with Ethernet, WLAN or Bluetooth connections, may be coupled to the processor 701 via these interfaces 707 , 704 and 705 .

[0065] The electronic system 700 further comprises a data store 702 and a data memory 703 (here a RAM). The data memory 703 is arranged to temporarily store or cache data or computer instructions for processing by the processor 701. The data store 702 is arranged as a long-term storage device for recording sensor data obtained, for example, from a microphone 711, which data is provided to, i.e. read out, from the CNN unit 708. The data storage device 702 may also store audio data representing audio messages that a public announcement system may transmit to persons moving within a given space.

[0066] It should be noted that the above descriptions are merely example configurations, and alternative configurations may be implemented using additional or other sensors, storage devices, interfaces, etc.

[0067] It should be understood that the present embodiment describes a method with an exemplary ordering of method steps. However, the particular ordering of method steps is provided for illustrative purposes only and should not be construed as binding.

[0068] It should be noted that the division of the electronic system of Figure 9 into units is done for illustrative purposes only, and the present disclosure is not limited to any particular division of functionality in particular units. For example, at least some of the circuitry may be realized by respective programmed processors, field programmable gate arrays (FPGAs), etc.

[0069] All units and entities described in this specification and claimed in the accompanying claims may be implemented as integrated circuit logic, for example on a chip, unless otherwise stated, and the functions provided by such units and entities may be implemented by software, unless otherwise stated.

[0070] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be understood that computer programs providing such software control, and the transmission, storage, or other medium on which such computer programs are provided, are contemplated as aspects of the present disclosure.

[0071] The present technology can also be configured as follows. (1) Inputting an audio mixture signal with multiple sources into a deep neural network to obtain equalization parameters; performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources; The circuit is configured as follows: electronic equipment. (2) The electronic device according to (1), The deep neural network is trained to estimate optimal parameters for audio source separation. electronic equipment. (3) The electronic device according to (1) or (2), The separated source is a vocal electronic equipment. (4) An electronic device according to any one of (1) to (3), The separated source is a mixture containing subdued vocals electronic equipment. (5) An electronic device according to any one of (1) to (4), Performing audio equalization includes performing parametric equalization. electronic equipment. (6) An electronic device according to any one of (1) to (5), Performing audio equalization includes performing graphic equalization. electronic equipment. (7) An electronic device according to any one of (1) to (6), The optimal signal parameters include a gain parameter and / or a frequency parameter and / or a bandwidth parameter. electronic equipment. (8) The electronic device according to (5), Performing the parametric equalization includes band-pass filtering the received audio input signal based on the equalization parameters. electronic equipment. (9) The electronic device according to (5), Performing the parametric equalization further includes allocating the received audio input signal to a plurality of bandpass filters to obtain a plurality of bandpass filtered signals. electronic equipment. (10) The electronic device according to (9), Performing the parametric equalization further includes multiplying each bandpass filtered signal by a corresponding gain parameter to obtain a plurality of amplified signals. electronic equipment. (11) The electronic device according to (6), Performing graphic equalization includes performing band notch filtering, band peak filtering, or shelving filtering. electronic equipment. (12) inputting an audio mixture signal including multiple sources into a deep neural network to obtain equalization parameters; performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources; Includes method. (13) A computer program comprising instructions that, when executed on a processor, cause the processor to: Input the audio mixture signal containing multiple sources to the deep neural network to obtain the equalization parameters. performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources; Computer program.

Claims

1. Inputting the audio mixture signal having multiple sources into a deep neural network to obtain gain parameters, and / or bandwidth parameters and / or frequency parameters for audio source separation as equalization parameters; performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources; The circuit is configured as follows: electronic equipment.

2. 2. The electronic device according to claim 1, The deep neural network is trained to estimate optimal parameters for audio source separation. electronic equipment.

3. 2. The electronic device according to claim 1, The separated source is a vocal electronic equipment.

4. 2. The electronic device according to claim 1, The separated source is a mixture containing subdued vocals electronic equipment.

5. 2. The electronic device according to claim 1, Performing audio equalization includes performing parametric equalization. electronic equipment.

6. 2. The electronic device according to claim 1, Performing audio equalization includes performing graphic equalization. electronic equipment.

7. 6. The electronic device according to claim 5, Performing the parametric equalization includes band-pass filtering the received audio input signal based on the equalization parameters. electronic equipment.

8. 6. The electronic device according to claim 5, Performing the parametric equalization includes allocating the received audio input signal to a plurality of bandpass filters to obtain a plurality of bandpass filtered signals. It further includes electronic equipment.

9. The electronic device according to claim 7, Performing the parametric equalization further includes multiplying each bandpass filtered signal by a corresponding gain parameter to obtain a plurality of amplified signals. electronic equipment.

10. The electronic device according to claim 5, Performing graphic equalization includes performing band notch filtering, band peak filtering, or shelving filtering. electronic equipment.

11. inputting an audio mixture signal including multiple sources into a deep neural network to obtain gain parameters, and / or bandwidth parameters and / or frequency parameters for audio source separation as equalization parameters; performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources; Includes method.

12. A computer program comprising instructions that, when executed on a processor, cause the processor to: Inputting an audio mixture signal including multiple sources into a deep neural network to obtain gain parameters, bandwidth parameters, and / or frequency parameters for audio source separation as equalization parameters; performing audio equalization on the audio mixture signal based on the equalization parameters obtained from the deep neural network to obtain separated sources; Computer program.

Images