Audio source extraction

Abstract

An electronic device comprising circuitry configured to perform audio processing on an audio mixture (200) based on an audio source activity profile (201) to obtain a target audio source (203).

Description

[0001] ELECTRONIC DEVICES AND METHODS

[0002] The present disclosure generally pertains to the field of audio processing, in particular to devices, methods and computer programs for audio source separation and mixing.

[0003] TECHNICAL BACKGROUND

[0004] There is a lot of audio content available, for example, in the form of compact disks (CD), tapes, audio data files which can be downloaded from the internet, but also in the form of sound tracks of videos, e.g. stored on a digital video disk or the like, etc. Typically, audio content is already mixed, e.g., for a mono or stereo setting without keeping original audio source signals from the original audio sources which have been used for production of the audio content. However, there exist situations or applications where a remixing of the audio content is envisaged.

[0005] Although there generally exist techniques for remixing audio content, it is generally desirable to improve devices and methods for remixing of audio content.

[0006] SUMMARY

[0007] According to a first aspect, the disclosure provides an electronic device comprising circuitry configured to perform audio processing on an audio mixture based on an audio source activity profile to obtain a target audio source.

[0008] According to a second aspect, the disclosure provides a method comprising performing audio processing on an audio mixture based on an audio source activity profile to extract a target audio source.

[0009] According to a third aspect, the disclosure provides a method for training a neural network, the method comprising: extracting an audio source from audio data comprising a plurality of audio sources based on a random activity profile and comparing the extracted audio source with a desired audio source to obtain a comparison result which is used to update the neural network parameters to increase the similarity with the desired audio source.

[0010] Further aspects are set forth in the dependent claims, the following description and the drawings.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Embodiments are explained by way of example with respect to the accompanying drawings, in which: Fig. 1 schematically shows a general approach of audio upmixing / remixing by means of blind source separation (BSS), such as music source separation (MSS).

[0013] Fig. 2 schematically shows an “one-to-one” and “one-to-many” approach for extracting a specific audio source by performing audio source separation. In this approach, a dedicated model is trained for one audio source type and / or for several audio source types.

[0014] Fig. 3 schematically shows an “one-to-one” label conditioned approach for extracting a specific audio source by performing audio source separation.

[0015] Fig. 4 schematically shows a process of extracting a target audio source from an audio based on an audio source activity profile.

[0016] Fig. 5 schematically shows an audio mixture signal over time during song playback, wherein the active time intervals of an audio source to be extracted are denoted, e.g., by a user.

[0017] Fig. 6 schematically shows a diagram of an audio source activity signal over time during song playback.

[0018] Fig. 7 schematically shows an embodiment of a process and system of training a neural network for performing audio source separation to extract a desired audio source from an audio mixture.

[0019] Fig. 8 shows a flow diagram visualizing a method for extracting a target audio source from an audio mixture.

[0020] Fig. 9 shows a flow diagram visualizing a method for training a neural network to perform audio source separation for extracting a desired audio source from an audio mixture.

[0021] Fig. 10 shows a block diagram depicting an embodiment of an electronic device that can implement the processes of audio source separation based on an audio source activity signal and audio processing.

[0022] DETAILED DESCRIPTION OF EMBODIMENTS

[0023] Before a detailed description of the embodiments under reference of Figs. 1 to 10 is given, general explanations are made.

[0024] An electronic device comprising circuitry configured to perform audio processing on an audio mixture based on an audio source activity profile to obtain a target audio source is provided.

[0025] In some embodiments, performing audio processing may comprise performing audio source separation on the audio mixture to extract the target audio source. In some embodiments, the audio source activity profile may comprise an audio source activity value indicating whether the target audio source is present in the audio mixture. The audio source activity value may be the value of a conditioning signal based on the source activity profile, which is used by a neural network to specify and extract more accurately the target audio source that a user desires to extract. This conditioning signal may be for example a vector that indicates whether the audio source is active or not in at least one time interval during the audio mixture. In other words, the vector is the activity profile which is used as conditioning signal for the neural network.

[0026] In some embodiments, the audio source activity profile may be a ternary vector that may take a first value indicating that the target audio source is not active, a second value indicating that the target audio source is active and a third value indicating that it is not known whether the target audio source is active or not. For example, the first value may be “0” or “not active”, the second value may be “1” or “active”, and the third value may be “-1” or “don’t know”. In other words, a DNN may be trained with a ternary vector that takes three values, namely “active”, “not active” and “don’t know”, which are represented numerically by “1”, “0” and “-1”. During training, a random portion of the random activity profile may be replaced by “-1” and the network could learn to rely on the label information from time instances which are labeled as “1” and “0”.

[0027] In some embodiments, the audio source activity profile may comprise a time interval in which the target audio source is present in the audio mixture. For example, the audio source activity profile may comprise at least one time interval in which the target audio source is present in the audio mixture. Alternatively, the audio source activity profile may comprise more than one time intervals in which the target audio source is present in the audio mixture.

[0028] In some embodiments, the audio source activity value may be a binary value over time with values “0” and “1”. Alternatively, the audio source activity value may be a Boolean value, e.g., “true” or “false”. For example, a Boolean signal with value “true” may indicate that the audio segment includes the audio source to be extracted and a Boolean signal with value “false” may indicate that the audio segment does not include the audio source to be extracted, or the like.

[0029] In some embodiments, the audio source activity value “0” may indicate that the target audio source is not active at a time instance and the audio source activity value “1” may indicate that the target audio source is active at the time instance. In addition, the audio source activity value “0” may indicate that the target audio source is not active in a time interval comprising at least one time instance, or the like. Still further, the audio source activity value “1” may indicate that the target audio source is active in a time interval comprising at least one time instance, or the like.

[0030] In some embodiments, the audio processing may be performed by a neural network.

[0031] In some embodiments, the neural network may be a deep neural network (DNN).

[0032] In some embodiments, the audio source activity profile may be specified by a user using a graphical user interface (GUI). For example, the user may use the GUI to mark up the audio mixture by labeling the time intervals in which the target source which the user desires to be extracted is active.

[0033] The embodiments also disclose a method comprising performing audio processing on the audio mixture based on an audio source activity profile to extract the target audio source.

[0034] In some embodiments, the method may further comprise performing audio source separation on the audio mixture to extract the target audio source. For example, performing audio processing comprises performing audio source separation on the audio mixture to obtain the target audio source.

[0035] In some embodiments, the audio source activity profile may comprise an audio source activity value indicating whether the target audio source is present in the audio mixture.

[0036] In some embodiments, the audio source activity profile may comprise a time interval in which the target audio source is present in the audio mixture.

[0037] In some embodiments, the audio source activity value may be a binary vector over time with “0” and “1”.

[0038] In some embodiments, the audio source activity value “0” may indicate that the target audio source is not active at a time instance and the audio source activity value “1” may indicate that the target audio source is active at the time instance.

[0039] In some embodiments, the audio processing may be performed by a deep neural network (DNN).

[0040] In some embodiments, the audio source activity profile may be specified by a user using a graphical user interface (GUI). For example, the user may use the GUI to mark up the audio mixture by labeling the time intervals in which the target source which the user desires to be extracted is active.

[0041] The embodiments also disclose a method for training a neural network, the method comprising extracting an audio source from audio data comprising a plurality of audio sources based on a random activity profile and comparing the extracted audio source with a desired audio source to obtain a comparison result which is used to update the neural network parameters to increase the similarity with the desired audio source. For example, extracting an audio source may comprise estimating an audio source, wherein audio data comprising a plurality of audio sources may be mixture audio data. The comparison result which is used to update the neural network parameters to increase the similarity with the desired audio source may be for example weight updates used to update the parameters of the neural network during training phase.

[0042] In some embodiments, the method for training a neural network may further comprises acquiring an activity profile of an audio source comprised in the audio data and having greater similarity with the desired audio source and feeding back to the neural network the comparison result. For example, acquiring an activity profile may comprise selecting e.g., by a target selection, an audio source comprised in the mixture audio source data, wherein the selected audio source has the greater similarity with the desired audio source. This audio source is selected as training target, e.g., target audio source data. The comparison result may be for example weight updates which are fed back to the neural network, and which are used to update the parameters of the neural network during training phase.

[0043] Conventional methods to perform audio remixing / upmixing by means of audio source separation

[0044] Fig. 1 schematically shows a general approach of audio upmixing / remixing by means of blind source separation (BSS), such as music source separation (MSS).

[0045] First, source separation (also called “demixing”) is performed which decomposes a source audio signal 1 comprising multiple channels Minand audio from multiple audio sources Source 1, Source 2, . . ., Source K (e.g., instruments, voice, etc.) into “separations”, here into source estimates 2a-2d for each channel i, wherein K is an integer number and denotes the number of audio sources. In the embodiment here, the source audio signal l is a stereo signal having two channels i = 1 and i = 2. As the separation of the audio source signal may be imperfect, for example, due to the mixing of the audio sources, a residual signal 3 (r(n)) is generated in addition to the separated audio source signals 2a-2d. The residual signal may for example represent a difference between the input audio content and the sum of all separated audio source signals. The audio signal emitted by each audio source is represented in the input audio content 1 by its respective recorded sound waves. For input audio content having more than one audio channel, such as stereo or surround sound input audio content, also a spatial information for the audio sources is typically included or represented by the input audio content, e.g., by the proportion of the audio source signal included in the different audio channels. The separation of the input audio content 1 into separated audio source signals 2a-2d and a residual 3 is performed based on blind source separation or other techniques which are able to separate audio sources.

[0046] In a second step, the separations 2a-2d and the possible residual 3 are remixed and rendered to a new loudspeaker signal 4, here a signal comprising five channels 4a-4e, namely a 5.0 channel system. Based on the separated audio source signals and the residual signal, an output audio content is generated by mixing the separated audio source signals and the residual signal taking into account spatial information. The output audio content is exemplary illustrated and denoted with reference number 4 in Fig. 1.

[0047] In the following, the number of audio channels of the input audio content is referred to as Minand the number of audio channels of the output audio content is referred to as Mout. As the input audio content 1 in the example of Fig. 1 has two channels i = 1 and i =2 and the output audio content 4 in the example of Fig. 1 has five channels 4a-4e, Min= 2 and Mout= 5. The approach in Fig. 1 is generally referred to as remixing, and in particular as upmixing if Min< Mout. In the example of the Fig. 1 the number of audio channels Min = 2 of the input audio content 1 is smaller than the number of audio channels Mout = 5 of the output audio content 4, which is, thus, an upmixing from the stereo input audio content 1 to 5.0 surround sound output audio content 4.

[0048] Fig. 2 schematically shows an “one-to-one” and “one-to-many” approach for extracting a specific audio source by performing audio source separation. In this approach, a dedicated model is trained for one audio source type and / or for several audio source types.

[0049] In Fig. 2a, a dedicated model is trained for one audio source type. An audio signal comprising a variety of audio sources, such as guitar, drums, vocals and other, is input to the dedicated model for a specific audio source to be extracted. In this case, every dedicated model is trained to extract a specific audio source. For example, a dedicated model for the “vocals” source, e.g., a “vocals model” may be used to extract the “vocals” source from an audio mixture. Accordingly, a dedicated model for the “drums” source, e.g., a “drums model” may be used to extract the “drums” source from an audio mixture. A dedicated model for the “guitar” source, e.g., a “guitar model” may be used to extract the “guitar” source from an audio mixture. A dedicated model for the “other” source, e.g., a “other model” may be used to extract the “other” source from an audio mixture. And Fig. 2b schematically shows a model that is trained for several audio source types. In this case, one model is trained to perform source separation on the input audio signal and to separate it to the audio sources included in the audio signal. For example, such a model receives as an input an audio signal comprising a variety of audio sources, such as guitar, drums, vocals and other. This model which has a shared section, a bottleneck section and a number of heads corresponding to the number of audio sources comprised in the input audio mixture, here four heads, i.e., headl, head2, head3, and head4, separates the audio sources comprised in the input audio mixture. Namely, here separates the audio signal to a “bass” separation, a “drums” separation, a “vocals” separation and an “other” separation.

[0050] Fig. 3 schematically shows an “one-to-one” label conditioned approach for extracting a specific audio source by performing audio source separation. In this approach, a lab el -conditioned neural network is used where a conditioning mechanism based on some text or audio is used to specify the source that should be extracted. In Fig. 3 the conditioning mechanism specifies which one of the sources included in an audio mixture the user wants to extract. For example, if the user wants to extract the “bass” source from the audio mixture, then the conditioning mechanism is based on a text or on an audio that specifies the specific “bass” source that should be extracted. Accordingly, if the user wants to extract the “drums” source from the audio mixture, then the conditioning mechanism is based on a text or on an audio that specifies the specific “drums” source that should be extracted. If the user wants to extract the “vocals” source from the audio mixture, then the conditioning mechanism is based on a text or on an audio that specifies the specific “vocals” source that should be extracted. If the user wants to extract the “other” source from the audio mixture, then the conditioning mechanism is based on a text or on an audio that specifies the specific “other” source that should be extracted. In such case the user has to know exactly what sound he wants to extract, i.e., how it’s called such that he can give a text description or how it sounds such that he can provide similar examples.

[0051] Typically, lab el -conditioned source separation allows to extract during training unseen sources by providing the right conditioning signal, e.g., extracting footsteps from an audio mixture by giving audio samples of footsteps or by describing it by text. Therefore, it could be used in a so called few-shot or even zero-shot setup. Generally, using lab el -conditioned networks requires the user to have either prototypical samples, e.g., for the case that the conditioning signal is audio, or a textual description, e.g., for the case that the conditioning signal is text at hand, which “describe” the audio source that the user wants to extract. However, very often the user may not be aware of what type of sound is contained in an audio mixture that the user is listening. For example, a new instrument that the user never encountered before. Therefore, it may be difficult for the user to provide prototypical samples and / or a textual description of the audio source to be extracted and which needs to be used for the conditioning, e.g., in the label-conditioned source separation.

[0052] In view of the above, it has been recognized that an audio source activity profile may be used to perform audio processing, such as audio source separation, on an audio mixture to extract a target audio source.

[0053] Inference phase

[0054] Fig. 4 schematically shows a process of extracting a target audio source from an audio based on an audio source activity profile.

[0055] An audio 200, which is an audio signal of an audio mixture, is processed, based on an audio source activity profile 201, by a neural network, such as the Deep Neural Network (DNN) 202 to obtain a desired audio source, here target audio source 203. The DNN 202 performs audio processing, e.g., audio source separation, on the audio 200 based on the audio source activity profile 201 to extract the target audio source 203.

[0056] For example, the DNN 202 is trained to perform audio source separation such that it learns to extract the target audio source from the audio 200 which has the closest audio source activity profile 201 (active or not active) to the activity profile that is given by the user. The audio source activity profile 201 may be represented as a binary vector comprising a binary value over time e.g., “0” and “1”, namely an audio source activity value comprised in the audio source activity profile 201.

[0057] It should be note that here the audio source activity value is a binary value, e.g., “0” or “1”, where “0” indicates that the audio source is not active at a specific time instance and “1” that the audio source is active, as it is described in more detail in Figs. 5 and 6 in the following. Alternatively, the audio source activity value may be a Boolean value, e.g., “true” or “false”. The audio source activity value is the value of a conditioning signal based on the source activity, which is used by the DNN 202 to specify and extract more accurately the source that a user wants to extract. This conditioning signal is for example a vector that indicates whether the audio source is active or not. It should be further noted that the DNN 202 did not see the audio mixture 200 during its training phase. The DNN 202, through its training, is able to perform the above-described audio processing and to extract the desired audio source also to an unknown audio mixture.

[0058] Fig. 5 schematically shows an audio mixture signal over time during song playback, wherein the active time intervals of an audio source to be extracted are denoted, e.g., by a user. For example, the user may be requested to label the time intervals, e.g., the source activity, where the audio source that the user desires to extract is active.

[0059] In the embodiment of Fig. 5, the abscissa displays the time and the ordinate the amplitude of the audio mixture signal. The vertical dashed lines represent the time instances t0, tr, t2, t3, t4. During song playback, the time intervals where the source is active are t0to tr, and t2to t3. The time instance t4indicates the end of the audio mixture. These time intervals may be indicated by the user, for example, via a graphical user interface (GUI), as the time intervals that the desired audio source to be extracted is active. In other words, the user marks up the audio mixture by labeling the time intervals in which the target audio source is active.

[0060] As described above, specific time intervals are provided by the user, without limiting the present disclosure in that regard. Alternatively, the user may give only rough activity boundaries, i.e., rough time intervals (e.g., like with a lasso tool in image processing) which are then automatically refined to the closest time intervals where there is an active audio source. Furthermore, the DNN 202 from Fig. 4 is trained in such a way that it can also handle “fuzzy” activity profiles.

[0061] It should be noted that in the embodiment of Fig. 5, the user has labeled for the entire audio mixture the time intervals where the desired audio source is active and in which time intervals the desired audio source is not active. Alternatively, in a case for example of very long audio files, the user may only label a specific region of the audio file. For example, the user may indicate if the desired audio source is active or not, only in a specific time interval, e.g., between time instance 0 and time instance t2. The neural network (see DNN 202 in Fig. 4) then may only use such region to extract information for the desired audio source (target audio source), while still carrying out the source separation on the whole audio file. In other words, the neural network is trained to generalize for the whole audio mixture the audio source included in the time interval marked from the user. For example, this may be done by training the neural network with an activity vector, e.g., a ternary vector, that takes three values, namely “active”, “not active” and “don’t know” which could be represented numerically by “1”, “0” and “-1”. During training, a random portion of the random activity profile (see 701 in Fig. 7) could be replaced by “-1” and the network learns to only rely on the label information from time instances that are labeled “1” and “0”.

[0062] Fig. 6 schematically shows a diagram of an audio source activity signal over time during song playback. The audio source activity signal, according to this embodiment, is a binary signal having as values “0” or “1” and indicates whether the audio source to be extracted is active or not at a specific time interval. Here, the audio source activity signal has a value “active”, and a value “not active”.

[0063] In the embodiment of Fig. 6, the abscissa displays the time and the ordinate the value of the audio source activity signal. The horizontal dashed line represents the value of the audio source activity signal, here two values, and the vertical dashed lines represent the time instances t0, tr, t2, t3, t4. The duration period of the audio is from 0 to time instance t4. The horizontal solid lines represent the values of the audio source activity signal during the duration period of the audio. Here, between 0 and time instance t0the audio source activity signal is “not active”, between time instance t0and time instance t4the audio source activity signal is “active”, between time instance t4and time instance t2the audio source activity signal is “not active”, between time instance t2and time instance t3the audio source activity signal is “active”, and between time instance t3and time instance t4the audio source activity signal is “not active”.

[0064] As described above, between 0 and time instance t0, between time instance t4and time instance t2, and between time instance t3and time instance t4the value of the audio source activity signal is “0” and thus, the audio source activity signal is “not active”, which indicates that during these periods the audio mixture does not include the audio source that e.g. a user wants to extract, thereby during audio processing (performed by DNN 202 in Fig. 4) there is no need to perform source separation on the audio. Between time instance t0and time instance t4and between time instance t2and time instance t3the value of the audio source activity signal is “1” and thus, the audio source activity signal is “active”, which indicates that during these periods the audio includes the audio source that e.g., a user wants to extract, thereby during audio processing (performed by DNN 202 in Fig. 4) the neural network performs source separation on the audio.

[0065] In the embodiment of Fig. 6, the audio source activity signal is a digital signal, such as a binary signal, i.e. a signal having “0” and “1” values, without limiting the present disclosure in that regard. Alternatively, the audio source activity signal may be a digital signal, such as a Boolean signal i.e. a signal having true and false values, or the like. For example, a Boolean signal with value “true” may indicate that the audio includes the audio source to be extracted and a Boolean signal with value “false” may indicate that the audio does not include the audio source to be extracted, or the like. Furthermore, the audio source activity signal could be continuous valued such that its values indicate the energy of the target audio source for the current audio segment.

[0066] It should be noted that the audio source activity signal is a conditioning signal based on the source activity, which is used by the neural network (see DNN 202 in Fig. 4) to specify and extract more accurately the source that a user wants to extract. This conditioning signal may for example be a vector that indicates whether the audio source is active or not.

[0067] It should be further noted that in the embodiment of Fig. 6, the activity information of the audio signal, i.e., if the audio source to be extracted is active or not and in which time intervals is active and in which time intervals is not active, is provided in the time-domain, without limiting the present disclosure in that regard. Alternatively, instead of providing the activity information in the time-domain, the user may specify the time-frequency (TF)-regions in the short-time Fourier transformation (STFT) domain where the audio source is active. In this case, the conditioning signal may be changed from a vector to a matrix / time-frequency mask.

[0068] Training phase

[0069] Fig. 7 schematically shows an embodiment of a process and system of training a neural network for performing audio source separation to extract a desired audio source from an audio mixture.

[0070] A variety of audio data 700, namely audio source data 1, . . ., audio source data K, are received by a target selection 708 and by a mixer 707. The variety of audio data 700 comprises audio source data 1 to audio source data K, wherein K is an integer number and denotes the number of audio sources, and thus the number of audio source data. The mixer 707 mixes the variety of audio data 700, i.e., audio source data 1 to audio source data K, to obtain a mixture audio source data 709. A random activity profile 701 is received by the target selection 708. A plurality of activity profiles, which are the activity profiles of the variety of audio data 700, namely activity profile 1, . . ., activity profile K, are received by the target selection 708. The target selection 708 performs audio source selection based on the received random activity profile 701 and the received activity profile 1 to activity profile K to obtain a target audio source data 705. The mixture audio source data 709 and the random activity profile 701 are fed to a DNN 702 to obtain an estimated audio source data 703. The target audio source data 705 and the estimated audio source data 703 are received by a comparison 704. The comparison 704 compares the received target audio source data 705 and the received estimated audio source data 703 to obtain weight updates 706. The weight updates 706 are applied to the weights of the DNN 702 to increase the similarity, i.e., minimize the loss, between the estimated audio source data 703 and the target audio source data 705. The weight updates 706 are computed by the comparison 704 from back- propagation of the loss that exists between the target audio source data 705 and the estimated audio source data 703. Feeding the mixture audio source data 709 and the random activity profile 701 to the DNN 702 to obtain an estimated audio source data 703 and applying the weight updates 706 to the weights of the DNN 702 to increase the similarity between the estimated audio source data 703 and the target audio source data 705 are performed iteratively many times during training until a final network is obtained to use for inference.

[0071] In the embodiment of Fig. 7, it is described a process and system of training a neural network, using the weight updates 706, for performing audio source separation. During training, two steps are performed iteratively many times until the neural network is fully trained and a final network is obtained which is used in the inference phase. At the first step, the random activity profile 701 and the mixture audio source data 709 are input to the DNN 702 to obtain the estimated audio source data 703. During training, the estimated audio source data 703 may be wrong, i.e., far removed or not similar, to the target audio source data 705 obtained by the target selection 708 and which are the desired audio data. At the second step, the comparison 704 compares the estimated audio source data 703 with the target audio source data 705 to obtain the weight updates 706, which are input to the DNN 702 to improve the result of the DNN 702 during training. The weight updates 706 are applied to the weights of the DNN 702 to increase the similarity, and thus, to minimize the loss, between the estimated audio source data 703 and the target audio source data 705. In other words, the weight updates 706 are used to train the DNN 702 in order for the DNN, at the inference phase, to provide an improved estimated audio source data 703 that are closer / more similar to the target audio source data 705, i.e., closer to the desired audio data.

[0072] It should be noted that the comparison 704 is performed between the estimated audio source data 703 and the desired audio source data 705 to obtain a comparison result, e.g., the weight updates 706. For the comparison 704, a regression loss may be used, such as “mean squared error” since two audio waveforms are compared. Using, e.g., the backpropagation algorithm, the weight updates 706 can be computed.

[0073] It should be further noted that the target selection 708 compares the activity profile 1, . . . , activity profile K with the random activity profile 701 to obtain a similarity computation result, e.g., a Hamming distance. The Hamming distance can be used for the target selection 708 as the activ- ity profiles are binary. The similarity computation result (Hamming distance) indicates the similarity between the activity profile 1 to K and the random activity profile 701, wherein the smallest Hamming distance indicates the greater similarity between one of the activity profiles 1 to K and the random activity profile 701. The target selection 708 uses the similarity computation result to select an audio source as the training target, here the target audio source data 705. After the neural network is trained (see DNN 202 in Fig. 4), it is able to extract any audio source the user desires to extract by using the activity profile of the target / desired audio source that the user provides via a GUI, as described in more detail in Figs. 4, 5, 6. The activity profile of the target / desired audio source is for example the value of the audio source activity signal that indicates whether the target / desired audio sound is “active” or “not active” in at least one time instance and / or time interval of the audio mixture.

[0074] As described above, during the training phase, a random query activity profile, here the random activity profile 701, is created and compared to all the activity profiles of the audio sources that are present in the audio mixture. Then, the audio source is used as target which has the smallest Hamming distance. By this, the neural network learns to extract the audio source which is closest in terms of its activity profile.

[0075] Alternatively, instead of creating a random query activity profile, the activity profile of one of the audio sources may be used and modified / perturbed slightly in order to model the imperfect activity labeling of the user.

[0076] In the embodiment of Fig. 7, the variety of audio data, namely the audio source data 1, . . ., audio source data K, may be audio data comprising audio mixtures having all possible instruments, without limiting the present disclosure in that regard. Alternatively, or additionally, the variety of audio data may be audio data comprising audio mixtures of a specific instrument each time, e.g., bass, drums, guitar, piano, other, vocals, and the like. In this manner the neural network is trained to distinguish between high instrument and low instrument. Still alternatively or additionally, the variety of audio data may be audio data from different videos, for example, YouTube videos and the like. In this manner the neural network is trained based on a giant database having a large variety of audio data, which may lead in improving the result (extracted audio source) acquired from the trained neural network. This for example may be an extracted audio source being as similar as possible to the desired audio source. It should be noted that the deep neural network (DNN), in order to do the audio source separation, is trained such that it learns to extract the source from the audio mixture which has the closest activity profile to the activity profile that is given by the user. The activity profile may be represented as a binary vector over time with “0” and “1”, where “0” indicates that the source is not active at the time instance and “1” that the source is active.

[0077] It should be further noted that the DNN 702 is trained in a classical supervised way, wherein the DNN inputs are the mixture and the activity profile of the target source, and the desired DNN output is the audio data for the target source.

[0078] It should be further noted that, in the embodiment of Fig. 7, the target selection is not performed by the DNN, but it is performed outside the DNN, without limiting the present embodiment in that regard.

[0079] Methods and Implementation

[0080] Fig. 8 shows a flow diagram visualizing a method for extracting a target audio source from an audio mixture.

[0081] At 800, the neural network (see DNN 202 in Fig. 4) receives an audio (see 200 in Fig. 4). At 801, the neural network (see DNN 202 in Fig. 4) receives an audio source activity profile comprising an audio source activity signal (see audio source activity value 201 in Fig. 4). At 802, audio processing is performed by the neural network on the received audio (see 200 in Fig. 4) based on the received audio source activity profile comprising the audio source activity signal (see audio source activity value 201 in Fig. 4) to obtain a processed audio, which is a target audio source (see 203 in Fig. 4).

[0082] Fig. 9 shows a flow diagram visualizing a method for training a neural network to perform audio source separation for extracting a desired audio source from an audio mixture.

[0083] At 900, a target selection (see 708 in Fig. 7) receives a variety of audio source data (see 700 in Fig. 7) comprising a plurality of audio source data, namely audio source data 1 to audio source data K, wherein K is an integer number and denotes the number of audio sources, and thus the number of audio source data. At 901, the target selection (see 708 in Fig. 7) receives the activity profiles of the audio sources included in each audio data of the variety of audio source data (see 700 in Fig. 7). At 902, the target selection (see 708 in Fig. 7) receives a random activity profile (see 701 in Fig. 7). At 903, the target selection (see 708 in Fig. 7) selects, based on the received audio source data and the random activity profile, target audio source data (see 705 in Fig. 7). At 904, the random activity profile is input to a neural network (see DNN 702 in Fig. 7) together with a mixture audio source data (see 709 in Fig. 7), which is obtained by a mixer (see 707 in Fig. 7) mixing the plurality of audio source data (see 700 in Fig. 7), to obtain an estimated audio source data (see 703 in Fig. 7). At 905, a comparison (see 704 in Fig. 7) compares the estimated audio source data (see 703 in Fig. 7) with the target audio source data (see 705 in Fig. 7) to obtain a comparison result comprising weight updates (see 706 in Fig. 7). At 906, for training purposes, the weight updates are fed to the neural network to update the weights of the neural network.

[0084] After the neural network is trained (see DNN 202 in Fig. 4), it is able to extract any audio source the user desires to extract by using the activity profile of the target / desired audio source that the user provides via a GUI, as described in more detail in Figs. 4, 5, 6. The activity profile of the target / desired audio source is for example the value of the audio source activity signal that indicates whether the target / desired audio sound is “active” or “not active” in at least one tie instance and / or time interval of the audio mixture.

[0085] It should be noted that the neural network may be trained to extract a target audio source using only the activity profile (e.g., “active” or “not active”, “0” or “1”) of the desired audio source (target audio source). The activity profile (e.g., “active” or “not active”, “0” or “1”) of the desired audio source (target audio source) may be labeled by the user during the whole audio mixture. Further, the neural network may be trained to extract a target audio source using the audio mixture and the activity profile (e.g., “active” or “not active”, “0” or “1”) of the desired audio source (target audio source). Still further, the neural network may be trained to extract a target audio source using at least the activity profile (e.g., “active” or “not active”, “0” or “1”) of the desired audio source (target audio source) which is labeled by the user only in a specific part of the audio mixture. In other words, the neural network is trained to generalize for the whole audio mixture, the activity profile of the desired audio source indicated in a specific part of the audio mixture. This can be done by training the DNN with an activity vector that can take three values: “active”, “not active” and “don’t know” which can be represented numerically by “1”, “0” and “- 1”. During training, a random portion of the random activity profile 701 in Fig. 7 can be replaced by “-1” and the network will learn to only rely on the label information from time instances where we labeled “1” and “0”.

[0086] Fig. 10 shows a block diagram depicting an embodiment of an electronic device that can implement the processes of audio source separation based on an audio source activity signal and audio processing. The electronic device 1200 comprises a CPU 1201 as processor. The electronic device 1200 further comprises a microphone array 1210, a loudspeaker array 1211 and a deep neural network unit 1220 that are connected to the processor 1201. The DNN unit may for example be an artificial neural network in hardware, e.g., a neural network on GPUs or any other hardware specialized for the purpose of implementing an artificial neural network. Loudspeaker array

[0087] 1211 consists of one or more loudspeakers that are distributed over a predefined space and is configured to render 3D audio. The electronic device 1200 further comprises a user interface

[0088] 1212 that is connected to the processor 1201. This user interface 1212 acts as a man-machine interface and enables a dialogue between an administrator and the electronic system. The user interface 1212 may be a graphical user interface (GUI). For example, the user may label as active or not active the time intervals within the target audio sound is included or not, as described in Figs. 4, 5, and 6. Still further, an administrator may make configurations to the system using this user interface 1212. The electronic device 1200 further comprises a Bluetooth interface 1204, and a WLAN interface 1205. These units 1204, 1205 act as I / O interfaces for data communication with external devices. For example, additional loudspeakers, microphones, and video cameras with Ethernet, WLAN or Bluetooth connection may be coupled to the processor 1201 via these interfaces 1204, and 1205.

[0089] The electronic system 1200 further comprises a data storage 1202 and a data memory 1203 (here a RAM). The data memory 1203 is arranged to temporarily store or cache data or computer instructions for processing by the processor 1201. The data storage 1202 is arranged as a long-term storage, e.g., for recording sensor data obtained from the microphone array 1210 and provided to or retrieved from the DNN unit 1220. The data storage 1202 may also store audio data that represents audio messages, which the public announcement system may transport to people moving in the predefined space.

[0090] It should be noted that the description above is only an example configuration. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces, or the like.

[0091] It should be further noted that alternatively the electronic device 1200 may be implemented with a digital signal processor (DSP) or a graphics processing unit (GPU), without limiting the present disclosure in that regard.

[0092] It should also be noted that the division of the electronic device of Fig. 10 into units is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units. For instance, at least parts of the circuitry could be implemented by a respectively programmed processor, field programmable gate array (FPGA), dedicated circuits, and the like.

[0093] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is, however, given for illustrative purposes only and should not be construed as binding.

[0094] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example, on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0095] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0096] Note that the present technology can also be configured as described below. Note that the present technology can also be configured as described below.

[0097] (1) An electronic device comprising circuitry configured to perform audio processing on an audio mixture (200) based on an audio source activity profile (201) to obtain a target audio source (203).

[0098] (2) The electronic device of (1), wherein performing audio processing comprises performing audio source separation on the audio mixture (200) to extract the target audio source (203).

[0099] (3) The electronic device of (1) or (2), wherein the audio source activity profile (201) comprises an audio source activity value (“0”, “1”) indicating whether the target audio source (203) is present in the audio mixture (200).

[0100] (4) The electronic device of (1) or (2), wherein the audio source activity profile (201) is a ternary vector that takes a first value indicating that the target audio source (203) is not active, a second value indicating that the target audio source (203) is active and a third value indicating that it is not known whether the target audio source (203) is active or not. (5) The electronic device of (3), wherein the audio source activity profile (201) comprises a time interval (t0- t2- t3) in which the target audio source (203) is present in the audio mixture (200).

[0101] (6) The electronic device of (3), wherein the audio source activity value (“0”, “1”) is a binary vector over time with “0” and “1”.

[0102] (7) The electronic device of (6), wherein the audio source activity value “0” indicates that the target audio source (203) is not active at a time instance (t0, t2, t3, t4) and the audio source activity value “1” indicates that the target audio source (203) is active at the time instance (t0, t4, ^2tf3> G-)-

[0103] (8) The electronic device of anyone of (1) to (7), wherein the audio processing is performed by a neural network.

[0104] (9) The electronic device of (8), wherein the neural network is a deep neural network (202).

[0105] (10) The electronic device of anyone of (1) to (9), wherein the audio source activity profile (201) is specified by a user using a graphical user interface (GUI).

[0106] (11) A method comprising performing audio processing on an audio mixture (200) based on an audio source activity profile (201) to extract a target audio source (203).

[0107] (12) The method of (11) further comprises performing audio source separation on the audio mixture (200) to extract the target audio source (203).

[0108] (13) The method of (11) or (12), wherein the audio source activity profile (201) comprises an audio source activity value (“0”, “1”) indicating whether the target audio source (203) is present in the audio mixture (200).

[0109] (14) The method of (13), wherein the audio source activity profile (201) comprises a time interval (t0- G, t2- t3) in which the target audio source (203) is present in the audio mixture (200).

[0110] (15) The method of (13), wherein the audio source activity value (“0”, “1”) is a binary vector over time with “0” and “1”.

[0111] (16) The method of (15), wherein the audio source activity value “0” indicates that the target audio source (203) is not active at a time instance (t0, t2, t3, t4) and the audio source activity value “1” indicates that the target audio source (203) is active at the time instance (t0, t4, t2, t3, t4). (17) The method of anyone of (11) to (15), wherein the audio processing is performed by a deep neural network (202).

[0112] (18) The method of anyone of (11) to (17), wherein the audio source activity profile (201) is specified by a user using a graphical user interface (GUI). (19) A method for training a neural network, the method comprising: extracting an audio source (703) from audio data (709) comprising a plurality of audio sources (700) based on a random activity profile (701); and comparing the extracted audio source (703) with a desired audio source (705) to obtain a comparison result (706) which is used to update the neural network parameters (706) to increase the similarity with the desired audio source (705)

[0113] (20) The method of (19), wherein the method for training a neural network further comprises: acquiring an activity profile of an audio source comprised in the audio data (700) and having the greater similarity with the desired audio source (705); and feeding back to the neural network (202) the comparison result (706).

Claims

CLAIMS1. An electronic device comprising circuitry configured to perform audio processing on an audio mixture based on an audio source activity profile to obtain a target audio source.

2. The electronic device of claim 1, wherein performing audio processing comprises performing audio source separation on the audio mixture to extract the target audio source.

3. The electronic device of claim 1 or 2, wherein the audio source activity profile comprises an audio source activity value indicating whether the target audio source is present in the audio mixture.

4. The electronic device of claim 1 or 2, wherein the audio source activity profile is a ternary vector that takes a first value indicating that the target audio source is not active, a second value indicating that the target audio source is active and a third value indicating that it is not known whether the target audio source is active or not.

5. The electronic device of claim 3, wherein the audio source activity profile comprises a time interval in which the target audio source is present in the audio mixture.

6. The electronic device of claim 3, wherein the audio source activity value is a binary vector over time with “0” and “1”.

7. The electronic device of claim 6, wherein the audio source activity value “0” indicates that the target audio source is not active at a time instance and the audio source activity value “ 1” indicates that the target audio source is active at the time instance.

8. The electronic device of claim 1, wherein the audio processing is performed by a neural network.

9. The electronic device of claim 8, wherein the neural network is a deep neural network.

10. The electronic device of claim 1, wherein the audio source activity profile is specified by a user using a graphical user interface.

11. A method comprising performing audio processing on an audio mixture based on an audio source activity profile to extract a target audio source.

12. The method of claim 11 further comprises performing audio source separation on the audio mixture to extract the target audio source.

13. The method of claim 11 or 12, wherein the audio source activity profile comprises an audio source activity value indicating whether the target audio source is present in the audio mixture.

14. The method of claim 13, wherein the audio source activity profile comprises a time interval in which the target audio source is present in the audio mixture.

15. The method of claim 13, wherein the audio source activity value is a binary vector over time with “0” and “1”.

16. The method of claim 15, wherein the audio source activity value “0” indicates that the target audio source is not active at a time instance and the audio source activity value “1” indicates that the target audio source is active at the time instance.

17. The method of claim 11, wherein the audio processing is performed by a deep neural network.

18. The method of claim 11, wherein the audio source activity profile is specified by a user using a graphical user interface.

19. A method for training a neural network, the method comprising: extracting an audio source from audio data comprising a plurality of audio sources based on a random activity profile; and comparing the extracted audio source with a desired audio source to obtain a comparison result which is used to update the neural network parameters to increase the similarity with the desired audio source.

20. The method of claim 19, wherein the method for training a neural network further comprises: acquiring an activity profile of an audio source comprised in the audio data and having the greater similarity with the desired audio source; and feeding back to the neural network the comparison result.

Images