Apparatus, method and computer program for blind source separation and remixing

Abstract

An electronic device comprises a circuit configured to: perform (402; 602; 902; 1002) blind source separation (201) based on a received input signal (1) to obtain separated sources; perform (403; 605; 906; 1005) time domain filtering (202) on at least one of the separated sources to obtain at least one filtered separated source; and perform (404; 606; 907; 1006) remixing or upmixing (203) based on at least one filtered separated source.

Description

Technical Field

[0001] The present disclosure relates generally to the field of audio processing, and more particularly to apparatus, methods and computer programs for blind source separation and remixing or upmixing. Background Art

[0002] There are many audio contents available, for example in the form of compact discs (CDs), tapes, audio data files that can be downloaded from the Internet, but also in the form of soundtracks of videos, for example stored on digital video disks, etc. Typically, the audio contents have been mixed, for example for a mono or stereo setting, without retaining the original audio source signal from the original audio source that has been used for the production of the audio contents. However, there are situations or applications where a remix or upmix of the audio contents is envisaged. For example, in the case where the audio contents should be played on a device with more available audio channels than the audio contents provide, for example, a mono audio content will be played on a stereo device, a stereo audio content will be played on a surround sound device with six audio channels, etc.

[0003]

[0004] While techniques generally exist for remixing audio content, improved apparatus and methods for remixing or upmixing audio content are generally desirable. Summary of the invention

[0004] According to a first aspect, the present disclosure provides an electronic device comprising a circuit configured to: perform blind source separation based on a received input signal to obtain separated sources; perform time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and perform remixing or upmixing based on at least one filtered separated source and based on the separated sources to obtain a remixed or upmixed signal.

[0005] According to another aspect, the present disclosure provides a method, comprising: performing blind source separation based on a received input signal to obtain separated sources; performing time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and performing remixing or upmixing based on the at least one filtered separated source and based on the separated sources to obtain a remixed or upmixed signal.

[0006] According to another aspect, the present disclosure provides a computer program comprising instructions, which, when executed on a processor, cause the processor to perform the following steps: perform blind source separation based on a received input signal to obtain separated sources; perform time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and perform remixing or upmixing based on the at least one filtered separated source and based on the separated sources to obtain a remixed or upmixed signal.

[0007] Further aspects are set out in the dependent claims, the following description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments are explained by way of example with reference to the accompanying drawings, in which:

[0009] Figure 1 A general method of audio upmixing / remixing via audio source separation (BSS) is schematically shown;

[0010] Figure 2 The process of remixing / upmixing based on the time domain filtering method is schematically shown;

[0011] Figure 3 Schematically showing examples of a waveform of a "Vocal" separation and a waveform of a "Vocal" truth value, both represented on a logarithmic scale;

[0012] Figure 4 A flow chart showing a method for visualizing a temporal filtering approach;

[0013] Figure 5 A diagram of a time domain filtering method is schematically shown;

[0014] Figure 6 shows the relationship between the reverberation time (T 60 Flow chart of the time domain filtering method for visualizing the parameters s and w related to time;

[0015] Figure 7 Schematically illustrates the process of remixing / upmixing based on band-limited filtering and based on time-domain filtering methods;

[0016] Figure 8 shows a graph visualizing the correlation of separated sources including "Vocal", "Drums", "Bass", and "Other" with parameters τ, s, w;

[0017] Figure 9a A flowchart of a method for visualizing time domain filtering based on parameter τ when a "drum sound" signal is detected in a "drum sound" separation source is shown;

[0018] Figure 9b An example of a "drum sound" audio signal is schematically shown;

[0019] Fig.10 A flowchart of a method for visualizing time domain filtering based on a parameter τ′ calculated based on short-term energy associated with a received input signal is shown; and

[0020] Fig.11 Embodiments of electronic devices that may be used as remixing / upmixing systems are schematically described. DETAILED DESCRIPTION

[0021] In reference Figures 1 to 11Before giving a detailed description of the embodiments, some general explanations are made.

[0022] An embodiment discloses an electronic device comprising a circuit configured to: perform blind source separation based on a received input signal to obtain separated sources; perform time domain filtering (202) on at least one of the separated sources to obtain at least one filtered separated source; and perform remixing or upmixing based on the at least one filtered separated source.

[0023] The circuit of the electronic device may include a processor, such as a CPU, memory (RAM, ROM, etc.), memory and / or storage, interface, etc. The circuit may include an input device (mouse, keyboard, camera, etc.), an output device (display (e.g., liquid crystal, (organic) light emitting diode, etc.), a speaker, etc., a (wireless) interface, etc., or may be connected to an input device (mouse, keyboard, camera, etc.), an output device (display (e.g., liquid crystal, (organic) light emitting diode, etc.), a speaker, etc., a (wireless) interface, etc., as is generally known for electronic devices (computers, smart phones, etc.). In addition, the circuit may include a sensor for sensing still images or video image data (image sensor, camera sensor, video sensor, etc.), a sensor for sensing environmental parameters (e.g., radar, humidity, light, temperature), etc., or may be connected to a sensor for sensing still images or video image data (image sensor, camera sensor, video sensor, etc.), a sensor for sensing environmental parameters (e.g., radar, humidity, light, temperature), etc.

[0024] In audio source separation, an input signal including multiple sources (e.g., musical instruments, voices, etc.) is decomposed to separate. Audio source separation can be unsupervised (called "blind source separation", BSS) or partially supervised. "Blind" means that blind source separation does not have to have information about the original source. For example, it may not be necessary to know how many sources the original signal contains or which sound information of the input signal belongs to which original source. The purpose of blind source separation is to decompose the original signal without knowing the previous separation. The blind source separation unit can use any blind source separation technology known to the technician. In (blind) source separation, the source signal with the minimum correlation or maximum independence in the sense of probability or information theory can be searched, or based on non-negative matrix factorization, the structural constraints on the audio source signal can be found. The method for performing (blind) source separation is known to the technician and is based on, for example, principal component analysis, singular value decomposition, (non) correlation component analysis, non-negative matrix factorization, artificial neural network, etc.

[0025] Although some embodiments use blind source separation to generate separated audio source signals, the present disclosure is not limited to embodiments in which further information is not used to separate the audio source signals, but in some embodiments, further information is used to generate separated audio source signals. Such further information may be, for example, information about the mixing process, information about the type of audio source included in the input audio content, information about the spatial location of the audio source included in the input audio content, etc.

[0026] The input signal may be any type of audio signal, may be in the form of an analog signal, a digital signal, may be derived from a compact disc, a digital video disc, etc., may be a data file, such as a wave file, an mp3 file, etc., and the present disclosure is not limited to a specific format of the input audio content. The input audio content may be, for example, a stereo audio signal having a first channel input audio signal and a second channel input audio signal, and the present disclosure is not limited to input audio content having two audio channels. In other embodiments, the input audio content may include any number of channels, such as a remix of a 5.1 audio signal, etc.

[0027] The input signal may include one or more source signals. Specifically, the input signal may include several audio sources. An audio source may be any entity that generates sound waves, such as a musical instrument, speech, vocal music, artificially generated sound (e.g., from a synthesizer), etc.

[0028] The input audio content may represent or comprise mixed audio sources, which means that sound information is not available for all audio sources individually for the input audio content, but that sound information of different audio sources is, for example, at least partially overlapping or mixed.

[0029] The circuit can be configured to perform remixing or upmixing based on at least one filtered separated source and based on other separated sources obtained by blind source separation to obtain a remix signal or an upmix signal. The remixing or upmixing can be configured to perform remixing or upmixing of separated sources (here "vocal", "bass", "drums" and "other") to produce a remix signal or an upmix signal that can be sent to a speaker system. The remixing or upmixing can be further configured to perform remixing or upmixing of one or more separated sources to produce a remix signal or an upmix signal that can be sent to one or more output channels of the speaker system.

[0030] The separations produced from the input signal by blind source separation may, for example, include a "vocal" separation, a "bass" separation, a "drums" separation, and an "other" separation. In the "vocal" separation, all sounds belonging to the vocals may be included, in the "bass" separation, all noises below a predetermined threshold frequency may be included, in the "drums" separation, all noises belonging to the "drums" in the song / music may be included, and in the "other" separation, all remaining sounds may be included.

[0031] Source separation obtained by a music source separation (MSS) system may result in artifacts such as interference, crosstalk or noise. Time domain filtering may be performed on the separated sources produced by blind source separation to reduce the effects of such artifacts, crosstalk or noise.

[0032] Performing time domain filtering can improve the signal-to-distortion ratio (SDR) of the separated sources. That is, performing time domain filtering can reduce artifacts and / or interference that occur in silent parts of musical instruments. Therefore, the objective performance of the separated sources as well as the subjective quality can be improved. In particular, if the separated sources are listened to separately, the performance and quality of the separated sources can be significantly improved. As known to the skilled person, especially when it comes to frequency domain methods, MSS systems never output true zero values. Therefore, time domain filtering can include looking at the output of the MSS system and checking whether "true zeros" occur for long periods of time.

[0033] Furthermore, the separation may also include a residual comprising "other" sounds that cannot be clearly identified as a specific instrument or source.

[0034] The circuit may be configured to perform time domain filtering by processing a window around the current sample and setting the current sample to zero if there is no sample in the window with an amplitude greater than or equal to a threshold. Thus, performing time domain filtering may include setting all signal samples around the window and not exceeding a predetermined threshold to zero.

[0035] The threshold used in the time domain filtering may be based on a predetermined threshold coefficient and on a maximum signal determined from the separated sources.

[0036] The circuit can be configured to perform time domain filtering based on the window size.

[0037] Furthermore, the circuit can be configured to perform time domain filtering based on window shifting.Performing time domain filtering can, for example, include shifting the window to the left to look at past samples; or can include shifting the window to the right to look more at future samples.

[0038] The present disclosure is not limited to a particular number of audio channels and all types of remixing, upmixing, and downmixing may be achieved.

[0039] In some embodiments, the circuit may be configured to perform adaptive time-domain filtering.

[0040] For example, the circuit may be configured to perform adaptive time domain filtering based on the reverberation time. The circuit may, for example, be configured to determine a window size and / or a window shift based on the retrieved reverberation time. The reverberation time may, for example, be T 60 Time, which is a measure of the time required for sound to "fade away" in an enclosed area after the source of the sound has stopped.

[0041] Additionally or alternatively, the circuit may be configured to perform adaptive time-domain filtering of the first separated source based on the detection signal in the second separated source. This solves the fact that the quality of the upmix / remix may depend on the quality of the source separation. A common problem with separating sources into instruments such as "bass", "drums", "others" and "vocals" is that the "drums" and "vocals" are not clearly separated. For example, a portion of the "drums" signal may be mistakenly separated as "vocals". If the remix / upmix system is not aware of the separation failure, the listener will perceive annoying artifacts. For example, if the "vocals" is placed in front of the listener and the "others" is placed behind the listener, it can be perceived that the drums are moving between the front and back. The fact that the "drums" and "vocals" are not clearly separated is a common problem with source separation. If the remix / upmix system is not aware of the separation failure, the listener will perceive annoying artifacts.

[0042] Performing adaptive time-domain filtering of the first separation source based on the detection signal in the second separation source can solve this problem. Specifically, the circuit can be specifically configured to set the threshold coefficient based on the "drum sound" signal detected in the "drum sound" separation source. For example, the signal can be perceived by detecting the "drum sound" signal in the "drum sound" separation source and performing time-domain filtering on the "vocal" separation based on the detected "drum sound" signal.

[0043] In some embodiments, performing time-domain filtering may include performing time-domain filtering on the separated sources “Vocals” and “Drums” based on the parameter τ.

[0044] In some embodiments, the circuit may be further configured to calculate a short-time energy based on the received input signal, and further configured to determine a parameter τ′ based on the calculated short-time energy.

[0045] The circuit may be further configured to output the remixed signal or the upmixed signal to a speaker system.

[0046] In some embodiments, the circuit can be further configured to perform band-limited filtering. Band-limited filtering can be performed by using a filter bank or the like to obtain a band-limited signal. For example, band-limited filtering can be performed on each separated source, and a band-limited signal can be obtained. Time domain filtering can be performed separately on each band-limited separated source signal.

[0047] In some embodiments, performing band-limited filtering may include performing band-limited filtering on the separated sources “vocals” and “drums” to obtain band-limited “vocals” and band-limited “drums”.

[0048] In some embodiments, performing time-domain filtering may include performing time-domain filtering on the band-limited “vocals” and the band-limited “drums” to obtain filtered “vocals” and filtered “drums”.

[0049] Performing a remix or upmix of one or more of the separate sources may include remixing or upmixing of a selected one of the separate sources or a selected subgroup of the separate sources. For example, performing a remix or upmix of one or more of the separate sources may include isolating the selected separation from the separate source. Optionally, performing a remix or upmix may include removing a specific separation from the separate sources. For example, performing a remix or upmix of one or more of the separate sources may include isolating or removing a vocal separation. This may be helpful to listen to a single source separation or mute it, such as in a karaoke setting.

[0050] An embodiment also discloses a method, comprising: performing blind source separation based on a received input signal to obtain separated sources; performing time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and performing remixing or upmixing based on at least one filtered separated source.

[0051] According to another aspect, the present disclosure provides a computer program comprising instructions, which, when executed on a processor, cause the processor to perform the following operations: perform blind source separation based on a received input signal to obtain separated sources; perform time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and perform remixing or upmixing based on the at least one filtered separated source.

[0052] The term "signal" as used herein is not limited to any particular format and may be an analog signal, a digital signal, or a signal stored in a data file, a data stream, or any other format.

[0053] Embodiments will now be described with reference to the accompanying drawings.

[0054] Audio upmixing / remixing via Blind Source Separation (BSS)

[0055] Figure 1 A general approach to audio upmixing / remixing via blind source separation (BSS) is schematically illustrated.

[0056] First, a source separation (also called "degradation") is performed, which decomposes a source audio signal 1 including multiple channels I and audio from multiple audio sources Source 1, Source 2...Source K (e.g., musical instruments, voices, etc.) into "separations", here into source estimates 2a to 2d for each channel i, where K is an integer and represents the number of audio sources. In the embodiment here, the source audio signal 1 is a stereo signal with two channels i=1 and i=2. Because the separation of the audio source signals may be imperfect, for example, due to a mix of audio sources, a residual signal 3 (r(n)) is generated in addition to the separated audio signals 2a to 2d. The residual signal can, for example, represent the difference between the input audio content and the sum of all separated audio source signals. The audio signal emitted by each audio source is represented in the input audio content 1 by its corresponding recorded sound wave. For input audio content with more than one audio channel (such as stereo or surround sound input audio content), the spatial information of the audio source is usually also included or represented by the input audio content, for example by the proportion of the audio source signals included in the different audio channels. Based on blind source separation or other techniques capable of separating audio sources, the input audio content 1 is separated into separated audio source signals 2a to 2d and a residue 3.

[0057] In a second step, the separations 2a to 2d and the possible residue 3 are remixed and presented as a new loudspeaker signal 4, here a signal comprising five channels 4a to 4e (i.e. a 5.0 channel system). Based on the separation audio source signal and the residue signal, an output audio content is generated by mixing the separation audio source signal and the residue signal based on the spatial information. The output audio content is presented in Figure 1 It is shown by way of example in FIG. 1 and is denoted by reference numeral 4 .

[0058] In the following, the number of audio channels of the input audio content is referred to as M. in , and the number of audio channels of the output audio content is referred to as M out .because Figure 1 The input audio content 1 in the example has two channels i=1 and i=2, and Figure 1 The output audio content 4 in the example of FIG. 4 has five channels 4a to 4e, so M in =2 and M out =5. Figure 1 The method in is generally referred to as remixing, and in particular, if M in <M out , it is called upmixing. Figure 1 In the example of FIG. 1 , the number of audio channels of the input audio content 1 is M. in =2 is less than the number M of audio channels of the output audio content 4 out =5, so this is an upmix from stereo input audio content 1 to 5.0 surround output audio content 4.

[0059] Remixing / upmixing based on time domain filtering

[0060] Figure 2 The process of remixing / upmixing based on the time domain filtering method is schematically shown. The process includes blind source separation 201, time domain filtering 202 and remixing / upmixing 203. Figure 1 The source 1, 2, ... K) has multiple channels (e.g., M in =2) input signal (see Figure 1 The input signal 1) is input to the blind source separation 201 and is decomposed into Figure 1 The separation (see Figure 1 Separation sources 2a to 2d in ), here broken down into "Vocal", "Drums", "Bass" and "Other". Separation sources (see Figure 1 The separated signals 2) (here "Vocal", "Drums", "Bass" and "Others") are transmitted to the time domain filtering 202 and the remixing / upmixing 203. Specifically, the "Vocal" separation source and the "Drums" separation source are transmitted to the time domain filtering 202, while the "Bass" separation source and the "Others" separation source are transmitted to the remixing / upmixing 203.

[0061] The time domain filtering 202 is configured to perform time domain filtering on the “vocals” and the “drums” to generate filtered “vocals” and filtered “drums” (see Figure 4 403 in Figure 5 , and corresponding description). The time domain filtering 202 is, for example, improving the signal-to-distortion ratio (SDR) of "vocals" and "drums". The resulting filtered "vocals" and filtered "drums" are transmitted to the remix / upmix 203. In addition, "Bass" and "Others" are also transmitted to the remix / upmix 203. The remix / upmix 203 remixes / upmixes the filtered "vocals" and filters "drums", "Bass" and "Others". In addition, the remix / upmix 203 sends the remix / upmix signal to the speaker system 204.

[0062] Figure 3 Schematically shown are examples of waveforms for vocal separation and waveforms for vocal "truth", both represented on a logarithmic scale. The upper waveform is the waveform of the vocal separation output of a Music Source Separation (MSS) system, while the lower waveform is the truth waveform associated with the vocal separation. In both waveforms (upper and lower), the x-axis represents time in seconds, extending from 0 seconds to 210 seconds, and the y-axis represents the corresponding amplitude of the signal (here in dB). The vocal "truth" waveform is the waveform of the vocal recorded independently of all instruments. The vocal "truth" waveform is the waveform of the vocal before it was mixed with the instruments to obtain one audio signal and then the audio signal (see Figure 1 and Figure 2 The waveform of the vocal separation is the waveform of the audio signal (see Figure 1 and Figure 2 The waveform obtained after performing blind source separation on the input signal in 1). As can be taken from these two waveforms, the separation obtained by the MSS system (see waveforms above) is noisy. In particular, where the vocal truth value represents silence, the vocal separation includes noise. This noise represents artifacts and / or interference generated by the MSS, such as crosstalk from "drums" to "vocals".

[0063] Figure 4 FIG. 4 is a flow chart showing a method for visualizing blind source separation based on time domain filtering. The method can be performed, for example, by a processor ( Fig.11 At 400, an input signal is received. At 401, the received input signal is stored in a buffer. At 402, blind source separation is performed based on the received input signal (e.g., see Figure 1 ,as well as Figure 2 At 401 in FIG. 201 ) to obtain separated sources including “vocals”, “bass”, “drums”, and “others”. At 403, time domain filtering is performed on “vocals” and “drums” to obtain filtered “vocals” and filtered “drums”. At 404, remixing / upmixing is performed based on the filtered “vocals” and filtered “drums” obtained at 403 and based on the “bass” and “others” obtained at 402 to obtain a remixed / upmixed signal (see FIG. 202 ). Figure 2 At 405, the remixed / upmixed signal is output to the speaker system ( Figure 2 204).

[0064] In the following, the time domain filtering 403 performed on "Vocal" and "Drums" is described in more detail to obtain Filtered "Vocal" and Filtered "Drums". The idea of ​​the time domain filtering 403 is to exploit the fact that many small amplitude samples in the separation (here "Vocal" and "Drums") are noise and can therefore be safely set to zero. Since the time domain filtering 403 works in the time domain, this knowledge can be exploited.

[0065] use The waveforms representing the separated sources (here, for example, "vocal" and "drum"), where i=1, ..., I is the channel index. For example, 1=2 corresponds to reference Figure 1 The stereo case is described in more detail.

[0066] In time domain filtering, consider the current sample at time instance n The window around it and if there is no amplitude greater than or equal to samples, where τ represents the threshold coefficient, that is, if

[0067]

[0068] Where w represents the window length, s represents the window shift, and m represents the samples in window w, then for i=1,…,I, Set to zero.

[0069] If (Formula 1) is true, then Set to zero.

[0070] Window shift s shifts the window w by a shift s. In the time domain filtering method, the reverberation tail should not be truncated. The reverberation tail is the persistence of the sound after the sound is generated, that is, the reflection of the sound continues and the amplitude decreases until it reaches zero amplitude. The reverberation tail is very important, especially for "vocals". That is, the window can be shifted to the left, allowing more viewing of past samples rather than future samples. If s<0, the window is shifted to the left, and if s>0, the window is shifted to the right. If the window is shifted to the left, more past samples are included in the window (that is, "more viewing of past samples"). Similarly, if the window is shifted to the right, more future samples are included in the window (that is, "more viewing of future samples").

[0071] That is, the time domain filter 403 acts as a "gate" that only detects the presence of a signal when the signal within the window exceeds a threshold. The signal is allowed to pass.

[0072] Figure 5 A diagram of this time domain filtering approach is schematically shown. The waveforms of the separated sources (here for example "Vocal" and "Drums") are shown Where i = 1, ..., I represents the channel index, and n represents the current sample. The time instances (respectively the number of samples), w represents the window length, and s represents the window shift.

[0073] The technician can select the parameters w, s and τ according to the specific use case. For example, the example values ​​of these parameters w, s and τ can be found using a validation set and finding the value that maximizes the signal-to-distortion ratio (SDR), as defined in "E. Vincent, RGribonval and C. Févotte, Performance measurement in blind audio sources separation, IEEE Trans. Audio, Speech and Language Processing, 14 (4), pp 14621489, 2006".

[0074] Figure 6 The flowchart of the method for visualizing adaptive time domain filtering is shown, where the parameters s and w are suitable for the reverberation time (T 60 time). The method may be performed, for example, by a processor ( Fig.11 At 600, an input signal is received. At 601, the received input signal is stored in a buffer. At 602, blind source separation is performed based on the received input signal to obtain separated sources including "vocal", "bass", "drums" and "others". At 603, the reverberation time (T 60 time), and the process continues at 604. At 604, parameters s, w for time domain filtering are determined based on the retrieved reverberation time. At 605, time domain filtering is performed on "Vocal" and "Drums" based on the parameters s, w for time domain filtering determined at 604 to obtain Filtered "Vocal" and Filtered "Drums". At 606, remixing / upmixing is performed based on the Filtered "Vocal" and Filtered "Drums" obtained at 605 and based on the "Bass" and "Others" obtained at 602 to obtain a remixed / upmixed signal (see Figure 2 At 607, the remixed / upmixed signal is output to the speaker system 204 (see Figure 2 ).

[0075] In the following, how the reverberation time is retrieved at 603 is described in more detail. Reverberation time is a measure of the time required for a sound to "fade away" in an enclosed area after the sound source has stopped. The reverberation time may, for example, be defined as the time it takes for a sound to gradually decay to a level 60 dB below its original level (T 60 time). The reverberation time T 60 Given by the Sabine equation:

[0076]

[0077] Among them, c 20 is the speed of sound in the room (at 20 degrees Celsius), V is in m 3 The volume of the room in units of m 2 is the total surface area of ​​the room in units, a is the average absorption coefficient of the room surface, and the product Sa is the total absorption. That is, when the parameters V, S, a of the room are known (for example, in the case of recording), T can be determined as defined above 60 time.

[0078] Alternatively, the reverberation time may be determined by analyzing the waveform, as defined by Rama Ratnam et al., "Blindestimation of reverberation time", J. Acoust. Soc. Am. 114 (5), November 2003.

[0079] Still alternatively, the reverberation time may be obtained from knowledge about the audio processing chain that produced the input signal (eg the reverberation time may be a predetermined parameter set in a reverberation processor, such as an algorithm or convolution reverberation used in the processing chain).

[0080] In the following, how the reverberation time (T 60 time) to adaptively determine the parameters s, w. Retrieve the parameters s, w so that the reverberation tail (see Figure 5 ) is not truncated. That is, the gate can be (See Figure 4 The gate is "open" for at least as long after each separated source signal (T 403) has been transmitted and until the reverberation falls below a certain threshold. The threshold is chosen so as to prevent truncation of the reverberation tail. That is, the "gate" will be adapted to the reverberation time (T 60 time).

[0081] Figure 7 The process of remixing / upmixing based on band-limited filtering and based on time-domain filtering is schematically shown. Figure 7 The process is similar to Figure 2 The process includes blind source separation 201, time domain filtering 202, remixing / upmixing 203 and band-limiting filtering 205. Figure 1 The source 1, 2, ... K) has multiple channels (e.g., M in =2) input signal (see Figure 1 The input signal 1) is input to the blind source separation 201 and is decomposed into Figure 1 The separation (see Figure 1 Separation sources 2a to 2d in ), here broken down into "Vocal", "Drums", "Bass" and "Other". Separation sources (see Figure 1 The separated signals in 2) (here "Vocal", "Drum", "Bass" and "Others") are transmitted to the band-limited filter 205 and the remix / upmix 203. Specifically, "Vocal" and "Drum" are transmitted to the band-limited filter 205, while "Bass" and "Others" are transmitted to the remix / upmix 203.

[0082] Band-limited filtering 205 is configured to perform band-limited filtering on "vocals" and "drum sounds" using a filter bank to obtain band-limited signals, namely, band-limited "vocals" and band-limited "drum sounds". Technicians can select the parameters of the filter bank according to the specific use case. Time-domain filtering 202 can be configured to perform time-domain filtering on the band-limited "vocals" and the band-limited "drum sounds" to generate filtered "vocals" and filtered "drum sounds". The generated filtered "vocals" and filtered "drum sounds" are transmitted to remixing / upmixing 203. Remixing / upmixing 203 remixes / upmixes filtered "vocals", filtered "drum sounds", "bass" and "others". In addition, remixing / upmixing 203 sends the remixed / upmixed signal to the speaker system 204. The idea of ​​the time-domain filtering method is to utilize the fact that time-domain filtering will improve the signal-to-distortion ratio (SDR) of "vocals" and "drum sounds". Since time domain filtering is performed separately on each of the band-limited separated source signals (here, band-limited “vocals” and band-limited “drums”), the SDR of the band-limited “vocals” and the band-limited “drums” can be improved.

[0083] The parameters τ, s, w can be selected according to the instrument.

[0084] Figure 8 An exemplary graph is shown that visualizes possible correlations for separated sources including "Vocal", "Drums", "Bass", and "Others" with parameters τ, s, w. The parameter τ represents a threshold coefficient, the parameter s represents a window shift, and the parameter w represents a window length. The parameters τ, s, w depend on the instrument, and when the input signal is separated into "Vocal", "Drums", "Bass", and "Others", the parameters τ, s, w depend on each of the above separated sources. The optimal values ​​for the parameters τ, s, w can be determined, for example, by a validation dataset, as shown above Figure 5 In this embodiment, for example, for "Vocal", parameter τ is 0.3, parameter s is 0.3, and parameter w is 2. For "Drum", parameter τ is 0.4, parameter s is 0.3, and parameter w is 3. For "Bass", parameter τ is 0.2, parameter s is -0.3, and parameter w is 4. For "Other", parameter τ is 0.1, parameter s is -0.3, and parameter w is 5.

[0085] Blind source separation (BSS) is usually not perfect. For example, crosstalk can cause interference in the separated sources. In particular, it has been recognized that crosstalk occurs from drums to vocals.

[0086] Figure 9aA flow chart of a visual time-domain filtering method is shown, wherein a parameter τ is adapted to the detection of a drum signal in a "drum sound" separation. In this embodiment, if a "drum sound" signal is detected in the "drum sound" separation, a threshold coefficient τ for the "vocal" separation source can be increased to avoid crosstalk from "drum sound" to "vocal". At 900, an input signal is received. At 901, the received input signal is stored in a buffer. At 902, blind source separation is performed on the received input signal to obtain a separation source including "vocal", "bass", "drum sound" and "others". At 903, it is detected whether a drum signal exists in the "drum sound" separation. If a drum signal is detected in the "drum sound" separation, the method continues at 904. If a drum signal is not detected in the "drum sound" separation, the method continues at 905. At 904, the parameter τ is set to 1, and the method continues at 906. At 905, the parameter τ is set to 0.2, and the method continues at 906. At 906, based on the parameter τ determined at 904 or 905, time domain filtering is performed on “Vocal” and “Drum” to obtain filtered “Vocal” and filtered “Drum”. At 907, remixing / upmixing is performed based on the filtered “Vocal” and filtered “Drum” obtained at 906 and based on the “Bass” and “Others” obtained at 902 to obtain a remixed / upmixed signal (see Figure 2 At 908, the remixed / upmixed signal is output to the speaker system 204 (see Figure 2 ).

[0087] about Figure 9b , describing in more detail how to detect whether a drum signal is present in the "drum sound" separation at 903. Figure 9b An example of a "drum sound" audio signal (here, the signal of a snare drum) is schematically shown. The snare drum causes a strong signal peak and a reverberation tail in the drum sound signal. In 903, it is detected whether a snare drum event is currently detected in the "drum sound" separation. In this embodiment, the y-axis represents the amplitude of the signal (here, the snare drum), and the x-axis represents time. Specifically, time t1 represents the time when a sound "attack" is detected in the "drum sound" separation, where the sound "attack" is a strong signal peak, for example, a signal with a high amplitude. Time t2 represents the time when a release is detected in the "drum sound" separation, where a release is detected when the amplitude of the signal is below a predetermined threshold. The amplitude of the signal can, for example, be related to the short-time energy of the signal. The short-time energy of the "drum sound" signal x(n) is given by the following formula:

[0088]

[0089] where x(n) is the signal and k is the index in the time window [-K, K] where the short-term energy is calculated.

[0090] Fig.10 A flow chart of a method for visualizing time domain filtering based on a parameter τ calculated based on a short-time energy associated with a received input signal is shown. At 1000, an input signal is received. At 1001, the received input signal is stored in a buffer. At 1002, blind source separation is performed on the received input signal to obtain separated sources including "vocals", "bass", "drums", and "others". At 1003, short-time energy is calculated based on the received input signal (see Formula 2 above). At 1004, a parameter τ′ is determined based on the calculated short-time energy. At 1005, based on the parameter τ′ determined at 1004, time domain filtering is performed on "vocals" and "drums" to obtain filtered "vocals" and filtered "drums". At 1006, remixing / upmixing is performed based on the filtered "vocals" and filtered "drums" obtained at 1005 and based on the "bass" and "others" obtained at 1002 to obtain a remixed / upmixed signal (see Figure 2 At 1007, the remixed / upmixed signal is output to the speaker system 204 (see Figure 2 ).

[0091] That is, when the threshold exceeds the threshold , the door can be "open" (see Figure 4 403 in ), and the new threshold τ′ reflecting the short-time energy is given by the following formula:

[0092]

[0093] Wherein, τ represents a threshold coefficient, and i represents a channel index.

[0094] That is, the "door" can change dynamically.

[0095] Implementation

[0096] Fig.11 An embodiment of an electronic device that can implement the remixing / upmixing process based on time domain filtering as described above is schematically described. The electronic device 1200 includes a CPU 1201 as a processor. The electronic device 1200 further includes a microphone array 1210, a speaker array 1211 and a convolutional neural network unit 1220 connected to the processor 1201. The processor 1201 can, for example, implement blind source separation 201, remixing / upmixing 203, time domain filtering 202 and / or band-limited filtering 205, which are implemented with reference to Figure 2 and Figure 7 The CNN unit may be, for example, an artificial neural network in hardware form, such as a neural network on a GPU or any other hardware dedicated to the purpose of implementing an artificial neural network. The speaker array 1211 consists of one or more speakers distributed in a predetermined space (see Figure 2 and Figure 7 204) and is configured to present 3D audio, as described in the above embodiments. The electronic device 1200 further includes a user interface 1212 connected to the processor 1201. The user interface 1212 acts as a human-machine interface and enables a dialogue between an administrator and the electronic system. For example, an administrator can use the user interface 1212 to configure the system. The electronic device 1200 further includes an Ethernet interface 1221, a Bluetooth interface 1204, and a WLAN interface 1205. These units 1204, 1205 act as I / O interfaces for data communication with external devices. For example, additional speakers, microphones, and cameras with Ethernet, WLAN, or Bluetooth connections can be coupled to the processor 1201 via these interfaces 1221, 1204, and 1205.

[0097] The electronic device 1200 further comprises a data storage 1202 and a data memory 1203 (here RAM). The data memory 1203 is arranged to temporarily store or cache data or computer instructions for processing by the processor 1201. The data storage 1202 is arranged as a long-term memory, for example for recording sensory data obtained from the microphone array 1210 and provided to or retrieved from the CNN unit 1220. The data storage 1202 may also store audio data representing an audio message, which the public address system may transmit to a person moving in a predetermined space.

[0098] It should be noted that the above description is only an example configuration. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces, etc.

[0099] It should be appreciated that the embodiments describe methods with an exemplary order of method steps. However, the specific order of method steps is given only for illustrative purposes and should not be interpreted as being binding.

[0100] It should also be noted that Fig.11 The electronic system is divided into units for illustrative purposes only, and the present disclosure is not limited to any specific functional division in a specific unit. For example, at least a portion of the circuit can be implemented by a separately programmed processor, a field programmable gate array (FPGA), a dedicated circuit, etc.

[0101] If not stated otherwise, all units and entities described in this specification and claimed in the appended claims may be implemented as integrated circuit logic, for example, on a chip, and if not stated otherwise, the functions provided by such units and entities may be implemented by software.

[0102] To the extent that the embodiments of the present disclosure described above are implemented at least in part using software controlled data processing equipment, it should be understood that providing such software controlled computer programs and transmission, storage or other media providing such computer programs are contemplated as aspects of the present disclosure.

[0103] Note that this technique can also be configured as follows:

[0104] (1) An electronic device comprising a circuit, wherein the circuit is configured to:

[0105] performing (402; 602; 902; 1002) blind source separation (201) based on the received input signal (1) to obtain separated sources;

[0106] performing (403; 605; 906; 1005) time domain filtering (202) on at least one of the separated sources to obtain at least one filtered separated source; and

[0107] Remixing or upmixing (203) is performed (404; 606; 907; 1006) based on at least one filtered separated source.

[0108] (2) The electronic device according to (1), wherein the circuit is configured to process the current sample The surrounding window (w, s) and if there is no sample with an amplitude greater than or equal to the threshold in the window (w, s), the current sample Set to zero to perform time-domain filtering.

[0109] (3) The electronic device according to (1) or (2), wherein the circuit is configured to be based on the threshold coefficient (τ) and based on the maximum signal Determine the threshold value.

[0110] (4) An electronic device according to any one of (1) to (3), wherein the circuit is configured to perform (404; 606; 907; 1006) remixing or upmixing (203) based on at least one filtered separated source and based on the separated source obtained by blind source separation (201) to obtain a remixed signal or an upmixed signal.

[0111] (5) The electronic device of any one of (1) to (4), wherein the circuit is configured to perform time-domain filtering (202) based on a window size (w).

[0112] (6) The electronic device of (2), wherein the circuit is configured to perform time domain filtering (202) based on the window shift (s).

[0113] (7) An electronic device according to any one of (1) to (6), wherein the result of the blind source separation (201) includes vocal separation or drum separation, and wherein the circuit is configured to perform time domain filtering (202) on the vocal separation and the drum separation, respectively, to obtain filtered vocals or filtered drums.

[0114] (8) The electronic device according to any one of (1) to (7), wherein the circuit is further configured to output (405; 607; 908; 1007) the remix signal or the upmix signal to the speaker system (204).

[0115] (9) The electronic device according to any one of (1) to (8), wherein the circuit is further configured to perform band-limited filtering (205) on the separated source to obtain the band-limited separated source.

[0116] (10) The electronic device of any one of (1) to (9), wherein the circuit is configured to perform adaptive time-domain filtering (202).

[0117] (11) The electronic device of (10), wherein the circuit is configured to perform adaptive time-domain filtering (202) based on reverberation time.

[0118] (12) The electronic device of (11), wherein the circuit is configured to determine the window size and / or the window shift based on the reverberation time.

[0119] (13) The electronic device of (10), wherein the circuit is configured to perform adaptive time-domain filtering (202) of the first separated source based on the detection signal in the second separated source.

[0120] (14) The electronic device according to (13), wherein the second signal is a drum sound signal in a drum sound separation source.

[0121] (15) An electronic device according to any one of (1) to (14), wherein the circuit is further configured to calculate (1003) short-time energy based on the received input signal (1), and is further configured to determine (1004) a threshold coefficient (τ′) for time domain filtering (202) based on the calculated short-time energy.

[0122] (16) An electronic device according to any one of (1) to (15), wherein the remixing or upmixing (203) of one or more of the separate sources includes remixing or upmixing of a selected one of the separate sources or a selected subgroup of the separate sources.

[0123] (17) The electronic device according to (16), wherein the remixing or upmixing (203) of one or more of the separate sources includes isolating a specific separate source from the separate sources.

[0124] (18) The electronic device according to (16), wherein the remixing or upmixing (203) of one or more of the separate sources includes removing a specific separate source from the separate sources.

[0125] (19) A method comprising:

[0126] performing (402; 602; 902; 1002) blind source separation (201) based on the received input signal (1) to obtain separated sources;

[0127] performing (403; 605; 906; 1005) time domain filtering (202) on at least one of the separated sources to obtain at least one filtered separated source; and

[0128] Remixing or upmixing (203) is performed (404; 606; 907; 1006) based on at least one filtered separated source.

[0129] (20) A computer program comprising instructions which, when executed on a processor, cause the processor to perform the following steps:

[0130] performing (402; 602; 902; 1002) blind source separation (201) based on the received input signal (1) to obtain separated sources;

[0131] performing (403; 605; 906; 1005) time domain filtering (202) on at least one of the separated sources to obtain at least one filtered separated source; and

[0132] Remixing or upmixing (203) is performed (404; 606; 907; 1006) based on at least one filtered separated source.

Claims

1. An electronic device comprising a circuit, wherein the circuit is configured to: performing blind source separation based on the received input signal to obtain separated sources; performing time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and performing a remix or an upmix based on the at least one filtered separated source, in, The circuit is configured to perform the time domain filtering by processing a window around a current sample and setting the current sample to zero if there is no sample in the window with an amplitude greater than or equal to a threshold, And wherein the circuit is configured to perform the remixing or upmixing based on the at least one filtered separated source and based on the separated source obtained by the blind source separation to obtain a remix signal or an upmix signal.

2. The electronic device according to claim 1, wherein: The circuit is configured to determine a threshold based on a threshold coefficient and based on a maximum signal.

3. The electronic device according to claim 1, wherein: The circuit is configured to perform the time-domain filtering based on a window size.

4. The electronic device according to claim 1, wherein: The circuit is configured to perform the time-domain filtering based on window shifting.

5. The electronic device according to claim 1, wherein: The result of the blind source separation includes vocal separation or drum sound separation, and wherein the circuit is configured to perform the time domain filtering on the vocal separation and the drum sound separation respectively to obtain filtered vocals or filtered drum sounds.

6. The electronic device according to claim 1, wherein: The circuit is further configured to output the remix signal or the upmix signal to a speaker system.

7. The electronic device according to claim 1, wherein: The circuit is further configured to perform band-limited filtering on the separated source and obtain a band-limited separated source.

8. The electronic device according to claim 1, wherein: The circuit is configured to perform adaptive time-domain filtering.

9. The electronic device according to claim 8, wherein: The circuit is configured to perform adaptive time-domain filtering based on reverberation time.

10. The electronic device according to claim 9, wherein: The circuit is configured to determine a window size and / or a window shift based on the reverberation time.

11. The electronic device according to claim 8, wherein: The circuit is configured to perform adaptive time-domain filtering of the first separated source based on the detection signal in the second separated source.

12. The electronic device according to claim 11, wherein: The second signal is a drum signal in the drum separation source.

13. The electronic device according to claim 1, wherein: The circuit is further configured to calculate short-time energy based on the received input signal, and is further configured to determine a threshold coefficient for the time domain filtering based on the calculated short-time energy.

14. The electronic device according to claim 1, wherein: The remix or upmix of one or more of the separate sources comprises a remix or upmix of a selected one of the separate sources or a selected subgroup of the separate sources.

15. The electronic device according to claim 14, wherein: The remixing or upmixing of one or more of the separate sources comprises isolating a particular separate source from the separate sources.

16. The electronic device according to claim 14, wherein: The remixing or upmixing of one or more of the separate sources comprises removing a particular separate source from the separate sources.

17. A method for blind source separation and remixing or upmixing, comprising: performing blind source separation based on the received input signal to obtain separated sources; performing time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and performing a remix or an upmix based on the at least one filtered separated source, wherein the time domain filtering is performed by processing a window around a current sample and setting the current sample to zero if there is no sample in the window with an amplitude greater than or equal to a threshold, And wherein said remixing or upmixing is performed based on said at least one filtered separated source and based on said separated source obtained by said blind source separation to obtain a remix signal or an upmix signal.

18. A computer storage medium storing a computer program comprising instructions which, when executed on a processor, cause the processor to perform the following steps: performing blind source separation based on the received input signal to obtain separated sources; performing time domain filtering on at least one of the separated sources to obtain at least one filtered separated source; and performing a remix or an upmix based on the at least one filtered separated source, in, performing the time domain filtering by processing a window around a current sample and setting the current sample to zero if there is no sample in the window with an amplitude greater than or equal to a threshold, And wherein said remixing or upmixing is performed based on said at least one filtered separated source and based on said separated source obtained by said blind source separation to obtain a remix signal or an upmix signal.

Images