Electronic device, method and computer program product

Abstract

The electronic device comprises circuitry configured to analyze the results of the stereo or multi-channel source separation to determine one or more time-varying parameters, and to create a spatially dynamic audio object based on the one or more time-varying parameters.

Description

[Technical field]

[0001] The present disclosure relates generally to the field of audio processing, and in particular to devices, methods and computer programs for source separation and mixing. [Background technology]

[0002] There is a great deal of audio content available, for example in the form of compact discs (CDs), tapes, audio data files downloadable from the Internet, as well as, for example, video soundtracks stored on digital video discs. Typically, the audio content has already been mixed, for example for a mono or stereo setting, without preserving the original audio source signals from the original audio sources used to generate the audio content, however, there are situations or applications where mixing of audio content is envisioned.

[0003] With the advent of spatial audio object oriented systems like Dolby Atmos, DTS-X, or the more recent Sony 360RA, we need to find some way to enjoy the large legacy content that was not originally interwoven with the concept of audio oriented objects. Some existing upmixing systems attempt to extract spectral-based features, while others attempt to add external effects to spatially render legacy content. Thus, while techniques generally exist for mixing audio content, it is generally desirable to improve devices and methods for mixing audio content. Summary of the Invention

[0004] According to a first aspect, the present disclosure provides an electronic device comprising circuitry configured to analyze results of stereo or multi-channel source separation to determine one or more time-varying parameters and to create a spatially dynamic audio object based on the one or more time-varying parameters.

[0005] According to a further aspect, the present disclosure provides a method including analyzing results of the stereo or multi-channel source separation to determine one or more time-varying parameters, and creating a spatially dynamic audio object based on the one or more time-varying parameters.

[0006] Further aspects are set out in the dependent claims, the following description and the drawings. [Brief description of the drawings]

[0007] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates a schematic of a general approach for audio upmixing / remixing by blind source separation (BSS), such as music source separation (MSS). [Diagram 2] 13 illustrates a schematic process of automatic time-dependent spatial side mixing of separated sources, where a monopole placement is performed based on the calculated side-mid ratio. [Diagram 3] 3 shows a detailed exemplary embodiment of the process of spatial upmixing of separated sources as described in FIG. 2. [Figure 4] Fig. 4a illustrates generally an embodiment of a beat detection process as described in Fig. 3 performed on an original stereo signal; Fig. 4b illustrates generally an embodiment of a beat detection process performed in the process of spatial upmixing of separated sources described in Fig. 3; [Diagram 5]Fig. 5(a) is a schematic diagram illustrating an embodiment of a side-mid ratio calculation performed in the process of spatial upmixing of separated sources described in Fig. 3; Fig. 5(b) illustrates a result of the side-mid ratio calculation described in Fig. 5(a); and Fig. 5(c) is a schematic diagram illustrating an embodiment of a silence suppression process that may be performed during the side-mid ratio calculation process of separated sources described in Fig. 5(a). [Figure 6a.b] Fig. 6(a) illustrates generally one embodiment of a segmentation process performed in the process of spatial upmixing of separated sources described in Fig. 3. Fig. 6(b) illustrates a beat-by-beat side-mid ratio clustering process, which is included in the segmentation process as described with reference to Fig. 6(a). [Figure 6c.d] Figure 6(c) provides an embodiment of a clustering process that can be applied to segment the separated sources. Figure 6(d) shows the side-mid ratios per beat clustered into segments as described in Figure 6(a). [Figure 7] Fig. 6a shows a schematic of a time smoothing process in which the side-mid ratios of the separated sources are averaged over segments of the separated sources. Fig. 6b shows an example of the smoothing process. The first segment S identified by the segmentation process in Fig. 6a is associated with a smoothed side-mid ratio. [Figure 8a] 13 illustrates an example embodiment of a position mapping that determines the position of a monopole used to render a separated source. [Figure 8b] 13 illustrates a further exemplary embodiment of a position mapping for determining the position of a monopole used for rendering a separated source. [Figure 8c] 13 illustrates a further exemplary embodiment of a position mapping for determining the position of a monopole used for rendering a separated source. [Figure 9] A visualization of how the position mapping relates to the specified positions of the two monopoles used to render the left and right stereo channels of a separated source. [Figure 10] An embodiment of 3D audio rendering based on a digitized monopole synthesis algorithm is provided. [Figure 11] 1 illustrates a schematic diagram of an embodiment of a process for automatic time-varying spatial upmixing of four separated sources. [Figure 12] 1 shows a flow diagram for visualizing a method for performing time-varying spatial upmixing of separated sources. [Figure 13] 1 illustrates a schematic representation of an embodiment of an electronic device capable of implementing a process of automatic time-dependent spatial upmixing of separated sources. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Before proceeding to a detailed description of the embodiments with reference to Figures 1 to 11, some general remarks will be made.

[0009] The present embodiment discloses an electronic device including a circuit configured to analyze results of stereo or multi-channel source separation to determine one or more time-varying parameters, and to create a spatially dynamic audio object based on the one or more time-varying parameters.

[0010] In this way, the electronic device can provide spatial audio object-oriented audio content, which creates a more natural sound compared to conventional stereo audio content. By taking into account time-varying parameters, for example, time-dependent spatial upmixing, which pre-fulfills the original balance of the content, can be achieved by analyzing the results of multi-channel (source) separation and creating spatially dynamic audio objects.

[0011] The circuitry of an electronic device includes a processor, which may be, for example, a CPU, a memory (RAM, ROM, etc.), and / or a storage interface, etc. The circuitry may also include or be connected to input means (mouse, keyboard, camera, etc.), output means (displays (LCD, (organic) LED, etc.)), loudspeakers, etc., (wireless) interfaces, etc., as commonly known for electronic devices (computers, smartphones, etc.). Additionally, the electronic device may be any audio-enabled product that produces a multi-channel spatial rendering, such as a television, a soundbar, a multi-channel (playback) system, a virtualization device on headphones, a binaural headphone, etc.

[0012] As mentioned at the beginning, there are many audio contents that are already mixed as a stereo audio content signal, which has two audio channels. Specifically, in traditional stereo, each sound of the audio signal is fixed to a specific channel. For example, in one channel there may be fixed instruments such as guitar, drums, etc., and in the other channel there may be fixed instruments such as guitar, vocals, etc. Thus, the sound of each channel is associated with a particular speaker.

[0013] Thus, the circuitry may be configured to determine as time-varying parameters the signal level between the separated channels - a parameter describing loudness, and / or a spectral balance parameter, and / or a main ambience indicator, and / or a dry-wet indicator, and / or a parameter describing percussive-harmonic content.

[0014] Furthermore, the location mapping may include audio object positioning, which may be, for example, genre-dependent or may be dynamically calculated based on a combination of different indexes. The location mapping may be implemented, for example, using an algorithm as described in the following embodiments. For example, you can modify parameters of audio objects like diffusion in monopole synthesis using the Dry / Wet Primary / Ambient indicators or in combination with any of the ratios of separated sources, thus creating a more enveloping sound field.

[0015] When performing upmixing, the electronic device may modify the original content, taking into account especially the balance of instruments in case of stereo content.

[0016] In particular, the circuitry may be configured to determine parameters describing the balance of instruments in stereo content as time-varying parameters, and to create a spatially dynamic audio object based on the balance of instruments in the stereo content.

[0017] The circuitry may be configured to determine a side-mid ratio of the separated sources as a time-varying parameter and to create a spatially dynamic audio object based on the side-mid ratio.

[0018] In this way, electronic devices can create spatial mixes that are content-dependent and more naturally and intuitively match the original intent of the mixing engineer or composer. The derived metadata can also be used as a starting point for audio engineers to create new spatial mixes.

[0019] The circuitry may be configured to determine spatial positioning parameters for the audio object based on one or more time-varying parameters obtained from the result of stereo or multi-channel source separation.

[0020] Determining the spatial positioning parameters may include performing a position mapping based on a positioning index, which allows for selecting a position for the audio object from an array of possible positions. Furthermore, performing the position mapping may automatically create a spatial object audio mix from an analysis of existing multi-channel content, etc.

[0021] In an embodiment, the circuitry may be further configured to perform segmentation based on a side-mid ratio to obtain segments of the separated sources.

[0022] In some embodiments, the side-mid ratio calculation may include a silence suppression process, which may include silence detection in the stereo channels. If silence is present in the separated source, the side-mid ratio may be set to zero.

[0023] The circuitry may be configured to dynamically adapt positioning parameters of the audio objects. The spatial positioning parameters may be, for example, a positioning index, an array of positioning indices, a vector of positions, an array of positions, etc. Some embodiments may use a positioning index depending on the original balance between the separated channels of the music source separation process, without the invention being limited in that respect.

[0024] Deriving spatial positioning parameters may result in a spatial mix, where each separated (instrument) source may be treated separately. The spatial mix may be content dependent and may be natural, intuitive, and consistent with the user's original intent. The derived content may be derived metadata, which may be used as a starting point to create new spatial mixes, etc.

[0025] The circuitry may be configured to create spatially dynamic audio objects by monopole synthesis. For example, the circuitry may be configured to dynamically adapt a spread in the monopole synthesis. In particular, the spatially dynamic audio object may be a monopole.

[0026] The circuitry may be configured to dynamically create a first monopole used to render a left channel of the separated source and a second monopole used to render a right channel of the separated source based on one or more time-varying parameters.

[0027] The circuitry may be configured to generate a time-dependent spatial upmix from the result of the multi-channel source separation that preserves the original balance of the content.

[0028] The circuitry may be configured to perform a segmentation process to obtain separated segments of the source based on the time-varying parameters.

[0029] In an embodiment, the automatic time-varying spatial upmixing is based on the results of a similarity analysis of the multi-channel content. The automatic time-varying spatial upmixing can be implemented, for example, using an algorithm as described in the following embodiments.

[0030] The circuitry may be configured to perform cluster detection based on the time-varying parameters. Cluster detection may be performed using an algorithm such as that described in the following embodiments.

[0031] The circuitry may be configured to perform a smoothing operation on the separated segments of the source.

[0032] The circuitry may be configured to perform a beat detection process to analyze the results of the multi-channel source separation.

[0033] The time-varying parameters may be determined on a beat-by-beat, window-by-window, or frame-by-frame basis for the separated source.

[0034] The present embodiment also discloses a method that includes analyzing the results of the stereo or multi-channel source separation to determine one or more time-varying parameters, and creating a spatially dynamic audio object based on the one or more time-varying parameters.

[0035] The present embodiment also discloses a computer program comprising instructions that, when the program is executed by a computer, cause the computer to carry out the methods and processes described in the above and following embodiments.

[0036] Hereinafter, the present embodiment will be described with reference to the drawings.

[0037] The process of the embodiments described in detail below starts with a (music) source separation approach (see FIG. 1 and corresponding description), e.g. using stereo content. After source separation, the energies of the left and right channels are compared to each other, in particular using a Side / Mid ratio calculation (see FIG. 5a, c, d and corresponding description). This ratio is then used to derive a time-varying index (see Fig. 8a,b,c,d and corresponding description) that points to an array of (predefined) positions that are finally used with an audio object-based rendering method (monopole synthesis in the specific embodiment of Fig. 9). To prevent unnatural, unpleasant or too fast positional variations (like spatial jumps in time), this ratio may be pre-divided (see Fig. 6a, b, c, d and corresponding description) and averaged in time clusters according to the music beat (see Fig. 7a, b and corresponding description), but this step is also optional and can be replaced by any other time smoothing method.

[0038] (Audio upmixing / remixing with Blind Source Separation (BSS)) FIG. 1 shows a schematic diagram of a general approach for audio upmixing / remixing by blind source separation (BSS), such as music source separation (MSS). Source separation (also called "demixing") involves decomposing a source audio signal 1 that contains audio from multiple audio sources, source 1, source 2, ..., source K (e.g., instruments, voices, etc.) into source estimates 2a-2d for each channel i, where K is an integer indicating the number of audio sources. In the present embodiment, the source audio signal 1 is a stereo signal with two channels i = 1 and i = 2. Since the separation of the audio source signals may be incomplete, for example due to mixing of the audio sources, a residual signal 3 (r(n)) is generated in addition to the separated audio source signals 2a-2d. This residual signal may for example represent the difference between the input audio content and the sum of all separated audio source signals. The audio signal emitted by each audio source is represented in the input audio content 1 by its respective recorded sound wave. For input audio content having two or more audio channels, such as stereo or surround sound input audio content, spatial information for the audio source is also typically included in or represented by the input audio content (e.g., the proportion of the audio source signal contained in the different audio channels). The separation of the input audio content 1 into separated audio source signals 2a-2d and a residual signal 3 is performed based on blind source separation or any other technique capable of separating separate audio sources.

[0039] In a second step, the separated signals 2a-2d and possibly the residual signal 3 are remixed and rendered into a new loudspeaker signal 4, here a signal comprising five channels 4a-4e, i.e. a 5.0 channel system. Based on the separated audio source signals and the residual signal, an output audio content is generated by mixing the separated audio source signals and the residual signal based on the spatial information. The output audio content is exemplarily shown in FIG. 1 and indicated by reference number 4.

[0040] In the following, the number of audio channels in the input audio content is M. in The number of audio channels in the output audio content is called M out The input audio content 1 in the example of FIG. 1 has two channels i=1 and i=2, and the output audio content 4 in the example of FIG. 1 has five channels 4a to 4e, so that M in = 2 and M out = 5. The technique in Figure 1 is generally called remixing, and in particular, in < M out In the example of Figure 1, the number of audio channels in the input audio content 1 is M in = 2 is the number of audio channels in the output audio content 4, M out = 5, so the result is an upmixing of the stereo input audio content 1 to the 5.0 surround sound output audio content 4.

[0041] In audio source separation, an input signal containing multiple sources (e.g. instruments, voices, etc.) is decomposed to separate them. Audio source separation can be unsupervised (called "blind source separation") or partially supervised. "Blind" means that blind source separation does not necessarily carry information about the original sources. For example, it is not necessary to know how many audio sources the original signal contains, or which sound information of the input signal belongs to which original audio source. The goal of blind source separation is to decompose the original signals as they are separated, without knowing the previous separation. The blind source separation unit can use any blind source separation technique known to those skilled in the art. In (blind) audio source separation, one can search for minimally correlated, i.e. maximally independent, audio source signals in a probabilistic or information-theoretic sense, or based on non-negative matrix factorization structure constraints on the audio source signals. Methods for performing (blind) source separation are known to those skilled in the art and are based, for example, on principal component analysis, singular value decomposition (independent component analysis, non-negative matrix factorization, artificial neural networks, etc.).

[0042] Some embodiments use blind source separation to generate the separated audio source signals, but this disclosure is not limited to embodiments in which no further information is used for the separation of the audio source signals, although in some embodiments further information is used for the generation of the separated audio source signals. Such further information is, for example, information regarding the mixing process, information regarding the types of audio sources contained in the input audio content, information regarding the spatial positions of the audio sources contained in the input audio content, etc.

[0043] The input audio signal can be any type of audio signal: it may be in the form of an analog signal, a digital signal, it may originate from a hard disk, a voice recorder, a compact disc, a digital video disc, etc., it may be a data file, such as a wave file, an mp3 file, etc., and this disclosure is not limited to any particular format of the input audio content. The input audio content may be, for example, a stereo audio signal having a first channel input audio signal and a second channel input audio signal, and the present disclosure is not limited to input audio content having two audio channels. The input audio signal may be a multi-channel content signal. For example, in other embodiments, the input audio content may include any number of channels, such as a remix of a 5.1 audio signal, etc. The input signal may include one or more source signals. In particular, the input signal may include several audio sources, which may be any entity that generates sound waves, such as musical instruments, voices, vocals, or artificially generated sounds, such as origins forming synthesizers.

[0044] The input audio content may represent or include mixed audio sources, meaning that audio information is not available separately for all audio sources of the input audio content, but rather the audio information of different audio sources, e.g., at least partially overlapping or mixed.

[0045] The separations produced by blind source separation from the input signal may include, for example, vocal separation, bass separation, drum separation and another separation, where vocal separation includes all sounds belonging to the human voice part, bass separation includes all sounds below a predefined threshold frequency, drum separation includes all sounds belonging to the drums of a song / tune and other separations may include all remaining sounds. Separating sources with a Music Source Separation (MSS) system can result in artifacts such as interference, crosstalk and noise.

[0046] (Time-dependent spatial upmixing with dynamic acoustic objects) According to an embodiment described in detail below, the side-mid ratio parameters obtained from the separated sources are used to modify parameters of audio objects in a virtual sound system used to render the separated sources. In particular, the diffusion in monopole synthesis (i.e. the position of the monopoles used to render separated sources) is affected, which creates a more enveloping sound field.

[0047] Figure 2 shows a schematic of the process of automatic time-dependent spatial side mixing of separated sources, where the placement of the monopoles is based on the calculated side-mid ratio. A stereo file 1 containing multiple sources (see Sources 1, 2,,,K in Figure 1) has two channels (Min=2), namely the left and right channels, input into Source Separation 2 (see Figure 1 above). The Source Separation 2 process breaks down the stereo file 1 into separations, namely a "bass" separation 2a, a "drums" separation 2b, an "other" separation 2c, and a "vocals" separation 2d. The "bass", "drums" and "vocals" separations 2a, 2b and 2d reflect the respective "instruments" of the mix contained in the stereo file 1, while the "other" separation 2c reflects the residuals. Each of separations 2a, 2b, 2c, and 2d is again a stereo file output by the processing of source separation 2.

[0048] The "Bass" separation 2a is processed using a Side-Mid Ratio Calculation 5 to determine the Side-Mid ratio of the bass separation. The Side-Mid Ratio Calculation 5 process compares the energy of the right channel and the energy of the left channel of a stereo file representing the bass separation to determine the Side-Mid ratio (described in more detail with respect to Figures 5a, 5b below). A position mapping 6a is performed based on the calculated side-mid ratio of the bass separation to derive the position of the monopole 7a that is used to render the bass separation 2a in an audio rendering system. The "drums" separation 2b is processed using a side-mid ratio calculation 5b to determine the side-mid ratio of the drum separation. A position mapping 6b is performed based on the calculated side-mid ratio to derive the position of a monopole 7b that is used to render the drum separation 2b in an audio rendering system. The "Other" separation 2c is processed using a Side-Mid Ratio Calculation 5c to determine a Side-Mid Ratio for the "Other" separation. A Position Mapping 6c is performed based on the calculated Side-Mid Ratio of the Other separation to derive a position of a Monopole 7c that is used to render the Other separation 2c in an audio rendering system. The "vocal" separation 2d is processed using a side-mid ratio calculation 5d to determine the side-mid ratio of the vocal separation. A position mapping 6d is performed based on the calculated side-mid ratio of the vocal separation to derive the position of a monopole 7d that is used to render the vocal separation 2d in the audio rendering system.

[0049] In the embodiment described above, the process of source separation breaks down a stereo file into the separations "Bass", "Drums", "Other", and "Vocals". These types of separations are provided for illustrative purposes only, but can be replaced with musical instrument types as trained on the DNN.

[0050] In the above embodiment, the audio upmixing is performed on a stereo file containing two channels. However, the embodiment is not limited to a stereo file. The input audio content may be a multi-channel content, such as a 5.0 audio file, a 5.1 audio file, etc.

[0051] FIG. 3 shows a detailed exemplary embodiment of the process of spatial upmixing of separated sources as described in FIG. 2 above. A process of Beat Detection 8 is performed on the separated sources 2a-2d (e.g. Bass, Drums, Other or Vocal Separation), or on the original stereo file (Stereo File 1 in Figure 2), to split the audio signal on a beat-by-beat basis. The separated sources are processed using a Side-Mid Ratio Calculation 5 to obtain a Side-Mid Ratio per beat. An embodiment of this process of calculating the side-mid ratio 5 is described in more detail with respect to Figures 5a and 5b and Equation 1 below. A segmentation 9 process is performed based on the side-mid ratio to obtain segments of separated sources. For example, the segmentation 9 process may include performing a clustering of the side-mid ratios per beat, as described in more detail with respect to Figures 6a-6c. For each segment, a smoothing 10 is performed on the side-mid ratios to obtain a per-segment side-mid ratio. A position mapping 6 is performed on the per-segment side-mid ratios in order to derive the position of the final monopole 7, i.e., to map the per-segment side-mid ratio to one of multiple possible positions where the final monopole 7 used to render the separated sources 2a-2d will be placed.

[0052] It will be appreciated that the monopole is just one example of an audio object that may be arranged according to the principles of processing in the example shown in Figure 3. Similarly, other audio objects may be arranged according to the principles of sample processing.

[0053] Furthermore, it is understood that this is only one example of a possible implementation, but each step can be replaced by other analysis methods, and audio object positioning can also be dynamically calculated, for example, dependent on genre or based on a combination of different indexes. For example, you can use the Dry / Wet or Primary / Ambient indicators instead of the Side-Mid ratio, or modify the parameters of your Audio Objects to spread with monopole synthesis in combination with the Side-Mid ratio to create a more enveloping sound field.

[0054] (Beat detection) The process of beat detection is performed on the original stereo signal (embodiment of FIG. 4a) or on a separate source (embodiment of FIG. 4b) and splits the audio signal in small sections (time windows).

[0055] Figure 4a illustrates in more detail an embodiment of the beat detection process performed in the course of spatial upmixing of separated sources as described above in Figure 3. In this embodiment, beat detection is performed on the original stereo signal (stereo file 1 in Figure 2) in order to split the stereo signal by beats.

[0056] In this embodiment of Fig. 4a, a process of beat detection 8 is performed on the original stereo signal in order to split the audio signal in small intervals (time windows). Beat detection is a windowing process that is particularly suitable for audio signals representing musical content.

[0057] Beat detection divides the audio signal of the original stereo signal (stereo file 1 in Figure 2) into time windows of fixed length. In certain genres of music, the tempo of the music (usually in beats per minute, bpm) is fairly constant so that the length of the beats is effectively fixed. However, tempo changes may occur such that the length of the windows defined by the beats changes as the song progresses from one section to the next. Any process for beat detection known to one skilled in the art can be used to implement beat detection process 8 of FIG. 4, such as, for example, the method for bpm determination disclosed in EP 1 377 959, the beat detection circuit disclosed in U.S. Pat. No. 2,686,294, or the system for calculating musical tempo as disclosed in U.S. Pat. No. 8,952,233. The process of beat detection typically results in a series of time markers, each time marker indicating the start of a respective beat. These time markers divide the audio signal in small sections (time windows) that can be used as subdivisions of the audio signal each time forming the further processing of the audio signal (e.g. determining audio characteristics such as the side / mid ratio as described with respect to Figures 5a-5c below).

[0058] FIG. 4b illustrates in schematic form in more detail an alternative embodiment of the beat detection process performed in the process of spatial upmixing of separated sources described in FIG. 3 above. In this embodiment, beat detection is performed on the separated sources 2a-2d to divide the separated source signals on a beat-by-beat basis, thus obtaining separated sources on a beat-by-beat basis.

[0059] A beat detection process is performed on the separated sources 2a-2d, as described above with reference to Fig. 4a, to divide the separated source signals beat by beat, thus obtaining separated sources per beat. As described above, beat detection divides the audio signals of the separated sources 2a-2d into time windows of specific length. In certain genres of music, the tempo of the music (usually in beats per minute, bpm) is fairly constant, such that the length of the beat is substantially fixed.

[0060] Beat detection is a windowing process that is particularly suited to audio signals representing musical content. Instead of beat detection, windowing (or framing) may be performed based on a defined constant window size and based on a predefined "hopping distance" (in samples). The window size can be chosen arbitrarily (e.g., 128 samples per window, 512 samples per window, etc.). The hopping distance can be chosen, for example, as equal to the window length, or an overlapping window / frame can be chosen.

[0061] In yet other embodiments, no beat detection or windowing is applied, but for example the side-mid ratio is processed on a sample by sample basis (corresponding to a window size of one sample).

[0062] (Side mid processing) FIG. 5a illustrates generally an embodiment of a side-mid ratio calculation performed in the process of spatial upmixing of split sources described in FIG. 3 above. A mid / side processing 5a (also called M / S processing) is performed on the separated sources 2a-2d to obtain mid and side signals mid and side of the separated sources 2a-2d, which are related to each other by determining a ratio rat of the energies of the mid and side signals for each beat of the separated sources 2a-2d.

[0063] The side signal side and the mid signal mid are calculated using Equation 1.

number

[0064] For each beat of the separated sources 2a-2d, the mid and side signals are related to each other by determining the ratio rat of the energies of the mid and side signals using Equation 2.

number

[0065] The energy of a signal is related to the amplitude of the signal and may be taken, for example, as the short-time energy as follows:

number

[0066] In this embodiment, the side-mid ratio is calculated on a beat-by-beat basis, thus leading to a smoother value (compared to a fixed window length). The beats are calculated based on the input stereo file, as explained with respect to FIG. 4 above. In the above embodiment, the energy of the side signal is calculated as 2 and mid signal energy mid 2 is used to determine the time-varying parameter rate to create spatially dynamic audio objects based on the time-varying parameters. However, it is not necessary to use this energy to calculate the time-varying parameters. In alternative embodiments, for example, the ratio of the amplitude differences |LR| / |L+R| may be used to determine the time-dependent factors.

[0067] Furthermore, in the above embodiment, a normalization factor of 0.5 is foreseen, however this normalization factor is provided only for regulatory reasons, it is not required as it does not affect the ratio and therefore can be ignored.

[0068] FIG. 5(b) illustrates the calculation results of the side-mid ratio explained in FIG. 5(a). In this example, the side-mid ratio obtained for the "other" separation 2c is displayed. The side-mid ratio of the other separation 2c is represented by curve 11 together with the signal 12 of the other separation 2c.

[0069] Silent portions within the separated sources may contain substantially unnoticeable artifacts, and therefore, as shown below with respect to the embodiment of FIG. 5c, in order to minimize such artifacts, the side-mid ratio may be automatically set to zero in silent portions of the separated sources 2a-2d.

[0070] The silent parts of the separated sources 2a to 2d are, for example, the energy L 2 and R 2 by comparing each of them with a predefined threshold level (or the total energy of both stereo channels L 2 +R 2 by comparing the threshold value to a predefined threshold level.

[0071] FIG. 5c illustrates generally one embodiment of a silence suppression process as it may be performed during the separated source side-mid ratio calculation process described in FIG. 5a above. The total energy L of the left stereo channel L and the right stereo channel R 2 +R 2 The silence detection 5d is carried out by determining the total detected energy L in both stereo channels. 2 +R 2 The total energy L 2 +R 2 is compared to a predefined threshold level. Total Energy L 2 +R 2 In the case where is less than a predefined threshold level thr (indicating the presence of silence on the separated sources 2a-2d), the side-mid ratio rat is automatically set to zero (rat = 0). 2 +R 2 If exceeds a predefined threshold level thr, the side-mid ratio rat is left unchanged (rat = rat).

[0072] In the above embodiment, the derivation of the mid / side ratio is described here as an example of a time-varying parameter, in other embodiments, the time-varying parameter may be, for example, signal level / loudness between the separated channels, spectral balance, primary / ambience, dry / wet, percussion / harmonic content, or other parameters that can be derived from a music information retrieval approach, and the present disclosure is not limited in this respect.

[0073] (Segmentation (cluster detection)) To prevent unnatural, unpleasant, or too fast positional variations, such as fast spatial jumps in time, the side-mid ratio can be segmented by beat and smoothed using a temporal smoothing method. For example, one embodiment of an exemplary segmentation process in which the side-mid ratio is segmented, as detailed in Figures 6a-6c below. In this way, the similarity of content derived from source separation can be analyzed.

[0074] FIG. 6a illustrates generally one embodiment of the segmentation process performed in the process of spatial upmixing of separated sources described in FIG. 3 above. The segmentation 9 process is performed based on the side-mid ratios to obtain segments of the separated sources. For example, the segmentation 9 process includes performing a clustering of the side-mid ratios per beat (or per window). That is, the segmentation 9 process is performed with the side-mid ratios per beat (or per window) clustered into segments to obtain the side-mid ratios per beat (or per window). As explained, the goal of segmentation 9 is to find homogeneous segments in the separated source and to divide the separated source into homogeneous segments. Each segment identified as homogeneous by the side-mid ratio is expected to be related to a specific section of music with certain common characteristics. For example, the start and end of a background composition (or guitar solo, etc.) can each mark the end of a specific section of the musical score. By identifying characteristic parts (herein referred to as "segments") of the separated sources, changes in the audio rendering due to repositioning the virtual monopole used to render the separated sources can be limited to the transition from one part to the next. In this way, automatic time-dependent spatial upmixing can be based on the results of a similarity analysis of the multi-channel content. It should be noted that in the above embodiment, the segmentation is done based on the side-mid ratio (or other time-varying parameters) which provides different results for each separated source (instrument). However, the time markers of segmentation (detection beats) of the clustering process are common to all separated signals. The segmentation is performed beat-synchronously with the original stereo signal and downmixed to mono. Between successive beats, time-varying parameters, such as the beat-by-beat mean of the mid-side ratio, are calculated for each separated signal.

[0075] FIG. 6b shows the clustering process of the side-mid ratios per beat, which is included in the segmentation process as described above with reference to FIG. 6a. An audio source, here separated sources 2a-d, contains a quantity B of beats displayed on a time axis (x-axis). The beats B (respectively the time length of each beat) are identified by the process described with reference to Figure 4 above. According to the process described with reference to Figure 5a above, a side-mid ratio rat(i) is obtained for each beat i in the series of beats B obtained by the beat detection process of Figure 4.

[0076] In Fig. 6b we show the side-mid ratio rat per beat, which is displayed on the Y-axis. Each side-mid ratio rat(i) for each beat i in the set of beats B is represented as a dot. In Fig. 6b, the dot representing the side-mid ratio rat(i) for beat B is mapped to the y-axis. As can be seen in Fig. 6b, the side-mid ratio rat(i) shows clustering in two clusters C1 and C2. That is, beats with similar side-mid ratio values ​​can be associated in either cluster C1 or cluster C2. Cluster C1 identifies a first segment S1 of the separated source. Cluster C2 identifies a second segment S2 of the separated source.

[0077] As mentioned above, the goal of audio clustering is to identify and group all beats with identical beat-by-beat side-mid ratios. Audio beats with different beat-by-beat side-mid ratio classifications are clustered into different segments. Any clustering algorithm known to those skilled in the art, e.g., Kmeans algorithm, agglomerative clustering (described in https: / / en.wikipedia.org / wiki / Hierarchical_clustering), etc., can be used to identify side-mid ratio clusters indicative of segments of an audio signal.

[0078] FIG. 6c provides one embodiment of a clustering process that may be applied to segment the separated sources. Initially, each beat is considered as one cluster. The following approach is applied iteratively to those clusters. In 61, the algorithm calculates a distance matrix, here the Bayesian Information Criterion BIC, for all clusters. The two clusters that are closer are considered to be merged into a new cluster. For this, in 62, it is determined if BIC<0. If it is determined in 62 that BIC<0, the two clusters are merged with C={C1, C2}. If in 62 it is determined that the BIC ≥ 0, the two clusters are not merged, otherwise they are not merged. In this way, the clusters are linked together until the distance exceeds a predefined value, at which point the clustering is terminated.

[0079] The distance measure for comparing two clusters using BIC can be written as a model selection criterion where one model is represented by two separate clusters C1 and C2, and the other model is the combined cluster C={C1, C2}. The formula for BIC can be specified as follows:

number

number

[0080] FIG. 6d shows a separated source segmented as described under reference to FIG. 6a above. The first segment S1 identified by the segmentation process of FIG. 6a begins at time instance t1 and ends at time instance t1. The second subsequent segment S2 begins at time instance t1 and ends at time instance t2. Similarly, the Nth segment begins at time instance t N-1 Starting at time instance t N It ends at time. Time instances t1,,,t2,,t3,,t4,,t5,,t6,,t7,,t8,,t9,,t10,,t11,,t12,,t13,,t14,,t15,,t16,,t17,,t18,,t19,,t20,,t21,,t22,,t23,,t24,,t25,,t26,,t2 N represents the segment boundary.

[0081] Figure 7a shows diagrammatically the time smoothing process, in which the side-mid ratios of the separated sources are averaged over segments of the separated sources. In FIG. 7a, the segment S resulting from the segmentation process 9 described above with reference to FIG. 6a is shown. n Based on this, a smoothing process 10 is performed on the beat-by-beat side-mid ratio rat(i) of the separated source, and the segment S n The smoothed side-mid ratio rat(------)(n) is obtained for each

[0082] By the segmentation process described in Fig. 6(a), the series of beats B obtained from beat detection is divided into several segments S. n Each segment S n Segment S contains multiple beats obtained by the beat detection process of FIG. n For each beat of a segment S, the side-mid ratio rat(i) is obtained. n For one segment S, the smoothed side-mid ratio rat(n) is n The side-mid ratio rat(i) obtained for all beats i in the

number

[0083] FIG. 7b shows an example of the smoothing process. The first segment S1 identified by the segmentation process of Fig. 6a is associated with a smoothed side-mid ratio rat(l). The second segment S2 is associated with a smoothed side-mid ratio rat(2). Similarly, the Nth segment is associated with a smoothed side-mid ratio rat(N). Time instances t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t20, t21, t22, t23, t24, t25, t26, t27, t28, t29, t21, t22, t23, t24, t25, t26, t27, t28, t29, t30, t31, t32, t33, t34, t35, t36, t37, t38, t39, t40, t41, t42, t43, t44, t45, t46, t47, t48, t49, t48, t49, t49, t41, t42, t43, t44, t45, t46, t47, t48, t49, t49, t50, t51, t52, t53, t54, t55, t56, t57, t58, t59, t60, t61, t62, t63, t64, t65, t66, t67, t6 Nare solid black vertical lines indicating segment boundaries. The smoothed side-mid ratios rat(n) are shown in Fig. 7b by the respective horizontal solid black lines.

[0084] According to an embodiment described in more detail herein, the position of the final monopole is determined based on the side-mid ratio, and in particular based on a smoothed side-mid ratio that attributes the side-mid ratios for all segments of the audio signal.

[0085] (Location Mapping) FIG. 8a illustrates an exemplary embodiment of a position mapping that determines the position of the monopole used to render the separated sources. This embodiment of FIG. 8a in particular uses a positioning index depending on the original balance between the separated channels of the music source separation process (e.g. the side-mid ratio, or the smoothed side-mid ratio as described in more detail above), but can be extended to other separation techniques.

[0086] FIG. 8a exemplarily illustrates how the location mapping determines the location of the monopole based on the side-mid ratio determined from the separated sources. On the left side of Fig. 8a, several segments S of isolated sources are shown, as identified by the segmentation process described in Figs. 6a-6d and the smoothing process described in Figs. 7a-7b. n The smoothed side-mid ratio rat(n) for On the right side of Figure 8a, possible positions of the two monopoles used to render the left and right stereo channels of the separated sources are shown. The possible positions m = 1,,,M of the two monopoles are represented by small circles. In the example of Fig. 8a, 17 possible positions (M = 17) are predicted for the left stereo channel as positions m = 1,,,M, which are arranged in a semicircle on the left side of the listener. 17 additional possible positions for the right stereo channel are predicted as positions m = 1,,,M, which are arranged in a semicircle on the right side of the listener. The filled circles (m = 1 and M = M) define the positions of the four (physical) speakers SP1, SP2, SP3, SP4 used to render the (virtual) monopoles. The first speaker SP1 is located at the front left, the second speaker SP2 at the front right, the third speaker SP3 at the rear left, and the fourth speaker SP4 at the rear left. The circles with dotted lines or dot patterns indicate the possible positions of the virtual speakers rendered by the speakers SP1, SP2, SP3, and SP4. As shown by the dashed lines, the smoothed side-mid ratio rat(l) of segment S1 corresponds to the respective monopole positions P L and P R is mapped by a mapping process to

[0087] It should be noted that it is difficult to render a virtual monopole directly at or very close to the location of a physical speaker, therefore possible monopole locations close to any of the speakers SP1, SP2, SP3, SP4 are marked with a dot pattern, and all other possible locations are marked with a dashed pattern.

[0088] In the embodiment of FIG. 8a described above, the number of possible positions is 17 per semicircle, but the number of possible positions may be any other number, such as 27 per semicircle.

[0089] Further, in the embodiment in the case of Fig. 8b, four physical speakers are used to render the monopole, however, in alternative embodiments, speaker systems with a different number of speakers can be used to render the virtual monopole, for example a 5.1 speaker system, a sound bar, binaural headphones, a speaker wall with many speakers, etc.

[0090] Fig. 8b shows a further exemplary embodiment of a position mapping that determines the location of the monopole used to render the separated source. Fig. 8(b) is similar to Fig. 8(a), except that the dashed lines show how the mapping of the smoothed side-mid ratio rat(3) of segment S3 corresponds to the unique monopole position P of the left and right stereo channels of the separated source. L and P R This indicates that it is. According to the embodiment described herein with reference to Figures 8a and 8b, the lower the smoothed side-mid ratio rat(n), the closer to the positions of the two front (physical) speakers SP1 and SP2 will be the monopole positions selected for the left and right stereo channels of the separated sources. The higher the side-mid ratio rat(n), the higher the smoothed side-mid ratio rat(n) and the closer to the positions of the two rear (physical) speakers SP3 and P4 are the selected monopole positions of the left and right stereo channels of the separated sources.

[0091] FIG. 8c shows the position mapping as performed for the maximum and minimum side-mid ratios of the separated sources, respectively. On the left side of Figure 8c, max indicates the maximum side-mid ratio determined from the separated sources, indicated by the dashed line, and the side-mid ratio rat = 0, indicated by the double dashed line. On the right side of Figure 8b are shown possible positions of two monopoles that could be used to render left and right stereo channels of the separated sources as described in Figures 8a and 8b above. As indicated by the dashed line, the maximum side-mid ratio max is mapped by the mapping process to a monopole position m=M which corresponds to the positions of the two back speakers SP3 and SP4. As shown by the double dashed lines, the side-mid ratio rat=0 is mapped to a monopole position m=1 of the two front speakers SP1 and SP2.

[0092] The mapping between the smoothed side-mid ratio rat(n) and its position can be any mapping of ratios to a predefined number of discrete positions, for example as shown in Figures 8a and 8b.

[0093] For example, the mapping process may be performed as follows.

number

[0094] Figures 8a, b, and c show how the location of a particular isolated source moves on a portion of a circle depending on the side-mid ratio. If the Side-Mid ratio is low (see Figure 8b), the left and right channels are very similar (in the extreme case, see Figure 8c, mono). In this case, the perceived width of the stereo image is narrower. The sources are therefore kept in their original positions in the spatial mix, like in a traditional 5.1 mix for the left and right front channels. If the Side-Mid ratio is high (see Figure 8a), the left and right channels are very different (in the extreme case, the content of each channel is completely different). The perceived width of the stereo image is wider. Thus, sources are shifted to more extreme positions in the spatial mix - closer to the left and right back channels in a traditional 5.1 mix, for example. By directly linking the characteristics of the Side-Mid ratio and the perceived stereo width, the system is able to preserve the mixing aesthetic of the original stereo content during relocation.

[0095] Figure 9 visualizes how the position mapping, which determines the position of the monopole based on the side-mid ratio determined from the separated source, relates to the specified positions of the two monopoles used to render the left and right stereo channels of the separated source. For each monopole position index m(n), a pair of position coordinates (x,y) is given in the left stereo channel. L are stored in a table in advance, and a pair of position coordinates (x, y) are R is pre-stored in a table. On the left side of Figure 9, it is shown that the position mapping, as described in reference to Figures 8a, b, c, selected position index m=9 as the location of the two monopoles used to render the left and right stereo channels of the separated sources, respectively. On the right side of Figure 9, this particular monopole position index m=9 is used to render the left and right stereo channels of the separated sources by a virtual acoustic rendering system (or a 3D acoustic rendering system), with the monopole position coordinates (x, y) respectively. L and the monopole position coordinates (x, y) R This visualizes how the For example, monopole synthesis techniques, binaural headphone techniques, etc., as described in more detail with respect to FIG. 10 below.

[0096] In the above mapping process, the side-mid ratio rat(n) (or alternatively rat(i)) is mapped to a discrete number of possible positions. Alternatively, the position mapping can be done in a non-discrete manner, e.g., by mapping the side-mid ratio rat(n) (or alternatively rat(i)) to the respective position coordinates (x,y). L and (x,y) R Alternatively, an algorithmic process may be used that maps directly to

[0097] Furthermore, in the above embodiment, the position mapping is described to be performed separately for the left and right stereo channels. However, in another embodiment, such position mapping may be performed only for one of the stereo channels (e.g., the left channel), and the monopole position of the other stereo channel (e.g., the right channel) may be obtained by mirroring the position of the mapped stereo channel (e.g., the left channel).

[0098] In the above embodiment, the determination of the monopole position for rendering a stereo signal of the separated sources is based on the side-mid ratio parameters obtained from the separated sources, however, in alternative embodiments, other parameters of the separated sources may be selected for determining the monopole position for rendering a stereo signal. For example, you can use the Dry / Wet and Primary / Ambient indicators to modify parameters of an audio object that are spread with monopole synthesis, creating a more enveloping sound field, or you can use a combination of such parameters to modify parameters of an audio object.

[0099] (Monopole synthesis) FIG. 10 provides an embodiment of 3D audio rendering based on a digitized monopole synthesis algorithm. The theoretical background of this technology is described in detail in U.S. Patent Publication No. 2016 / 0037282, which is incorporated herein by reference.

[0100] The technique implemented in this embodiment of US2016 / 0037282 is conceptually similar to wave field synthesis, where a limited number of acoustic enclosures are used to generate a defined sound field. However, the basic basis of the generation principle of this embodiment is specific, as the synthesis is based on least squares rather than attempting to model the sound field exactly.

[0101] The target sound field is modeled as at least one target monopole located at a defined target location. In one embodiment, the target sound field is modeled as a single target monopole. In another embodiment, the target sound field is modeled as multiple target monopoles, each located at a defined target location. For example, each target monopole may represent a noise cancellation source in a set of multiple noise cancellation sources located at a particular location in space. The location of the target monopole may be moving, for example, the target monopole may adapt to the movement of a noise source to be attenuated. If multiple target monopoles are used to represent the target sound field, the method of synthesizing the sound of the target monopole based on a set of defined synthetic monopoles as described below may be applied independently to each target monopole, and the resulting synthetic monopole contributions for each target monopole may be summed to reconstruct the target sound field.

[0102] The source signal x(n) is z -np and is fed into the delay section labeled by pwhere p = 1,,,N are the indices of the respective synthesis monopoles used to synthesize the target monopole signals. The delay and amplification unit according to the present embodiment applies equation (117) of the referenced US Patent Publication No. 2016 / 0037282 to synthesize the resulting signal y p (n)=s p (n) can be calculated. The result signal s p (n) is power amplified and output to speaker S p will be sent.

[0103] In this embodiment, the synthesis is thus performed in the form of delayed and amplified components of the source signal X.

[0104] According to this embodiment, the delay n for the composite monopole index p is p is the target monopole r0 and the generator r p The Euclidean distance r between p0 = | r p -r0| corresponds to the sound propagation time.

[0105] Moreover, according to this embodiment, the amplification factor a p = ρc / R po is the distance r = R p0 is inversely proportional to.

[0106] An alternative implementation of the system can use a modified amplification factor according to equation (118) of the referenced U.S. Patent Publication No. 2016 / 0037282.

[0107] (Example of spatial upmixing of stereo content) FIG. 11 illustrates a schematic of one embodiment of a process of time-dependent spatial upmixing of separated sources. The stereo content (see 1 in FIG. 2) is processed using a source separation process (e.g. BSS), ambience analysis, music information retrieval, etc. to obtain separated channels and / or derived content. A similarity analysis of this derived content is performed to obtain measures in time (e.g. side-mid ratio) to determine segments with similar characteristics (e.g. as described with respect to FIGS. 6a-6d above). A time-varying similarity index (e.g., m=1,,,M in Fig. 8a,b,c) is obtained based on the similarity of the derived source separated contents in time, and then the time-varying index is used to derive a spatial index for position mapping. The time-varying parameters may be signal level / loudness, spectral balance, primary / ambience, dry / wet, percussive / harmonic content, etc. between the separated channels. A spatial index is a vector / array of positioning indices, which refers to a vector / array of positions and the calculation of rendering parameters. An audio object rendering system, which is a multi-channel playback system such as binaural headphones, soundbars, etc., renders an audio signal to speakers.

[0108] FIG. 12 shows a flow diagram visualizing an exemplary method for performing time-dependent spatial upmixing of separated sound sources: bass 2a, drums 2b, others 2c and vocals 2d. At 90, Source Separation 2 (see Figures 2 and 3) receives an input audio signal (see Stereo File 1 in Figure 2). At 91, Source Separation 2 is performed on the input audio signal to obtain separated sources 2a-2d (see Figure 2). At 92, a Side-Mid Ratio calculation is performed for each separated source to obtain a Side-Mid Ratio (see Figures 5a, 5b). At 93, segmentation 9 is performed on the side-mid ratios to obtain segments (see Figs. 6a-6b). At 94, smoothing 9 is performed on the side-mid ratios based on the segments to obtain smoothed side-mid ratios (see Figs. 7a, 7b). At 95, location mapping is performed based on the smoothed side-mid ratios (see Figs. 8a-8c). During location mapping, spatial positioning parameters are derived, which depend on the time-varying parameters obtained during source separation. A monopole pair from the plurality of final monopoles 7 is determined for each of the separated sources 2a-2d (see FIG. 2) based on the position mapping 6 (see FIG. 3, FIGS. 8a-8c and FIG. 9). At 96, an audio signal is rendered based on the position mapping 6.

[0109] (Real-time processing) The above-mentioned process of upmixing / remixing, e.g. by dynamically determining parameters of audio objects to be rendered by a 3D audio rendering process, may be performed as a post-processing step on the audio source files, respectively, on the separated sources obtained from the audio source files by the source separation process. In such a post-processing scenario, the entire audio file can be processed. Thus, the side-mid ratio may be determined for every beat / window / frame of the separated source as described in the figures. The segmentation process described in figures 5a-5c and 6a-6d can be applied to the entire audio file.

[0110] However, the above process may also be implemented as a real-time system. For example, upmixing / remixing of a stereo file may be performed in real time on the received audio stream. If the audio signal is processed in real time, it is not appropriate to determine the segments of the audio stream only after receiving the complete audio file (such as a song). However, changes in audio characteristics or segment boundaries need to be detected "on the fly" during the streaming process so that while an audio file is being streamed, the rendering parameters of an audio object can be changed immediately upon detection of the changes.

[0111] For example, the smoothing may be performed by continuously determining a parameter such as a side-mid ratio and continuously determining the standard deviation σ of this parameter. The current change in the parameter may be related to the standard deviation σ. If the current change in the parameter is large relative to the standard deviation, the system may determine that there is a large change in the audio characteristics. Without limiting the disclosure in that regard, a significant change (jump) in an audio signal may be detected when the difference between subsequent parameters of the signal (e.g., the side-mid ratio per beat) is higher than a threshold, for example when the difference is equal to 2σ.

[0112] Such significant changes in audio characteristics detected on the fly can be treated like segment boundaries as described in the above embodiments, i.e., they may trigger a reconfiguration of parameters of the 3D audio rendering process, e.g., a relocation of monopole positions used in monopole synthesis.

[0113] (implementation) FIG. 13 illustrates diagrammatically one embodiment of an electronic device capable of implementing the process of automatic time-dependent spatial upmixing of separated sources, ie separation, as described above. The electronic device 700 comprises a CPU 701 as a processor. The electronic device 700 further includes a microphone array 711 and a loudspeaker array 710 connected to the processor 701. The processor 701 may implement, for example, source separation 2, side-mid ratio calculation 5 and position mapping 6, which realize in more detail the processes described with respect to Figures 2, 3, 8a-8c and 9. The loudspeaker array 710 consists of one or more loudspeakers distributed in a given space and configured to render 3D audio. The electronic device 700 further includes an audio interface 706 connected to the processor 701. The audio interface 706 serves as an input interface through which a user can input an audio signal, for example the audio interface may be a USB audio interface. Moreover, the electronic device 700 further comprises a user interface 709 connected to the processor 701. The user interface 709 serves as a man-machine interface and allows interaction between an administrator and the electronic system. For example, an administrator can use the user interface 709 to configure the system. The electronic device 701 further includes an Ethernet interface 707, a Bluetooth interface 704, and a WLAN interface 705. These units 704, 705 serve as input / output interfaces for data communication with external devices. Additional loudspeakers, microphones and video cameras, for example with Ethernet, WLAN or Bluetooth connections, may be coupled to the processor 701 via these interfaces 707, 704 and 705.

[0114] The electronic system 700 further comprises a data store 702 and a data memory 703 (here a RAM). The data memory 703 is arranged to temporarily store or cache data or computer instructions for processing by the processor 701. The data store 702 is arranged as a long-term storage for recording sensor data obtained, for example, from the microphone array 710. Data storage device 702 may also store audio data representing audio messages that a public announcement system may transmit to people moving within a given space.

[0115] It should be noted that the above descriptions are merely example configurations, and alternative configurations may be implemented using additional or other sensors, storage devices, interfaces, etc.

[0116] It should be understood that the above-described embodiments describe methods with exemplary ordering of method steps. However, the particular ordering of method steps is provided for illustrative purposes only and should not be construed as binding.

[0117] It should also be noted that the division of the electronic system of Figure 13 into units is done for illustrative purposes only, and the present disclosure is not limited to any particular division of functionality in particular units. For example, at least some of the circuitry may be realized by respectively programmed processors, field programmable gate arrays (FPGAs), dedicated circuitry, etc.

[0118] All units and entities described in this specification and recited in the accompanying claims may be implemented, for example as integrated circuit logic on a chip, unless otherwise stated, and the functions provided by such units and entities may be implemented by software, unless otherwise stated.

[0119] In so far as the embodiments of the present disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be understood that computer programs providing such software control, and the transmission, storage, or other media on which such computer programs are provided, are contemplated as aspects of the present disclosure.

[0120] The present technology can also be configured as follows. (1) An electronic device comprising circuitry configured to analyze results of stereo or multi-channel source separation to determine one or more time-varying parameters, and to create a spatially dynamic audio object based on the one or more time-varying parameters. (2) the circuitry is configured to determine, as a time-varying parameter, a parameter describing signal level loudness between the separated channels, and / or a spectral balance parameter, and / or a primary-ambience indicator, and / or a dry-wet indicator, and / or a parameter describing percussive-harmonic content. 1. An electronic device according to claim 1. (3) the circuitry is configured to determine parameters describing a balance of instruments in stereo content as time-varying parameters and to create the spatially dynamic audio object based on the balance of instruments in the stereo content. An electronic device according to (1) or (2). (4) the circuitry is configured to determine a side-mid ratio of the separated sources as a time-varying parameter and to create the spatially dynamic audio object based on the side-mid ratio. An electronic device according to any one of (1) to (3). (5) The circuitry is configured to determine spatial positioning parameters for the audio object based on the one or more time-varying parameters obtained from the stereo or multi-channel source separation. An electronic device according to any one of (1) to (4). (6) the circuitry is configured to dynamically adapt positioning parameters of the audio object. An electronic device according to any one of (1) to (5). (7) the circuitry is configured to create the spatially dynamic audio object by monopole synthesis; An electronic device according to any one of (1) to (6). (8) The circuit is configured to dynamically adapt spreading in monopole synthesis. An electronic device according to any one of (1) to (7). (9) the spatially dynamic audio object is a monopole An electronic device according to any one of (1) to (8). (10) The circuitry is configured to dynamically create a first monopole used to render a left channel of the separated source and a second monopole used to render a right channel of the separated source based on the one or more time-varying parameters. An electronic device according to any one of (1) to (9). (11) The circuit is configured to generate a time-dependent spatial upmix from the result of the multi-channel source separation, the time-dependent spatial upmix preserving the original balance of the content. An electronic device according to any one of (1) to (10). (12) The circuit is further configured to perform a segmentation process to obtain separated source segments based on the time-varying parameters. An electronic device according to any one of (1) to (11). (13) The circuitry is configured to perform cluster detection based on the time-varying parameters. An electronic device according to any one of (1) to (12). (14) The circuitry is configured to perform automatic time-dependent spatial upmixing based on results of a similarity analysis of the multi-channel content. An electronic device according to any one of (1) to (13). (15) The circuit is configured to perform a smoothing process on the separated source segments. An electronic device according to any one of (1) to (14). (16) The circuitry is configured to perform a beat detection process to analyze the results of the multi-channel source separation. An electronic device according to any one of (1) to (15). (17) The time-varying parameters are determined per beat, per window, or per frame of the separated source or original content. An electronic device according to any one of (1) to (16). (18) A method comprising the steps of analyzing results of stereo or multi-channel source separation to determine one or more time-varying parameters, and creating a spatially dynamic audio object based on the one or more time-varying parameters. (19) A computer program comprising instructions that, when executed by a computer, cause the computer to carry out the method according to (18).

Claims

1. A method for creating spatially dynamic audio objects comprising: determining a side-mid ratio of a separated source; and arranging monopoles based on the calculated side-mid ratio; The circuitry is configured to analyze a result of the stereo or multi-channel source separation to determine one or more time-varying parameters and to create the audio object based on the one or more time-varying parameters. Electronic devices.

2. The circuit is configured to determine as time-varying parameters a parameter describing the signal level loudness between the separated channels, and / or a spectral balance parameter, and / or a primary-ambience indicator, and / or a dry-wet indicator, and / or a parameter describing the percussive-harmonic content. The electronic device of claim 1 .

3. the circuitry is configured to determine spatial positioning parameters for the audio object based on the one or more time-varying parameters obtained from the stereo or multi-channel source separation result. The electronic device of claim 1 .

4. The circuitry is configured to dynamically adapt positioning parameters of the audio object. The electronic device of claim 1 .

5. The circuitry is configured to create the spatially dynamic audio object by monopole synthesis. The electronic device of claim 1 .

6. The circuit is configured to dynamically adapt spreading in monopole synthesis. The electronic device of claim 1 .

7. The spatially dynamic audio object is a monopole The electronic device of claim 1 .

8. The circuitry is configured to dynamically create a first monopole used to render a left channel of a separated source and a second monopole used to render a right channel of the separated source based on the one or more time-varying parameters. The electronic device of claim 1 .

9. The circuitry is configured to generate a time-dependent spatial upmix from the result of the multi-channel source separation, the time-dependent spatial upmix preserving the original balance of the content. The electronic device of claim 1 .

10. The circuitry is further configured to perform a segmentation process to obtain separated segments of the source based on the time-varying parameters. The electronic device of claim 1 .

11. The circuitry is configured to perform cluster detection based on the time-varying parameters. The electronic device of claim 1 .

12. The circuitry is configured to perform automatic time-dependent spatial upmixing based on results of a similarity analysis of the multi-channel content. The electronic device of claim 1 .

13. The circuitry is configured to perform a smoothing operation on the separated source segments. The electronic device of claim 1 .

14. The circuitry is configured to perform a beat detection process to analyze the results of the multi-channel source separation. The electronic device of claim 1 .

15. The time-varying parameters are determined on a beat-by-beat, window-by-window, or frame-by-frame basis of the separated source or original content. The electronic device of claim 1 .

16. A method for creating spatially dynamic audio objects by determining a side-mid ratio of separated sources and arranging monopoles based on the calculated side-mid ratio, comprising the steps of: Analyzing the result of the stereo or multi-channel source separation to determine one or more time-varying parameters, and creating the audio object based on the one or more time-varying parameters. method.

17. A computer program comprising instructions which, when executed by a computer, cause said computer to carry out the method according to claim 16.

Images