Music synthesizer with spatial metadata output

Abstract

To provide an apparatus for generating and / or processing audio signals.SOLUTION: One apparatus includes: a first stage for obtaining an audio signal; a second stage for modifying the audio signal based on one or more control signals for shaping sound represented by the audio signal; a third stage for generating spatial metadata related to the modified audio signal, based at least in part on the one or more control signals; and an output stage for outputting the modified audio signal together with the generated spatial metadata. Also described are corresponding methods, as well as corresponding programs and computer-readable storage media.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 240,383, filed September 3, 2021 (Docket No. D21053USP1), and European Application No. 21194849.2, filed September 3, 2021 (Docket No. D21053EP), which are incorporated herein by reference.

[0002] Technical Field The present disclosure relates to methods and apparatus for generating and / or processing audio signals. The present disclosure further describes techniques for synthesizing or processing sound with spatial metadata output. These techniques may be applied, for example, for music synthesizers and audio processors. [Background technology]

[0003] A synthesizer is an electronic musical instrument that generates audio signals. Generally, electronic musical instruments use electronic circuitry to generate sound, often in response to user input. In the broadest sense, this can incorporate analog instruments, DSP-based instruments running on dedicated hardware or on a computer as a "virtual" instrument, or sample-based instruments.

[0004] Existing synthesizers generate audio signals that are mono, stereo, or multi-channel. This means that when used in object-based productions (e.g., Dolby Atmos® productions), a spatialization process separate from the sound generation process is applied to represent the synthesizer audio signals as objects. Currently, this is achieved by first rendering the synthesizer's output for a specific channel configuration, importing the rendered audio into a digital audio workstation (DAW) such as Pro Tool, and then using a panner to generate the associated spatial metadata.

[0005] In general, there is a need for improved techniques for object-based generation of sound signals generated by a synthesizer and / or for object-based generation of sound signals processed by an audio processor. Summary of the Invention [Problem to be solved by the invention]

[0006] In view of the above, the present disclosure provides an apparatus for generating and / or processing an audio signal, as well as a corresponding method, a computer program, and a computer-readable storage medium, having the features of the respective independent claims. [Means for solving the problem]

[0007] According to an aspect of the present disclosure, an apparatus for generating or processing an audio signal is provided. The apparatus may include a first stage for obtaining an audio signal. The apparatus may further include a second stage for modifying the audio signal based on one or more control signals for shaping (e.g., modifying, changing) a sound represented by the audio signal. The control signals may be generated by one or more modulators that affect how the audio signal is modified. The one or more modulators may, for example, relate to or include a low frequency oscillator (LFO) and / or an envelope. The apparatus may further include a third stage for generating spatial metadata associated with the (modified) audio signal based at least in part on the one or more control signals. The spatial metadata may be metadata for instructing an external device on how to render the modified audio signal. For example, the spatial metadata may be metadata for object-based rendering. It may, for example, include an indication of a position (e.g., a Cartesian position in 3D space) and / or a size of an audio object related to (e.g., represented by) the audio signal. The apparatus may further include an output stage for outputting the modified audio signal together with the generated spatial metadata. In some implementations, two or more audio signals (i.e., a plurality of audio signals) may be processed in parallel by the apparatus. In some other implementations, two or more audio signals (i.e., a plurality of audio signals) may be processed serially. For example, in some such implementations, the processing for a given audio signal may depend on earlier audio signals and / or their metadata. As another example, in some implementations, the processing may involve modifying already existing spatial metadata of the audio signal obtained by the first stage.

[0008] The technique implemented by the proposed device configured as described above significantly widens the range of creative options available in object-based production and allows treating spatialization as an integrated part of the sound design process. Moreover, tedious editing of spatial metadata can be avoided and the creative intentions that led to specific shaping actions for the audio signal can be directly and efficiently implemented in the spatial metadata.

[0009] In some embodiments, the second stage may be adapted to apply a time-dependent modification to the audio signal, whereby the time-dependence of the modification may depend on said one or more control signals, whereby said one or more control signals may be time-dependent, whereby the time-dependence underlying the intended behavior of a sound source can be easily used to describe spatial properties of an audio object describing / representing the sound source.

[0010] In some embodiments, the second stage may include at least one of a filter, an amplifier, a low frequency oscillator, an audio delay, a driver, and a flanger to modify the audio signal.

[0011] In some embodiments, the second stage may be adapted to apply a filter to the audio signal. It is understood that the second stage may include said filter. The filter may be, for example, any of a high-pass filter, a low-pass filter, a band-pass filter, or a notch filter. A characteristic frequency of the filter may be controlled by (for example based on) said one or more control signals. Thus, the characteristic frequency may be time-dependent. The characteristic frequency of the filter may be, for example, a cut-off frequency. The one or more (time-dependent) control signals may be generated, for example, by an LFO. Thereby, the characteristic frequency may vary periodically in some implementations.

[0012] In some embodiments, the second stage may be adapted to apply an amplifier to the audio signal. It will be appreciated that the second stage may include said amplifier. The gain of the amplifier may be controlled by (e.g., based on) said one or more control signals. Thus, the gain may be time-dependent. The one or more control signals may be generated, for example, by an LFO. Thereby, the gain may vary periodically in some implementations.

[0013] In some embodiments, the second stage may be adapted to apply an envelope to the audio signal by using an amplifier. The third stage may then be adapted to generate said spatial metadata based at least in part on a shape of the envelope. For example, the spatial metadata may indicate a time-dependent position of the audio object, where the position changes according to the shape of the envelope (e.g., experiences a linear translation while the envelope exhibits a non-zero gain).

[0014] In some embodiments, obtaining the audio signal may include generating the audio signal by using one or more oscillators. It will be appreciated that the first stage may include said one or more oscillators. Such an apparatus may relate to a synthesizer, for example a music synthesizer.

[0015] In some embodiments, audio signals may be generated by the one or more oscillators based at least in part on the one or more control signals. For example, at least one of a frequency, a pulse width, and a phase of the one or more oscillators may be controlled by (e.g., based on) the one or more control signals. Control signals that affect the generation of audio signal(s) may thereby be used as a basis for generating the spatial metadata. This provides additional functionality for capturing artistic intent when generating the spatial metadata.

[0016] Alternatively, obtaining the audio signal may include receiving the audio signal. The audio signal may be received from an external source, such as a sound database. Such a device may relate to an audio processor, such as an effects audio processor.

[0017] In some embodiments, the one or more control signals may be based at least in part on user input.

[0018] In some embodiments, the output stage may be adapted to output one or more audio streams based on the modified audio signals together with the generated spatial metadata. For example, there may be one spatial metadata stream for each output audio stream. Furthermore, when two or more audio signals are processed in parallel by the device, there may be one output audio stream for each (modified) audio signal. Alternatively, at least one of the output audio streams may be generated by mixing two or more (modified) audio signals.

[0019] According to another aspect of the present disclosure, there is provided a method for generating or processing an audio signal. The method may include obtaining an audio signal. The method may further include modifying the audio signal based on one or more control signals to shape a sound represented by the audio signal. The method may further include generating spatial metadata related to the (modified) audio signal based at least in part on the one or more control signals. The method may further include outputting the modified audio signal together with the generated spatial metadata.

[0020] In some embodiments, the one or more control signals may be time dependent.

[0021] In some embodiments, the spatial metadata may be metadata for instructing an external device on how to render the modified audio signal.

[0022] In some embodiments, modifying the audio signal may include applying a time-dependent modification to the audio signal, where the time-dependence of the modification may depend on said one or more control signals.

[0023] In some embodiments, the method may further include modifying the audio signal with at least one of a filter, an amplifier, a low frequency oscillator, an audio delay, a driver, and a flanger.

[0024] In some embodiments, modifying the audio signal may include applying a filter to the audio signal, in which case a characteristic frequency of the filter may be controlled by said one or more control signals.

[0025] In some embodiments, the characteristic frequency of the filter may be the cutoff frequency.

[0026] In some embodiments, modifying the audio signal may include applying an amplifier to the audio signal, in which case the gain of the amplifier may be controlled by said one or more control signals.

[0027] In some embodiments, modifying the audio signal may include applying an envelope to the audio signal by using an amplifier. Generating the spatial metadata may then be based at least in part on a shape of the envelope.

[0028] In some embodiments, obtaining the audio signal may include generating the audio signal by using one or more oscillators.

[0029] In some embodiments, an audio signal may be generated by the one or more oscillators based at least in part on the one or more control signals.

[0030] Alternatively, obtaining the audio signal may include receiving the audio signal.

[0031] In some embodiments, the one or more control signals may be based at least in part on user input.

[0032] In some embodiments, outputting the modified audio signal may include outputting one or more audio streams based on the modified audio signal along with the generated spatial metadata.

[0033] According to another aspect, a computer program is provided that may include instructions that, when executed by a processor (e.g., a computer processor, a server processor, etc.), cause the processor to perform all of the steps of the methods described throughout this disclosure.

[0034] According to another aspect, there is provided a computer readable storage medium having stored thereon a computer program as described above.

[0035] According to yet another aspect, an apparatus is provided that includes a processor and a memory coupled to the processor. The processor may be adapted to perform all steps of the methods described throughout this disclosure. The apparatus may relate to, for example, a computer system, a server (e.g., a cloud-based server), or a system of servers (e.g., a cloud-based server).

[0036] It will be understood that apparatus features and method steps can be interchanged in many ways. In particular, details of the disclosed methods may be implemented by a corresponding apparatus and vice versa, as will be appreciated by those skilled in the art. Furthermore, it will be understood that any statements made above with respect to a method (and, e.g., steps thereof) equally apply to a corresponding apparatus (and, e.g., blocks, stages, units, etc. thereof) and vice versa. [Brief description of the drawings]

[0037] Exemplary embodiments of the invention are described below with reference to the accompanying drawings. [Figure 1] FIG. 1 is a block diagram illustrating an example of a synthesizer. [Diagram 2] FIG. 2 is a block diagram illustrating generally an audio processing chain including a synthesizer, a spatialization module, and an object-based rendering module. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a synthesizer according to an embodiment of the present disclosure. [Figure 4] 1 is a flow chart that generally illustrates an example of a method for generating and / or processing an audio signal, according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a block diagram of an apparatus for performing a method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] The drawings and the following description relate to preferred embodiments for illustrative purposes only. It should be noted from the following description that alternative embodiments of the structures and methods disclosed herein are readily recognizable as viable alternatives that may be employed without departing from the principles of what is claimed.

[0039] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that wherever practicable, similar or similar reference numerals may be used in the figures and may indicate similar or similar functionality. The figures depict embodiments of the disclosed apparatus (or methods) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0040] This disclosure relates generally to music synthesizers (and audio processors) with spatial metadata output. The generation of the metadata may be driven by control signals and / or internal modulation signals related to direct / literal user input. Thus, the control signals are typically time-dependent.

[0041] Roughly speaking, a synthesizer can be thought of as a collection of connected sound-generating / shaping and modulation elements. The sound-generating / shaping elements either generate or act directly on audio signals. For example, oscillators can generate raw tones / waveforms, filters can then shape their timbre and amplifiers can introduce level dynamics. In order to animate the generated sounds, these elements should change over time, and this is done by applying modulation elements that affect the behavior of the sound-generating / shaping elements instead of creating the sound themselves.

[0042] A simple example of a synthesizer 100 is shown in Fig. 1. The synthesizer 100 comprises a sound generation block (or stage, module, etc.) 110 and a sound modulation block (or stage, module, etc.) 120. The sound generation block 110 includes, for example, one or more oscillators (oscillator elements) 112, one or more filters (for example, voltage controlled filters (VCFs)) 114, and one or more amplifiers (for example, voltage controlled amplifiers (VCAs)) 116. The sound modulation block 120 can modulate the actual generation of the audio signal by the oscillators 112 and / or any subsequent shaping of the sound signal, for example, by the filters 114 or amplifiers 116. Here and below, it is understood that an audio signal can be an electronic representation of a sound signal. It is also understood that sound shaping corresponds to a modification / alteration of an audio signal representing a sound.

[0043] For example, the sound modulation block 120 may comprise a low frequency oscillator (LFO) 124 that can generate a cyclically subaudible (e.g., below 20 Hz) frequency output. This LFO 124 may be used to modulate the cutoff frequency of a filter (e.g., filter 114), or any other characteristic frequency of the filter. Another common element is an envelope 126 that generates a one-off modulation pattern (e.g., by amplifier 116) on the sound signal generated by the sound generation block 110. These modulation elements are often paired with a control surface 122 that is operated by the musician, for example a keyboard or a sequencer. Thus, the modulation may depend on a user input and may generally be time-dependent.

[0044] These above-mentioned elements of the sound generation block 110 and / or the sound modulation block 120 may be implemented with DSP, analog circuitry, digital circuitry, or any combination(s) thereof.

[0045] The output stage of the synthesizer 100 (not shown in FIG. 1) may comprise a mixing circuit that combines the internal audio signals (and optionally effects), the output of which is one or more audio signals. When multiple audio signals are output, the synthesizer 100 can be configured to output a signal set that has some notion of spatiality, such as a stereo or multi-channel signal. Furthermore, internal modulation signals, such as LFO-generated signals, can be used to affect spatial qualities, such as the pan position. In this case, the output is effectively rendered. The internal signals that are mixed together in the output stage may be voice signals, or oscillator element 112 outputs that are mixed to form multiple voice signals (e.g., in a multi-voice synthesizer), which may be based on oscillators, sample playback, or any other suitable tone generation technique. These signals may be output separately from the synthesizer 100 instead of being mixed.

[0046] Thus, the synthesizer 100 is based on rendered channel outputs such as mono or stereo, or even multi-channel output (eg, 3.1.2 multi-channel output, 5.1 multi-channel output, 5.1.2 multi-channel output, 7.1.4 multi-channel output, etc.).

[0047] To generate a stereo output in a synthesizer, several approaches are possible. One approach to generate stereo from mono is by using effects such as chorus, flanger, delay, or reverb. The synthesizer architecture may be primarily mono and involve a final stage effects processor that has a mono input and then applies different processing to create left and right signals to create a stereo output. In some cases, the final stage effects processor may be associated with a chorus effect output stage.

[0048] Another approach is to allow the user to pan the sound within the stereo field. This panning may be fully manual, or it may be tied to some characteristic of the sound such as pitch, or it may be modulated over time, for example with an LFO.

[0049] Additionally, multi-timbral synthesizers have the ability to play two or more programs simultaneously at once. This allows one program to be assigned to the bottom patch and one program to the top patch. These different programs or patches can be positioned across the stereo field.

[0050] The channel-rendered output of Synthesizer 100 can be used in an object-based workflow by taking this rendered output and then using a secondary tool to generate spatial metadata, such as the Dolby Atmos® Panner.

[0051] So when synthesizers are used in object-based (e.g. Dolby Atmos®) productions, the output must first be rendered and then spatialized in a second step. This means that any sound design decisions are finalized without the internal modulation signals used to generate the output being available. This limits the creative options available and makes it very difficult to consider spatialization as an integrated part of the sound design process.

[0052] An example of a processing chain 200 for such an object-based sound design process is shown in the block diagram of FIG. 2. The processing chain 200 comprises a synthesizer 205, a spatialization module 230, and an object-based rendering module (e.g., Dolby Atmos production suite) 240. The synthesizer 205 may correspond to the synthesizer 100 described above and may comprise a sound generation block 210 and a sound modulation block 220. The output of the synthesizer 205 is rendered into a specific (e.g., predefined) channel configuration, such as mono or stereo, or a multi-channel configuration (e.g., 3.1.2 multi-channel output, 5.1 multi-channel output, 5.1.2 multi-channel output, 7.1.4 multi-channel output, etc.). This rendered channel-based output is then provided to the spatialization module 230. The spatialization module 230 processes the channel-based output of the synthesizer 205 to create object-based audio content therefrom. The object-based audio content generated by the spatialization module 230 may then be used for object-based rendering by the object-based rendering module 240 into a desired (in principle any) channel configuration (e.g., depending on the intended speaker layout).

[0053] An example illustrating potential problems that may occur when using the processing chain 200 for object-based rendering is now described. According to this example, an LFO may be used to rhythmically change the filter cutoff frequency of a filter in the sound generation block 210. It may be desirable to also affect the spatial elevation of the signal with an LFO, so that the filter cutoff frequency is synchronized with the elevation. However, it is difficult to do so when using the processing chain 200. That is, the rhythmic change in the cutoff frequency may be intended to signify or follow a movement of the sound source (e.g., up and down). When generating spatial metadata to represent this movement based on the rendered channel-based output of the synthesizer 205, the internal modulation signal that controlled the adaptation of the cutoff frequency is no longer available. Instead, the intended spatial movement can only be inferred indirectly from the channels of the channel-based output, which may be tedious and / or inaccurate. For more complex examples, such as situations where synthesizer voice elevation positions are individually modulated with one or more LFOs, generating appropriate spatial metadata can be more tedious or even impossible since these voices may be mixed together.

[0054] The techniques of this disclosure relate to the incorporation of object spatialization into a synthesizer (or audio processor) architecture. This modification allows the synthesizer's internal control signals (internal modulation signals) used to shape the sound to also be used to affect spatialization. The synthesizer can then generate spatial metadata directly along with the audio without the need for an intermediate rendering step. Thus, the techniques of this disclosure directly output a collection of one or more audio streams and associated spatial metadata. This collection of signals can then be rendered or further processed and then rendered for playback.

[0055] This approach means that internal control signals or GUI controls used to position object outputs can also be used to adjust other aspects of sound generation (and vice versa). Also, individual oscillators can be output as objects, individual synthesizer voices can be output as objects, etc.

[0056] Although this disclosure may frequently refer to synthesizers (e.g., music synthesizers), it should be understood that the presented techniques may be equally applied to audio processors (e.g., audio effects processors) unless otherwise indicated. Thus, this disclosure may generally relate to apparatus for generating or processing audio signals (sound signals).

[0057] An example of an audio processing chain 300 including a synthesizer 305 consistent with techniques according to the present disclosure is shown generally in Figure 3. The processing chain 300 comprises a synthesizer 305 and an object-based rendering module 340 (separate from the synthesizer) for object-based rendering of the output of the synthesizer.

[0058] Broadly speaking, the present disclosure proposes incorporating a spatial output element (e.g. the third stage described below) that generates for each signal to be output a corresponding spatial metadata stream (or spatial metadata in general), where the spatial metadata instructs an external rendering unit on how to render that signal. In particular, the metadata stream may describe spatial attributes of its associated audio signal, such as its Cartesian position over time (e.g. in 3D space) and / or size. In this way, the spatial metadata may be metadata for object-based rendering of an audio object, where the audio track of the audio object is given by the audio signal. An example of such spatial metadata is Dolby Atmos® metadata. In general, this spatial metadata may be generated by direct and literal user input. But at the same time, it may also be derived from modulation signals (control signals) that may also be used to generate and / or shape the associated audio signal. Additionally, this spatial metadata may be generated by a combination of direct and literal user input and derived from modulation signals (control signals) that may also be used to generate and / or shape the associated audio signal.

[0059] The synthesizer 305 comprises a sound generation block 310, a sound modulation block 320, and a spatialization block 330. The synthesizer may further include an output stage (not shown in FIG. 3) for outputting an audio signal / stream together with associated spatial metadata.

[0060] The sound generation block 310 may comprise any of the elements described above for the sound generation block 110 of the synthesizer 100, such as oscillators, filters and / or amplifiers, as well as other elements for shaping the sound generated by the oscillators. Thus, sound generation by the sound generation block 310 may proceed in the same manner as by the sound generation block 110 described above. This does not exclude that the sound generation block 310 comprises additional elements and / or has additional functionality not described above. Further examples of possible elements of the sound generation block 310 are given below.

[0061] In general, conceptually, the sound generation block 310 may be considered as implementing a first stage for obtaining an audio signal and a second stage for modifying the audio signal (e.g., for shaping the sound represented by the audio signal).

[0062] In line with the above, the first stage of the synthesizer 305 of FIG. 3 may comprise one or more oscillators (i.e., oscillators of the sound generation block 310). These one or more oscillators may be used by the first stage to generate an audio signal. For example, each of the one or more oscillators may be an analog oscillator, an FM oscillator, or a wavetable oscillator. Depending on their style / implementation, these oscillators may have different operating parameters (e.g., oscillator parameters). For example, an analog oscillator may have frequency, pulse width, and / or gain / level as operating parameters. An FM oscillator may have frequency, ratio, FM depth, and / or gain / level as operating parameters. Furthermore, a wavetable oscillator may have frequency, wave index, bank index, and / or gain / level as operating parameters.

[0063] Additionally, generation of the audio signal by the one or more oscillators may be based at least in part on one or more control signals. For example, operating parameters (e.g., frequency, pulse width, and / or phase, and / or any of the operating parameters described above) of the one or more oscillators may be modulated under control of the one or more control signals.

[0064] While the above implementation of the first stage relates to a synthesizer (e.g., a music synthesizer), the present disclosure also relates to implementations (not shown in FIG. 3 ) in which the first stage receives an audio signal. For example, the audio signal may be received from an external source (e.g., an audio database, a sound database). Such implementations may relate to an audio processor, such as an effects audio processor.

[0065] In general, two or more audio signals (i.e., a plurality of audio signals) may be processed in parallel by a synthesizer or audio processor (i.e., a device for generating or processing audio signals in general). For such implementations, a combination of (internally) generating and receiving audio signals may also be feasible, e.g., implementations in which some audio signals are generated by an oscillator and some (other) audio signals are received from an external source.

[0066] In line with the above, and without any intended limitation, synthesizers are frequently mentioned, but embodiments of the present disclosure also relate to audio processors. The difference between these different implementations lies in whether the audio signal, which is later modified and supplemented with spatial metadata, is generated (internally) or received. It is understood that any other elements / functionality of the described synthesizer applies to the audio processor as well. That is, apart from how the audio signal is obtained (i.e., generated or received), the audio processor may have the same functionality as the synthesizer described throughout this disclosure.

[0067] As mentioned above, the second stage of the synthesizer 305 is a stage for modifying the audio signal (e.g. for shaping the sound represented by the audio signal). The modifying the audio signal is based on one or more (internal) control signals (e.g. internal modulation signals). The control signals may thus also be referred to as control signals for shaping the sound represented by the audio signal. As mentioned above, the control signals may be time-dependent (i.e. may vary over time). The second stage may thus be adapted to apply a time-dependent modification to the audio signal. The time-dependence of the modification may depend on the one or more control signals.

[0068] The second stage may comprise a filter (e.g., VCF) and / or an amplifier (e.g., VCA). In general, the second stage may comprise any, some, or all of the following elements for modifying the audio signal: one or more filters (e.g., VCF), one or more amplifiers (e.g., VCA), one or more LFOs, one or more audio delays, one or more drivers, and one or more flangers. The second stage may also have sound shaping elements, such as chorus and reverb. All of these elements may have respective operating parameters (e.g., synthesizer parameters, or audio processor parameters) that may be modified / altered / modulated according to one or more control signals.

[0069] For example, an amplifier may have gain as an operating parameter; a filter may have cutoff frequency and / or resonance (resonant frequency) as operating parameters; an LFO may have rate and / or scale as operating parameters; a delayer (delay effect) may have time, mix and / or feedback as operating parameters; a driver (drive effect) may have drive and / or mix as operating parameters; and a flanger (flanger effect) may have center, width, rate, regeneration and / or mix as operating parameters.

[0070] For each such operating parameter that is modified, there may be a respective control signal that controls the modification / modulation of this operating parameter. The control signal may be generated by a respective modulation source. Examples of modulation sources may include an LFO and / or an envelope.

[0071] In a first non-limiting example, the second stage may comprise or implement a filter (e.g., a VCF) that may be applied to the audio signal output from the first stage. In this case, the characteristic frequency of the filter may be controlled by said one or more control signals. In this sense, the characteristic frequency of the filter may be time-dependent. If the respective control signal is periodic (e.g., generated by an LFO), the characteristic frequency may vary periodically. In line with the above, the characteristic frequency of the filter may be a cut-off frequency. Alternatively, the characteristic frequency may be a resonant frequency.

[0072] In a second non-limiting example, the second stage may comprise or implement an amplifier (e.g., a VCA) that may be applied to the audio signal output from the first stage. In this case, the gain of the amplifier may be controlled by said one or more control signals. In this sense, the gain of the amplifier may be time-dependent. If the respective control signal is periodic (e.g., generated by an LFO), the gain may vary periodically.

[0073] Specifically, in the second example, the second stage may use an amplifier to apply an envelope (e.g., a gain profile) to the audio signal, in which case a corresponding control signal for the amplifier may represent the envelope.

[0074] The control signals may be generated by modulators (e.g., LFOs, etc.) as described above. In some implementations, at least some of the modulators may be modulated by other modulators under the control of respective control signals. Additionally, there may be built-in effects that are similarly modulated.

[0075] The (internal) control signals of the synthesizer 305 may be generated by an audio modulation block 320 which may comprise the same elements as the modulation block 120 of the synthesizer 100 described above. Thus, the sound modulation by the audio modulation block 320 may proceed in the same way as by the above-mentioned audio modulation block 120. This does not exclude that the audio modulation block 320 comprises additional elements and / or has additional functions not described above.

[0076] In general, the control signal may be generated by one or more modulators that affect how the audio signal is modified. The one or more modulators may, for example, be related to or include an LFO. As mentioned above, the operating parameters of the one or more modulators may themselves be subject to time-dependent modulation under control of an appropriate control signal. The one or more control signals may also be based at least in part on user input, as exemplified by the control surface (e.g., keyboard, control panel) 122 shown in FIG. 1.

[0077] The spatialization block 330 may be viewed as relating to or implementing a third stage of the synthesizer 305 for generating spatial metadata relating to the modified audio signal. As discussed above, this is done at least in part based on one or more control signals. Here, and below, it will be understood that the spatial metadata may be metadata for instructing an external device (e.g. a renderer or rendering module) on how to render the modified audio signal.

[0078] It will be appreciated that any of the control signals (e.g., the control signals mentioned above) may be taken as a basis for generating spatial metadata. For example, these control signals may be used as a basis for determining at least one of the position of the corresponding audio object in a horizontal plane, the altitude of the corresponding audio object and the size of the audio object. If the position of the audio object in a given plane (e.g., the horizontal plane) is determined based on the control signals, parameters of the linear translation may be determined (e.g., calculated) from the control signals, such as a control signal representing a time-dependent gain or gain profile. Similarly, parameters of a periodic movement (e.g., circular or elliptical movement) of the audio object in a given plane (e.g., the horizontal plane) may be determined (e.g., calculated) based on a periodic control signal, such as a control signal controlling a characteristic frequency of a filter.

[0079] In the first example above, an audio signal for an audio object that is perceived to rotate around a central position may be generated by periodically varying a characteristic frequency (e.g., cutoff, resonance) of a filter applied to the audio signal. A polar angle of the audio object (which may be used to derive appropriate Cartesian coordinates) may then be determined (e.g., calculated) based on the control signal used to modulate the characteristic frequency. Spatial metadata is thereby generated, at least in part, based on said one or more control signals (specifically, the control signal modulating the characteristic frequency of the filter).

[0080] In the second example mentioned above, an audio signal for an audio object that is perceived as moving in a room may be generated by using an amplifier to apply an envelope to the audio signal. Then, for example, a time-dependent position corresponding to a linear translation of the audio object (which may be used to derive appropriate Cartesian coordinates) may be determined (e.g., calculated) based on a control signal used to modulate the gain of the amplifier. In particular, the linear translation (or the time-dependent position, or spatial metadata in general) may be generated based on the shape of the envelope (gain profile) represented by the control signal. Thereby, here too, spatial metadata is generated, at least in part, based on said one or more control signals (in particular, a control signal modulating the gain of the amplifier).

[0081] In a third example, a plurality of audio signals may be generated by a plurality of oscillators. The relative (i.e., with respect to each other) spatial positions of the associated audio objects of the oscillators may then be modulated by an LFO that is also used to modulate their frequencies. For example, as a filter applied to the audio signals opens and closes cyclically under the control of the LFO, the objects move cyclically towards each other and then away from each other. This relative movement may be appropriately reflected in the generated metadata for the plurality of modified audio signals. Again, the spatial metadata is generated, at least in part, based on one or more control signals (specifically, the control signals generated by the LFOs).

[0082] In a fourth example, a plurality of audio signals may be generated by a plurality of oscillators. The relative spatial positions of the associated objects of the oscillators may then be controlled by an envelope that also controls the cutoff of a filter applied to the audio signal. The envelope may be triggered, for example, by an attached keyboard, or other suitable means for receiving user input. For example, when a key is first pressed, the sounds of the oscillators are felt to emanate from the same spatial position, but as the key continues to be pressed, the sound sources are felt to move away from each other spatially. As the sound sources move further away from each other, the filters may open wider, making the sound brighter. This relative movement may again be appropriately reflected in the generated metadata for the plurality of modified audio signals. Again, the spatial metadata is generated, at least in part, based on one or more control signals (specifically, the control signals generated by the envelopes).

[0083] In a fifth example, a random LFO shape may be used to control both the wavetable and the audio position of an oscillator that generates an audio signal (i.e., a voice). Over time, the audio object takes on random spatial positions, with each new position corresponding to a new wavetable. The random movement of the audio object may be appropriately reflected in the generated metadata for the audio signal. Again, the spatial metadata is generated, at least in part, based on one or more control signals (specifically, the control signals generated by the random LFO shape).

[0084] The output stage (fourth stage) of the synthesizer is an output stage for outputting the modified audio signal together with the generated spatial metadata. In particular, the output stage may be adapted to output one or more audio streams based on the modified audio signal together with, for each audio stream, the corresponding spatial metadata generated by the third stage. For example, there may be one spatial metadata stream for each output audio stream. When two or more audio signals are processed in parallel by the synthesizer 305, there may be one output audio stream for each (modified) audio signal. Alternatively, the modified audio signals may be mixed into an audio stream for output. In such a case, appropriate spatial metadata may be generated for the resulting audio stream. Alternatively, the spatial metadata for the output audio stream may be generated based on the spatial metadata of the individual modified audio signals to be mixed into the output audio stream.

[0085] Alternatively or additionally, two or more audio signals (i.e., multiple audio signals) may be processed serially by the synthesizer 305. For example, in some such implementations, the processing for a given audio signal may depend on previously processed audio signals and / or their metadata. As another example, in some implementations, the processing may involve modifying existing spatial metadata of the audio signal obtained by the first stage. In line with the above, this modification of the existing metadata may be based on an internal control signal (internal modulation signal).

[0086] Although the synthesizer 305 is described as outputting object-based audio content (i.e., modified audio signals together with generated spatial metadata), it may further be configured for output of rendered (e.g., channel-based) audio content. For example, the synthesizer 305 may be capable of rendering a binaural output for preview or for direct multi-channel input to a public announcement system. Thus, in addition to the output stage, the synthesizer may further comprise a rendering stage (or rendering unit, rendering module) for generating rendered audio content. However, it should be understood that the rendering stage is optional and the synthesizer may typically provide object-based audio content for external rendering.

[0087] A corresponding method 400 for generating or processing an audio signal is illustrated generally in the flow chart of Figure 4. The method 400 includes steps / processes S410 to S440.

[0088] Step S410 In, an audio signal is obtained. As mentioned above, this may involve generating or receiving the audio signal.

[0089] Step S420 In the method, an audio signal is modified based on one or more control signals to shape the sound represented by the audio signal.

[0090] Step S430 In the method, spatial metadata about the modified audio signal is generated based at least in part on one or more control signals.

[0091] lastly, Step S440 In, the modified audio signal is output together with the generated spatial metadata.

[0092] It is understood that step S410 may be performed by a first stage, step S420 may be performed by a second stage, step S430 may be performed by a third stage, and step S440 may be performed by an output stage of the above-mentioned apparatus for generating or processing an audio signal. It is further understood that any statements made above with respect to the respective stages apply equally to their corresponding method steps / processes, and repeated description may be omitted for brevity.

[0093] The present disclosure also relates to an apparatus (e.g., synthesizer, audio processor) for performing the methods and techniques described throughout the present disclosure. FIG. 5 shows an example of such an apparatus 500. The apparatus 500 comprises a processor 510 and a memory 520 coupled to the processor 510. The memory 520 may store instructions for the processor 510. The processor 510 may optionally receive an input 530 from an external source, such as a database. The input 530 may relate, for example, to a sound signal. Furthermore, the processor 510 may receive a user input 560, for example, via a suitable interface (e.g., keyboard, control panel, etc.). The user input may modify the sound generation or sound shaping, as described above. The processor 510 may be adapted to perform the methods / techniques described throughout the present disclosure. Thus, the processor 510 may output one or more (modified) sound signals (audio streams) 540 and associated metadata (metadata streams) 550.

[0094] The present disclosure also relates to a computer program comprising instructions which, when executed by a computer processor, cause the computer processor to perform the methods described throughout the present disclosure, and a computer readable storage medium storing said computer program.

[0095] Embodiments of the present disclosure may have in common that the (same) internal control signal (internal modulation signal) used to control the generation and / or modification / shaping of the audio signal is also used to generate spatial metadata for the audio signal. Compared to the case where no control signal is available to generate spatial metadata and any previous shaping operations applied to the audio signal are finalized to a channel-based output (e.g. mono, stereo or possibly multi-channel), this may allow additional flexibility in implementing creative intent and help to avoid tedious manual and potentially suboptimal editing of spatial metadata.

[0096] interpretation Aspects of the systems described herein may be implemented in a suitable computer-based sound processing system for generating and / or processing sound signals. One or more of the components, blocks, processes, or other functional components may be implemented through one or more computer programs that control the execution of one or more processor-based computing devices of the system. It should also be noted that the various functions disclosed herein may be described as data and / or instructions embodied in various machine-readable or computer-readable media, using any number of combinations of hardware, firmware, and / or in terms of their behavior, register transfers, logical components, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include various forms of physical (non-transitory) non-volatile media, such as, but not limited to, optical storage media, magnetic storage media, or semiconductor storage media.

[0097] In particular, it should be understood that the embodiments may include hardware, software, and electronic components or modules, which may be illustrated and described for purposes of explanation as if the majority of the components were implemented solely in hardware. However, those skilled in the art will recognize, based on reading this detailed description, that in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored in a non-transitory computer-readable medium) executable by one or more electronic processors, such as microprocessors and / or application specific integrated circuits ("ASICs"). Thus, it should be noted that a number of hardware and software-based devices, as well as a number of different structural components, may be utilized to implement the embodiments. For example, the blocks or stages described herein may include one or more electronic processors, one or more computer-readable media modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0098] Although one or more embodiments have been described with reference to specific embodiments by way of example, it is to be understood that the one or more embodiments are not limited to the disclosed embodiments. Rather, it is intended to cover various modifications and similar arrangements that would be apparent to those skilled in the art. Thus, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0099] It is also to be understood that the phraseology and terminology used herein is for purposes of description and should not be regarded as limiting. The use of "including," "comprising," or "having," and variations thereof, is intended to encompass the items listed and equivalents thereof, as well as additional items. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled," and variations thereof, are used broadly and encompass both direct and indirect mounting, connecting, supporting, and coupling.

[0100] Exemplary embodiments in bullet form Various aspects and implementations of the present disclosure can be understood from the following enumerated example embodiments (EEE), which are not claims. [EEE1] An apparatus for generating or processing an audio signal, the apparatus having: a first stage for obtaining an audio signal; a second stage for modifying the audio signal based on one or more control signals for shaping a sound represented by the audio signal; a third stage for generating spatial metadata relating to the modified audio signal based, at least in part, on the one or more control signals; and an output stage for outputting the modified audio signal together with the generated spatial metadata. [EEE2] The apparatus described in EEE1, wherein the one or more control signals are time-dependent. [EEE3] The device described in EEE1 or 2, wherein the spatial metadata is metadata for instructing an external device on how to render the modified audio signal. [EEE4] An apparatus described in any one of EEE1 to 3, wherein the second stage is adapted to apply a time-dependent modification to the audio signal, the time-dependence of the modification depending on the one or more control signals. [EEE5] The apparatus described in any one of EEE1 to 4, wherein the second stage has at least one of: a filter, an amplifier, a low frequency oscillator, an audio delay, a driver, and a flanger to modify the audio signal. [EEE6] An apparatus described in any one of EEE1 to 5, wherein the second stage is adapted to apply a filter to the audio signal, the characteristic frequency of the filter being controlled by the one or more control signals. [EEE7] The apparatus according to EEE6, wherein the characteristic frequency of the filter is a cut-off frequency. [EEE8] An apparatus described in any one of EEE1 to 7, wherein the second stage is adapted to apply an amplifier to the audio signal, the gain of the amplifier being controlled by the one or more control signals. [EEE9] The apparatus of claim 8, wherein the second stage is adapted to apply an envelope to the audio signal by using the amplifier, and the third stage is adapted to generate the spatial metadata based at least in part on a shape of the envelope. [EEE10] The apparatus of any one of EEE1 to 9, wherein obtaining the audio signal includes generating the audio signal by using one or more oscillators. [EEE11] The apparatus described in EEE10, wherein the audio signal is generated by the one or more oscillators based at least in part on the one or more control signals. [EEE12] 10. The apparatus of any one of claims EE1 to 9, wherein obtaining the audio signal comprises receiving the audio signal. [EEE13] 3. The apparatus of any one of claims EE1 to 12, wherein the one or more control signals are based at least in part on user input. [EEE14] 14. The apparatus of any one of EEE1 to 13, wherein the output stage is adapted to output one or more audio streams based on the modified audio signal together with the generated spatial metadata. [EEE15] A method for generating or processing an audio signal, the method comprising: obtaining an audio signal; modifying the audio signal based on one or more control signals for shaping a sound represented by the audio signal; generating spatial metadata relating to the modified audio signal based at least in part on the one or more control signals; and an output stage for outputting the modified audio signal together with the generated spatial metadata. [EEE16] The method of EEE15, wherein the one or more control signals are time dependent. [EEE17] The method according to EEE15 or 16, wherein the spatial metadata is metadata for instructing an external device on how to render the modified audio signal. [EEE18] A method according to any one of EEE15 to 17, wherein modifying the audio signal comprises applying a time-dependent modification to the audio signal, the time-dependence of the modification depending on the one or more control signals. [EEE19] A method according to any one of EEE15 to 18, wherein modifying the audio signal comprises modifying the audio signal by at least one of a filter, an amplifier, a low frequency oscillator, an audio delay, a driver, and a flanger. [EEE20] A method according to any one of EEE15 to 19, wherein modifying the audio signal comprises applying a filter to the audio signal, the characteristic frequency of the filter being controlled by the one or more control signals. [EEE21] The method according to EEE20, wherein the characteristic frequency of the filter is a cut-off frequency. [EEE22] A method according to any one of EEE15 to 21, wherein modifying the audio signal comprises applying an amplifier to the audio signal, the gain of the amplifier being controlled by the one or more control signals. [EEE23] The method of claim 8, wherein modifying the audio signal includes applying an envelope to the audio signal by using the amplifier, and generating the spatial metadata is based at least in part on a shape of the envelope. [EEE24] The method of any one of EEE15 to 23, wherein obtaining the audio signal includes generating the audio signal by using one or more oscillators. [EEE25] The method of EEE24, wherein the audio signal is generated by the one or more oscillators based at least in part on the one or more control signals. [EEE26] 24. The method of any one of claims 15 to 23, wherein obtaining the audio signal comprises receiving the audio signal. [EEE27] 27. The method of any one of EEE15 to 26, wherein the one or more control signals are based at least in part on user input. [EEE28] 28. The method of any one of EEE15 to 27, wherein outputting the modified audio signal comprises outputting one or more audio streams based on the modified audio signal together with the generated spatial metadata. [EEE29] A computer program comprising instructions which, when executed by a computer processor, cause the computer processor to carry out a method according to any one of claims EEE15 to 28. [EEE30] A computer-readable storage medium storing a computer program according to EEE29.

Claims

[Claim 1] 1. An apparatus for generating or processing an audio signal, the apparatus comprising: a first stage for obtaining an audio signal; a second stage for modifying the audio signal based on one or more control signals for shaping the sound represented by the audio signal; a third stage for generating spatial metadata about the modified audio signal based at least in part on the one or more control signals; and an output stage for outputting the modified audio signal together with the generated spatial metadata. Device.

Images