Method, apparatus and system for user generated content capture and adaptive rendering

Abstract

A method of processing audio data related to user-generated content is described. A method includes: obtaining the audio data; applying frame-by-frame audio enhancement to the audio data; generating metadata of the enhanced audio data based on one or more processing parameters of the frame-by-frame audio enhancement; and outputting the enhanced audio data together with the metadata. Another method includes obtaining the audio data and metadata of the audio data, where the metadata includes first metadata indicating one or more processing parameters of a previous frame-by-frame audio enhancement of the audio data; applying a recovery process to the audio data using the one or more processing parameters to at least partially reverse a previous frame-by-frame audio enhancement; and applying frame-by-frame audio enhancement or editing processing to the restored original audio data. Corresponding apparatus, programs, and computer-readable storage media are further described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 336,700, filed on April 29, 2022, and International Application No. PCT / CN2022 / 085777, filed on April 8, 2022, each of which is incorporated herein in its entirety. Technical Field

[0003] The present invention relates to methods, apparatus and systems for capturing and adaptive rendering of user-generated content (UGC). The present invention relates in particular to UGC content creation on mobile devices that enables adaptive rendering during playback and to adaptive rendering during playback. Background Art

[0004] Recently, UGC has become a trend to share personal moments in variable environments. UGC is mostly recorded by mobile devices. Due to consumer hardware limitations, system performance requirements, the diversity of capture methods, and playback environments, most of this content will have sound artifacts.

[0005] In order to overcome the sound quality issues caused by hardware limitations and recording environments, UGC audio can be enhanced to obtain a better listening experience. Using the information available at the time, certain audio enhancements can be applied in real time during or immediately after capture. This enhancement can be applied directly to the audio stream and generate an enhanced audio stream in real time. The enhanced audio can then be rendered without the need for specific software support on the playback device. Thus, UGC content creators can improve the audio quality of their content without paying extra effort and ensure that this enhancement is available to their content consumers to the greatest extent.

[0006] However, for further enhanced audio quality, there are also some audio enhancements that rely on additional information beyond the real-time available information. In addition, real-time enhancement after capture may not be compatible with end-to-end content processing and user experience.

[0007] Therefore, improved techniques for UGC capture and adaptive rendering are urgently needed. Summary of the invention

[0008] According to one aspect, a method for processing audio data associated with user-generated content is provided. For example, the method can be performed by a mobile device. The method can include obtaining audio data. Obtaining audio data can include or be equivalent to capturing audio data by a suitable capture device. The capture device can be part of the mobile device, or can be connected / connectable to the mobile device. Further, the capture device can be, for example, a binaural capture device that can record at least two channel recordings. The method can further include applying frame-by-frame audio enhancement to the audio data to obtain enhanced audio data. The method can further include generating metadata for the enhanced audio data based on one or more (e.g., multiple) processing parameters of the frame-by-frame audio enhancement. The method can also further include outputting the enhanced audio data together with the generated metadata.

[0009] Configured as described above, the proposed method can provide enhanced audio data suitable for direct playback by a playback device without further audio processing by the playback device. On the other hand, the method also provides contextual metadata for the enhanced audio data. The contextual metadata makes it possible to restore the original audio so that additional / additional audio enhancement can be performed by a playback device with different (e.g., better) processing capabilities, or so that audio editing can be performed using editing tools. Thus, rendering can be performed at the playback device in an adaptive manner based on the hardware capabilities of the device, the playback environment, user-specific settings, etc. In other words, providing contextual metadata allows end-to-end content processing from capture to playback that takes into account the characteristics of specific capture and rendering hardware, specific environments, user preferences, etc., thereby achieving optimal enhancement of audio data and listening experience.

[0010] In some embodiments, applying frame-by-frame audio enhancement to the audio data may include applying at least one of the following: noise management, loudness management, timbre management, and peak limiting. For example, noise management may involve denoising. For example, loudness management may involve level adjustment and / or dynamic range control.

[0011] By such processing, the enhanced audio data is suitable for being directly replayed by a playback device without the need for additional audio processing at the playback device. Therefore, the UGC generated by the proposed method is particularly suitable for consumption by mobile devices with generally limited processing capabilities, for example, for use in a streaming framework for devices without specific software support for reading metadata. On the other hand, if the device in the streaming framework has specific software support for reading metadata, the metadata and the enhanced audio data can be read, the metadata can be used to generate / restore the original audio from the enhanced audio data, and further enhanced audio can be generated based on the original audio.

[0012] In some embodiments, one or more processing parameters may include frequency band gains and / or full-band gains applied during frame-by-frame audio enhancement. Frequency band gains or full-band gains may include corresponding gains for each frame of audio data. Further, frequency band gains or full-band gains may include corresponding gains for each type of enhancement processing applied. Metadata may include actual gains or indications thereof.

[0013] Thus, in some embodiments, the one or more processing parameters may include at least one of: a frequency band gain for noise management, a full frequency band gain for loudness management, a frequency band gain for timbre management, and a full frequency band gain for peak limiting. Knowing these gains, a device (e.g., a playback device, an editing device) receiving the enhanced audio data can, if necessary, reverse any enhancement processing applied after capture to subsequently apply different audio enhancements and / or audio edits.

[0014] In some embodiments, frame-by-frame audio enhancement can be applied in real-time. That is, frame-by-frame audio enhancement can be real-time frame-by-frame audio enhancement. The enhanced audio data generated in this way will be particularly suitable for streaming applications, etc.

[0015] In some embodiments, metadata may be generated further based on the results of the analysis of multiple frames of audio data. In some embodiments, the analysis of multiple frames of audio data may generate long-term statistics of the audio data. For example, the long-term statistics may be file-based statistics. Additionally or alternatively, the analysis of multiple frames of audio data may generate one or more audio features of the audio data.

[0016] In some embodiments, the audio feature of the audio data may relate to at least one of the following: a content type of the audio data, an indication of a capture environment of the audio data, a signal-to-noise ratio of the audio data, an overall loudness of the audio data, and a spectral shape of the audio data. For example, the overall loudness of the audio data may relate to file loudness. For example, the spectral shape may relate to a spectral envelope.

[0017] Including such additional information in the metadata enables any device receiving the enhanced audio data and metadata to perform more complex audio enhancements that might not be possible in real time and / or perform audio enhancements that are tailored to particular use cases, environments, etc.

[0018] In some embodiments, the metadata may include first metadata generated based on one or more processing parameters of frame-by-frame audio enhancement and second metadata generated based on the results of analyzing multiple frames of audio data. Then, the method may further include compiling the first metadata and the second metadata to obtain compiled metadata as metadata for output (context metadata). For example, the first metadata may be referred to as enhanced metadata. For example, the second metadata may be referred to as long-term metadata.

[0019] According to another aspect, a method for processing audio data associated with user-generated content is provided. The method may include obtaining audio data. The method may further include obtaining metadata for the audio data. Wherein, the metadata may include first metadata, the first metadata indicating one or more processing parameters of a previous (earlier; for example, capture side) frame-by-frame audio enhancement of the audio data. Obtaining audio data and metadata may include or be equivalent to receiving a bitstream including audio data and metadata, including, for example, obtaining audio data and metadata from a storage medium. The method may further include applying a recovery process to the audio data using one or more processing parameters to at least partially reverse the previous frame-by-frame audio enhancement to obtain the original audio data. The method may further include applying a frame-by-frame audio enhancement to the original audio data to obtain enhanced audio data. Additionally or alternatively, the method may include applying an editing process to the original audio data to obtain edited audio data.

[0020] By restoring the original audio data, the playback / editing device can apply audio enhancement or audio editing based on its processing capabilities, user preferences, playback environment, long-term statistics, etc. Thus, end-to-end content processing and optimal user experience can be achieved. On the other hand, if the processing power is insufficient for audio enhancement, the received enhanced audio data can be directly rendered without additional processing.

[0021] In some embodiments, applying a restoration process to the audio data includes applying at least one of the following: background sound restoration, loudness restoration, peak restoration, and timbre restoration. Here, it should be understood that noise management / noise suppression can suppress background sound as noise, depending on the definition of "noise" and "background sound". For example, if speech is the main interest, footsteps may belong to noise, but if it is considered as part of the soundscape, footsteps may belong to background sound. Therefore, in the restoration process, noise management is reversed or partially reversed with reference to "background sound" restoration.

[0022] In some embodiments, the one or more processing parameters may include frequency band gains and / or full-band gains applied during a previous frame-by-frame audio enhancement. Thus, in some embodiments, the one or more processing parameters may include at least one of the following: frequency band gains for previous noise management, full-band gains for previous loudness management, full-band gains for previous peak limiting, and frequency band gains for previous timbre management.

[0023] In some embodiments, the metadata may further include second metadata indicating long-term statistics of the audio data and / or indicating one or more audio features of the audio data. The statistics of the audio data and / or the audio features of the audio data may be based on audio before or after a previous frame-by-frame audio enhancement, or, if applicable, even for audio data between two consecutive previous frame-by-frame audio enhancements.

[0024] In some embodiments, the audio features of the audio data may relate to at least one of: a content type of the audio data, an indication of a capture environment of the audio data, a signal-to-noise ratio of the audio data prior to previous frame-by-frame audio enhancement, an overall loudness of the audio data prior to previous frame-by-frame audio enhancement, and a spectral shape of the audio data prior to previous frame-by-frame audio enhancement.

[0025] In some embodiments, frame-by-frame audio enhancement can be applied to the original audio data based on the second metadata. Thus, more complex audio enhancement processing than real-time enhancement can be applied, thereby improving the auditory experience.

[0026] In some embodiments, applying frame-by-frame audio enhancement to the raw audio data may include applying at least one of: noise management, loudness management, peak limiting, and timbre management.

[0027] According to another aspect, an apparatus for processing audio data associated with user-generated content is provided. The apparatus may include a processing module for applying frame-by-frame audio enhancement to the audio data to obtain enhanced audio data, and for outputting the enhanced audio data. The apparatus may further include an analysis module for generating metadata for the enhanced audio data based on one or more processing parameters of the frame-by-frame audio enhancement, and for outputting the metadata. Additionally, the apparatus may further include a capture module for capturing the audio data.

[0028] In some embodiments, the processing module may be configured to apply at least one of the following to the audio data: noise management, loudness management, peak limiting, and timbre management.

[0029] In some embodiments, the one or more processing parameters may include frequency band gains and / or full-band gains applied during frame-by-frame audio enhancement.

[0030] In some embodiments, the one or more processing parameters may include at least one of: a frequency band gain for noise management, a full-band gain for loudness management, a full-band gain for peak limiting, and a frequency band gain for timbre management.

[0031] In some embodiments, the processing module may be configured to apply frame-by-frame audio enhancement in real-time.

[0032] In some embodiments, the analysis module can be configured to generate metadata further based on the results of analyzing multiple frames of audio data. In some embodiments, the analysis of multiple frames of audio data can generate long-term statistics of the audio data. In some embodiments, the analysis of multiple frames of audio data can generate one or more audio features of the audio data.

[0033] In some embodiments, the audio features of the audio data may relate to at least one of: a content type of the audio data, an indication of the capture environment of the audio data, a signal-to-noise ratio of the audio data, an overall loudness of the audio data, and a spectral shape of the audio data.

[0034] In some embodiments, the analysis module can be configured to generate first metadata based on one or more processing parameters of frame-by-frame audio enhancement and generate second metadata based on the results of analyzing multiple frames of audio data. The analysis module can be further configured to compile the first metadata and the second metadata, thereby obtaining the compiled metadata as metadata for output.

[0035] According to another aspect, a device for processing audio data associated with user-generated content is provided. The device may include an input module for receiving audio data and metadata of the audio data. The metadata may include first metadata indicating one or more processing parameters of a previous frame-by-frame audio enhancement of the audio data. The device may further include a processing module for applying a recovery process to the audio data using one or more processing parameters to at least partially reverse the previous frame-by-frame audio enhancement to obtain the original audio data. The device may further include at least one of a rendering module and an editing module. The rendering module may be a module for applying frame-by-frame audio enhancement to the original audio data to obtain enhanced audio data. The editing module may be a module for applying an editing process to the original audio data to obtain edited audio data.

[0036] In some embodiments, the processing module may be configured to apply at least one of the following to the audio data: background sound restoration, loudness restoration, peak restoration, and timbre restoration.

[0037] In some embodiments, the one or more processing parameters may include frequency band gains and / or full-band gains applied during a previous frame-by-frame audio enhancement. Thus, in some embodiments, the one or more processing parameters may include at least one of the following: frequency band gains for previous noise management, full-band gains for previous loudness management, full-band gains for previous peak limiting, and frequency band gains for previous timbre management.

[0038] In some embodiments, the metadata may further include second metadata indicating long-term statistics of the audio data and / or indicating one or more audio features of the audio data.

[0039] In some embodiments, the audio features of the audio data may relate to at least one of: a content type of the audio data, an indication of a capture environment of the audio data, a signal-to-noise ratio of the audio data prior to previous frame-by-frame audio enhancement, an overall loudness of the audio data prior to previous frame-by-frame audio enhancement, and a spectral shape of the audio data prior to previous frame-by-frame audio enhancement.

[0040] In some embodiments, the rendering module may be configured to apply frame-by-frame audio enhancement to the original audio data based on the second metadata.

[0041] In some embodiments, the rendering module may be configured to apply at least one of the following to the raw audio data: noise management, loudness management, peak limiting, and timbre management.

[0042] According to another aspect, a device for processing audio data related to user-generated content is provided. The device may include a processor and a memory, the memory being coupled to the processor and storing instructions for the processor. The processor may be configured to perform all steps of the method according to the aforementioned aspect and its embodiments.

[0043] According to a further aspect, a computer program is described. The computer program may include executable instructions for performing the method or method steps outlined throughout the present disclosure when executed by a computing device.

[0044] According to another aspect, a computer readable storage medium is described. The storage medium may store a computer program adapted to be executed on a processor and for performing the method or method steps outlined throughout the present disclosure when executed on the processor.

[0045] It should be noted that the methods and systems as outlined in this disclosure (including preferred embodiments thereof) can be used independently or in combination with other methods and systems disclosed in this document. In addition, all aspects of the methods and systems outlined in this disclosure can be combined arbitrarily. In particular, the features of the claims can be combined with each other in any manner.

[0046] It will be understood that device features and method steps can be interchanged in a variety of ways. In particular, as will be appreciated by the skilled person, details of the disclosed method(s) may be implemented by corresponding devices, and vice versa. Furthermore, any statements above regarding method(s) (and, for example, steps thereof) should be understood to apply equally to corresponding devices (and, for example, blocks, stages, units thereof), and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention is explained below in an exemplary manner with reference to the accompanying drawings, in which:

[0048] Figure 1 A conceptual diagram illustrating an example apparatus for UGC processing during / after capture according to an embodiment of the present disclosure;

[0049] Figure 2 is a flow chart illustrating an example method of UGC processing during / after capture according to an embodiment of the present disclosure;

[0050] Figure 3 A conceptual diagram illustrating an example apparatus for performing UGC processing for rendering;

[0051] Figure 4 A conceptual diagram illustrating an example apparatus for performing UGC processing for rendering according to an embodiment of the present disclosure is illustrated;

[0052] Figure 5 is a flowchart illustrating an example method for performing UGC processing for rendering according to an embodiment of the present disclosure; and

[0053] Figure 6 A conceptual diagram of an example computing device for performing techniques in accordance with embodiments of the present disclosure is illustrated. DETAILED DESCRIPTION

[0054] In general, the present disclosure relates to methods, apparatus and systems for UGC content creation, for example on a mobile device, which can be adaptively rendered based on information available at a playback device, and to methods, apparatus and systems for UGC adaptive rendering.

[0055] The real-time audio enhancement on the capture side can produce enhanced audio content, and the enhanced audio content can be rendered without the need for specific support at the playback device. On the other hand, there are also some more complex audio enhancements, which rely on extra information outside the real-time available information to further enhance the audio quality. According to the technology described herein, after the audio capture and real-time enhancement process are completed, this further audio enhancement that is usually stored as metadata with the audio stream is applied to the audio stream. Utilizing the playback device that can read the metadata, this further audio enhancement can be applied to audio content rendering or audio editing. Therefore, for some content consumers with a specific playback device that can read metadata, or for all content consumers after utilizing the software tool that can read metadata to edit content, the technology described herein can further improve the audio quality of UGC.

[0056] At a conceptual level, a capture and rendering ecosystem according to an embodiment of the present disclosure may be composed of or characterized by some or all of the following elements:

[0057] A binaural capture device that can record at least two channel recordings and a playback device that can render at least two channel recordings. The recording device and the playback device can be the same device, two connected devices, or two separate devices.

[0058] The capture device includes a processing module for enhancing the captured audio in a real-time manner. The processing includes at least one of the following: level adjustment, dynamic range control, noise management, and timbre management.

[0059] The capture device includes an analysis module for providing contextual information and long-term or file-based features from the audio recording. The analysis results will be stored as contextual metadata along with the enhanced audio content generated by the processing module.

[0060] The metadata includes: frame-by-frame analysis results, which include at least frequency band gains or full-band gains applied by one or more components of the processing module; and file-based global results, which include contextual information and at least one of the loudness, content type, etc. of the audio.

[0061] During playback, rendering is adaptive based on the availability of contextual metadata.

[0062] In one case, the playback device has access only to the enhanced audio, so during playback it will directly render the enhanced audio without processing, or process it without the help of contextual metadata.

[0063] In another case, the playback device can access the enhanced audio and contextual metadata. During playback, the playback device will further process the enhanced audio based on the contextual metadata to improve the listening experience.

[0064] The capture device and / or playback device may also have editing tools. When the editing tools have access to contextual metadata, editing of the enhanced audio will generate results comparable to those of the original audio.

[0065] Figure 1 An apparatus (e.g., device, system) 100 for processing audio data 105 associated with UGC is schematically illustrated. The apparatus 100 may relate to the capture side of UGC and may therefore correspond to or be included in a mobile device (e.g., a mobile phone, a tablet computer, a PDA, a laptop computer, etc.). The processing performed by the apparatus 100 may enable adaptive rendering at a rendering device or a playback device. The apparatus 100 includes a processing module 110 and an analysis module 120. Optionally, the apparatus 100 may further include a capture module (not shown) for capturing the audio data 105. For example, the capture module (or capture device) may be a binaural capture device that may record at least two channel recordings.

[0066] The processing module 110 is adapted to apply frame-by-frame audio enhancement to the audio data 105. Such frame-by-frame audio enhancement can be applied in real-time, i.e., during or immediately after capturing the UGC. As a result of the frame-by-frame audio enhancement, enhanced audio data 115 is obtained and output by the processing module 110. The processing module 110 can perform the aforementioned audio enhancement, which can be applied in real-time. Thus, the processing module 110 generates enhanced audio data 115 (enhanced audio), which can be rendered without specific support at the playback device.

[0067] Specifically, the processing module 110 may be configured to apply at least one of the following to the audio data 105: noise management, loudness management, peak limiting, and timbre management. Figure 1 The processing module 110 in the example device 100 includes a noise management module 130, a loudness management module 140 and a peak limit module 150. The optional timbre management module is not shown in the figure. It should be noted that, depending on the specific application, not all of the aforementioned modules for audio enhancement are present.

[0068] The audio enhancement performed by the processing module 110 can be based on corresponding processing parameters. For example, there may be a different (set of) processing parameters for each of noise management, loudness management, peak limiting, and timbre management (if present). As described in more detail below, the processing parameters include frequency band gains and / or full-band gains applied during frame-by-frame audio enhancement. The frequency band gains or full-band gains can include corresponding gains for each frame of audio data. Further, the frequency band gains or full-band gains can include corresponding gains for each type of enhancement processing applied.

[0069] The noise management module 130 may be adapted to apply noise management involving suppressing interfering noises that are often present in non-professional recording environments. Thus, noise management may involve, for example, denoising. The noise management module 130 may be implemented, for example, by a machine learning algorithm or a neural network including a recurrent neural network (RNN) or a convolutional neural network (CNN), the implementation details of which may be understood to be obvious to an expert in the field. Further, noise management may involve pitch filtering.

[0070] The processing parameters for noise management may include frequency band gains (e.g., multiple frequency band gains) for noise management. These frequency band gains may relate to gains within corresponding frequency bands in multiple frequency bands (e.g., frequency sub-bands). Further, there may be one such frequency band gain per frame and frequency band. For example, in the case of pitch filtering, the processing parameters for noise management may include filter parameters for pitch filtering, such as the center frequency of the filter.

[0071] The loudness management module 140 may be adapted to apply loudness management, which involves leveling the input audio stream (i.e., audio data 105) to a specific loudness range. Loudness management may involve level adjustment and / or range control. For example, the input audio stream may be leveled to a loudness range that is more suitable for playback by a playback device at a later time. Thus, loudness management may adjust the loudness of the audio stream to an appropriate range for a better listening experience.

[0072] Loudness management may be implemented by automatic gain control (AGC), dynamic range control (DRC), or a combination of both, the implementation details of which are understood to be obvious to experts in the art.

[0073] The processing parameters for loudness management may include gains for loudness management. These gains may involve full-band gains that are applied uniformly across the entire frequency range, i.e., uniformly across multiple frequency bands (e.g., frequency sub-bands). There may be one such gain per frame.

[0074] The peak limiting module 150 may be adapted to apply peak limiting, which involves ensuring that the amplitude of the enhanced input audio does not exceed a reasonable range allowed by audio storage, distribution and / or playback. Again, implementation details may be understood to be obvious to experts in the field.

[0075] The processing parameters for peak limiting may include gains for peak limiting. These gains may involve full-band gains that are applied uniformly to multiple frequency bands (e.g., frequency sub-bands). There may be one such gain per frame.

[0076] A timbre management module (not shown) may be adapted to apply timbre management, involving adjusting the timbre of the audio data 105. The processing parameters for timbre management may include frequency band gains (e.g., multiple frequency band gains) for timbre management. These frequency band gains may relate to gains within corresponding frequency bands in a plurality of frequency bands (e.g., frequency sub-bands). Further, there may be one such frequency band gain per frame and frequency band.

[0077] The processing module 110 provides one or more (e.g., multiple) processing parameters for frame-by-frame audio enhancement to the analysis module 120. The processing parameters may be provided in a frame-by-frame manner. For example, updated values ​​of the processing parameters may be provided for each frame or each predefined multiple frames (e.g., every other frame, every N frames, etc.). The processing parameters may include any, some, or all of the processing parameters 135 for noise management, the processing parameters 145 for loudness management, the processing parameters 155 for peak limiting, and the processing parameters (not shown) for timbre management.

[0078] As a further input, the analysis module 120 may receive (a version of) the audio data 105 .

[0079] The analysis module 120 is adapted to generate metadata 125 (context metadata) for the enhanced audio data 115. The metadata 125 is generated based on one or more processing parameters of the frame-by-frame audio enhancement. For example, the metadata 125 may include processing parameters (e.g., frequency band gain and / or full frequency band gain) or an indication thereof.

[0080] The analysis module 120 is also adapted to output metadata 125. In other words, the analysis module 120 analyzes the audio data 105 and / or the aforementioned audio enhancement performed by the processing module 110 to generate contextual metadata 125 for audio enhancement that relies on additional information beyond the real-time available information to further improve the audio quality. The generated contextual metadata 125 can be used by a specific playback device or editing tool to obtain better audio quality and user experience.

[0081] Based on one or more processing parameters of the frame-by-frame audio enhancement, the analysis module 120 may generate first metadata 165 (eg, enhancement metadata) as part of the contextual metadata 125. For example, as described above, the first metadata 165 may include the processing parameters or indications thereof.

[0082] In addition to the one or more processing parameters of the audio enhancement, the analysis module 120 may further generate contextual metadata 125 based on the results of analyzing the plurality of frames of the audio data 105. Such analysis of the plurality of frames of the audio data 105 (i.e., analysis of the audio data 105 over time) may generate long-term statistics (e.g., file-based statistics) of the audio data 105. Additionally or alternatively, the analysis of the plurality of frames of the audio data 105 may generate one or more audio features of the audio data 105. Examples of audio features that may be determined in this manner include the content type of the audio data 105 (e.g., music, speech, movie, effects, etc.), an indication of the capture environment of the audio data 105 (e.g., a quiet / noisy environment, an environment with / without echo or reverberation, etc.), a signal-to-noise ratio SNR of the audio data 105, an overall loudness of the audio data 105 (e.g., file loudness), and a spectral shape (e.g., spectral envelope) of the audio data 105. Based on the results of analyzing the plurality of frames of the audio data, the analysis module 120 may generate second metadata 175 (e.g., long-term metadata) as part of the contextual metadata 125. For example, the second metadata 175 may include long-term statistics and / or audio features, or an indication thereof. The first metadata 165 and the second metadata 175 may be compiled to obtain compiled metadata as the contextual metadata 125 for output. It should be understood that the contextual metadata 125 may include any one or both of the first metadata 165 based on one or more processing parameters and the second metadata 175 based on the analysis of multiple frames of the audio data 105.

[0083] exist Figure 1 In the example of , the analysis module 120 includes a processing statistics module 160, a long-term statistics module 170 and a metadata compiler module 180 (metadata compiler).

[0084] The processing statistics module 160 implements the generation of the first metadata 165 based on one or more processing parameters. The processing statistics module tracks key parameters of the processing applied in the processing module 110 so that at a later time, such as during playback, the rendering system can better estimate the original audio (before the capture side audio enhancement) based on the enhanced audio stream including the enhanced audio data 115 (enhanced audio data) and the metadata 125 (context metadata). Therefore, the analysis of the one or more processing parameters of the audio enhancement by the processing statistics module can generate processing statistics of the audio enhancement performed by the processing module 110.

[0085] The long-term statistics module 170 implements the generation of the second metadata 175 based on the analysis of multiple frames of the audio data 105 (i.e., the long-term analysis of the audio data). The long-term statistics module analyzes the context information of the audio data 105 over a longer time span than allowed in real-time processing (e.g., within a few frames or seconds, or over the entire file). In general, the statistics obtained in this way are more accurate and stable than real-time statistics. The metadata compiler module 180 finally collects information (e.g., the first metadata 165 and the second metadata 175) from the processing statistics module 160 and the long-term statistics module 170, and compiles it into a specific format so that the information can be obtained later with a metadata parser. In other words, the metadata compiler module 180 compiles the first metadata 165 and the second metadata 175 to obtain the compiled metadata as metadata 125 (context metadata) for output. As a result of the above processing, the device 100 outputs the enhanced audio data 115 together with the context metadata 125. For example, the enhanced audio data 115 and the context metadata 125 can be output as an enhanced audio stream in a suitable format. Depending on the capabilities of the device, the enhanced audio stream may be used for adaptive rendering on the playback device, as described further below.

[0086] Although an example apparatus 100 for UGC processing has been described above, the present disclosure is also directed to a corresponding UGC processing method. It should be understood that any statements made above about the apparatus 100 are also applicable to the corresponding method, and vice versa. Figure 2 An example of such a UGC processing (eg, processing of audio data associated with UGC) method 200 is illustrated in the flowchart of . The method 200 includes steps S210 to S240 and may be performed during / after capturing UGC. For example, the method may be performed by a mobile device.

[0087] exist Step S210 , obtaining audio data. Obtaining audio data may include or be equivalent to capturing audio data by a suitable capture device. For example, the capture device may be a binaural capture device that can record at least two channel recordings.

[0088] exist Step S220, applying frame-by-frame audio enhancement to the audio data to obtain enhanced audio data. This step may correspond to the processing of the processing module 110 described above. In general, applying frame-by-frame audio enhancement to the audio data may include applying at least one of the following: noise management (e.g., as performed by the noise management module 130), loudness management (e.g., as performed by the loudness management module 140), peak limiting (e.g., as performed by the peak limiting module 150), and timbre management (e.g., as performed by the timbre management module). Further, the frame-by-frame audio enhancement may be applied in a real-time manner (e.g., applied during or immediately after capturing the audio data), and thus may be referred to as real-time frame-by-frame audio enhancement.

[0089] exist Step S230 , metadata (context metadata) of the enhanced audio data is generated based on one or more processing parameters of the frame-by-frame audio enhancement. This step may correspond to the processing of the above-mentioned analysis module 120. Accordingly, the one or more processing parameters may include frequency band gains and / or full-band gains applied during frame-by-frame audio enhancement. Specifically, the one or more processing parameters may include at least one of the following: frequency band gains for noise management, full-band gains for loudness management, full-band gains for peak limiting, and frequency band gains for timbre management. In addition to the one or more processing parameters, metadata may be further generated based on the results of analyzing multiple frames (e.g., all) of the audio data (e.g., as performed by the long-term statistics module 170). Among them, the analysis of multiple frames of audio data can generate long-term statistics of audio data (e.g., file-based statistics) and / or one or more audio features of audio data (e.g., content type of audio data, indication of the capture environment of audio data, signal-to-noise ratio of audio data, overall loudness of audio data, and / or spectral shape of audio data, etc.).

[0090] Thus, the metadata may include first metadata (e.g., enhancement metadata) generated based on one or more processing parameters of frame-by-frame audio enhancement (e.g., as generated by the processing statistics module 160) and second metadata (e.g., long-term metadata) generated based on the results of analyzing multiple frames of audio data (e.g., as generated by the long-term statistics module 170). In this case, the first metadata and the second metadata may be compiled to obtain compiled metadata as metadata for output (e.g., as done by the metadata compiler module 180).

[0091] exist Step S240 , outputting the enhanced audio data together with the generated metadata.

[0092] Next, we will refer to Figures 3 to 5Possible implementations for processing UGC at a playback device or an editing device are described. Figure 3 A conceptual diagram of an example apparatus (eg, device, system) 300 for UGC processing for rendering, such as a general audio rendering system for UGC, is illustrated.

[0093] The device 300 includes a rendering module 310 having a noise management module 320, a loudness management module 330, a timbre management module 340, and a peak limiting module 350. The device 300 takes only the aforementioned enhanced audio data 305 as input and applies blind processing without the aid of any information other than the audio itself. Ultimately, the device 300 outputs a rendered output 315 for playback. Alternatively, the device 300 may receive but ignore any contextual metadata provided with the enhanced audio data 305.

[0094] Figure 4 An apparatus (e.g., device, system) 400 for processing enhanced audio data 405 associated with UGC (e.g., a rendering apparatus for UGC) is schematically illustrated. Apparatus 400 may be related to the playback side of UGC, and thus may correspond to or be included in a mobile device (e.g., a mobile phone, a tablet computer, a PDA, a laptop computer, etc.) or any other computing device. In contrast to the blind processing of apparatus 300, apparatus 400 is configured for context-aware processing of UGC based on received context metadata.

[0095] Thus, in addition to the enhanced audio 405, the apparatus 400 also takes as input the aforementioned contextual metadata 435, which can be used to properly guide the rendering process to generate a further enhanced rendered output 425. To this end, the apparatus 400 includes a metadata parser 430 (e.g., as part of an input module) and several processing components. The processing components in this example can be divided into two groups related to "restoration" and "rendering".

[0096] Generally, the apparatus 400 may include an input module (not shown) for receiving (enhanced) audio data 405 and (contextual) metadata 435 of the audio data, a processing module 410 for applying a restoration process to the audio data 405, and at least one of a rendering module 420 and an editing module (not shown). For example, the audio data 405 and the metadata 435 may be received in the form of a bitstream including the audio data 405 and the metadata 435, including retrieving the audio data 405 and the metadata 435 from a storage medium.

[0097] exist Figure 4In the example of , the apparatus 400 includes a metadata parser 430 (eg, as part of an input module). The metadata parser 430 takes as input context metadata 435 (eg, generated by the aforementioned metadata compiler 180 of the apparatus 100).

[0098] Consistent with the above, metadata 435 includes first metadata 440 indicating one or more processing parameters of a previous (earlier, e.g., capture-side) frame-by-frame audio enhancement of the audio data. Additionally or alternatively, metadata 435 includes second metadata 445 indicating long-term statistics of the audio data and / or indicating one or more audio features of the audio data (e.g., the content type of the audio data, an indication of the capture environment of the audio data, the signal-to-noise ratio of the audio data before the previous frame-by-frame audio enhancement, the overall loudness of the audio data before the previous frame-by-frame audio enhancement, and / or the spectral shape of the audio data before the previous frame-by-frame audio enhancement, etc.). The statistics of the audio data and / or the audio features of the audio data may be based on the audio before or after the previous frame-by-frame audio enhancement, or, if applicable, may even be for the audio data between two consecutive previous frame-by-frame audio enhancements.

[0099] The metadata parser 430 obtains information including processing statistics (e.g., first metadata 440) and / or long-term statistics (e.g., second metadata 445), which in turn is used to guide processing components such as the recovery module 410, the rendering module 420 and / or the editing module.

[0100] The "restoration" group of processing components generates (restored) original audio from the enhanced audio with the help of contextual metadata 435 (e.g., first metadata 440). Therefore, the processing module 410 is configured to apply a restoration process to the audio data 405 using the contextual metadata 435. Specifically, the processing module 410 can use one or more processing parameters (e.g., as indicated by the first metadata 440) to at least partially reverse the previous frame-by-frame audio enhancement (as performed on the capture side). Thereby, the processing module 410 obtains (restored) original audio data 415, which can correspond to the audio data before the audio enhancement on the UGC capture side or an approximation of the audio data.

[0101] Specifically, the processing module 410 may be configured to apply at least one of the following to the audio data 405 : background sound restoration, loudness restoration, peak restoration, and timbre restoration.

[0102] To this end, the processing module 410 may include corresponding modules in a peak recovery module (for peak recovery), a loudness recovery module 414 (for loudness recovery), a noise management recovery module 416 (for background sound (ambience) recovery), and a timbre management recovery module (not shown; for timbre recovery). Among them, each recovery process can "mirror" the audio enhancement applied on the UGC capture side. Figure 1 The restoration process may be applied in the reverse order compared to that performed by the apparatus 100 shown in FIG. 1 . For example, the type and / or order of enhancement processing performed on the UGC capture side may be communicated using metadata 435, may be communicated using separate metadata, or may be previously agreed upon (e.g., in the context of standardization, etc.).

[0103] Peak restoration aims to restore over-suppressed peaks in the enhanced audio 405. Loudness restoration attempts to restore audio levels to original levels and remove distortion introduced by loudness management. Noise management restoration (background sound restoration) brings back sound events that are considered noise (e.g., engine noise) and leaves the decision to suppress or keep these events to later processing, or to content creators using editing tools. It should be understood that noise management / noise suppression on the UGC capture side can suppress background sound as noise, depending on the definitions of "noise" and "background sound". Restoring background sound may be worth doing, especially in cases where the suppressed sound is related to a soundscape, etc.

[0104] As described above, the restoration process is based on one or more processing parameters indicated by the metadata 435 (e.g., by the first metadata 440). As further noted above, the one or more processing parameters may include band gains (e.g., band gains for previous noise management and / or band gains for previous timbre management) and / or full-band gains (e.g., full-band gains for previous loudness management and / or full-band gains for previous peak limiting) applied during previous frame-by-frame audio enhancement. Knowledge of these gains enables any enhancement processing previously performed to be reversed based on these gains.

[0105] The rendering module 420 may be configured to apply frame-by-frame audio enhancement to the (restored) original audio data 415 to obtain enhanced audio data as rendered output 425. The "rendering" group of processing components may be associated with Figure 1 The same or Figure 3The same as in the example apparatus 300 (example rendering system) in , including noise management, loudness management, timbre management, and peak limiting. Therefore, the rendering module 420 can be configured to apply at least one of the following to the (restored) original audio data: noise management (e.g., through the noise management module 422), loudness management (e.g., through the loudness management module 424), timbre management (e.g., through the timbre management module 426), and peak limiting (e.g., through the peak limiting module 428).

[0106] The above-described process may be guided by additional information available in the long-term statistics of the contextual metadata 435. In other words, the rendering module 420 may be configured to apply frame-by-frame audio enhancement to the original audio data 415 based on the second metadata 445.

[0107] For example, given the additional information available in the long-term statistics of the contextual metadata 435 (e.g., indicated by the second metadata 445), noise management can adjust the noise suppression previously applied to the enhanced audio 405, such as to avoid certain over-suppression, preserve sound events, or further suppress certain types of noise in the enhanced audio. Given the additional information available in the long-term statistics of the contextual metadata 435, loudness management can level the enhanced audio 405 (or more precisely, the original audio 415) to a more appropriate range. Timbre management can rebalance the timbre of the audio based on content analysis (i.e., based on the long-term statistics of the contextual metadata).

[0108] Peak limiting can ensure that the audio amplitude after the aforementioned enhancement does not exceed a reasonable range allowed for audio playback. Alternatively, the restored original audio 415 obtained by the "restore" group processing can be exported to an editing tool, where some or all of the processing in the "render" group can be applied by the content creator using controls (e.g., via the editing tool UI), and additional processing that is not part of the "render" group can be applied. Therefore, the editing module can be a module for applying editing processing to the original audio data to obtain edited audio data. Similarly, the editing can be based on, for example, the second metadata 445.

[0109] Although an example apparatus 400 for UGC processing for rendering / editing has been described above, the present disclosure also relates to a corresponding method for UGC processing for rendering / editing. It should be understood that any statement above about the apparatus 400 also applies to the corresponding method, and vice versa. Figure 5 An example of such a UGC processing (ie, processing of audio data related to UGC) method 500 is illustrated in a flowchart of . The method 500 includes steps S510 to S540 and can be performed at a playback device (eg, a mobile device or a general computing device) or an editing device.

[0110] exist Step S510 , obtaining audio data. This may include or be equivalent to receiving a bit stream including audio data, including, for example, obtaining audio data from a storage medium.

[0111] exist Step S520 , obtaining metadata for the audio data. The metadata includes first metadata indicating one or more processing parameters of a previous frame-by-frame audio enhancement of the audio data. Obtaining the metadata may include or be equivalent to receiving a bitstream including the metadata (e.g., together with the audio data), including, for example, retrieving the metadata (e.g., together with the audio data) from a storage medium. Step S530 , applying a restoration process to the audio data using one or more processing parameters to at least partially reverse a previous frame-by-frame audio enhancement to obtain the original audio data. For example, applying the restoration process to the audio data may include applying at least one of the following: background sound restoration, loudness restoration, peak restoration, and timbre restoration. Thus, the one or more processing parameters may include a band gain (e.g., a band gain for a previous noise management and / or a band gain for a previous timbre management) and / or a full-band gain (e.g., a full-band gain for a previous loudness management and / or a full-band gain for a previous peak limiting) applied during a previous frame-by-frame audio enhancement.

[0112] This step can be performed according to the processing of the above-mentioned recovery module 410 (and its submodules).

[0113] exist Step S540 , applying frame-by-frame audio enhancement to original audio data to obtain enhanced audio data, and / or applying editing processing to original audio data to obtain edited audio data.

[0114] Here, applying frame-by-frame audio enhancement to the original audio data may be based on second metadata included in the metadata. As described above, the second metadata may indicate long-term statistics of the audio data and / or indicate one or more audio features of the audio data (e.g., the content type of the audio data, an indication of the capture environment of the audio data, the signal-to-noise ratio of the audio data before the previous frame-by-frame audio enhancement, the overall loudness of the audio data before the previous frame-by-frame audio enhancement, and / or the spectral shape of the audio data before the previous frame-by-frame audio enhancement, etc.).

[0115] Similar to Figure 2 The processing applied in step S220 of the illustrated method 200 , applying frame-by-frame audio enhancement to the original audio data may include applying at least one of the following: noise management, loudness management, peak limiting, and timbre management.

[0116] Step S540 may be performed according to the processing of the rendering module 420 (and its submodules) or the editing module.

[0117] Examples of methods and apparatuses for UGC processing according to embodiments of the present disclosure have been described above. It should be understood that these methods and apparatuses can be implemented by appropriate configuration of a computing device (eg, apparatus, system). Figure 6 A block diagram of an example of such a computing device 600 is schematically illustrated in . The computing device 600 includes a processor 610 and a memory 620 coupled to the processor 610. The memory 620 stores instructions for the processor 610. The processor 610 is configured to execute the steps of the methods described herein and / or implement the modules of the apparatuses described herein.

[0118] The present disclosure further relates to a computer program comprising instructions that, when executed by a computing device, cause the computing device (eg, general purpose computing device 600 ) to perform steps of the methods described herein and / or modules that implement the apparatuses described herein.

[0119] The present disclosure also relates to a computer-readable storage medium storing such a computer program.

[0120] explain

[0121] Aspects of the systems described herein can be implemented in an appropriate computer-based sound processing network environment (e.g., a server or cloud environment) to process digital or digitized audio files. Portions of the adaptive audio system may include one or more networks comprising any desired number of independent machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0122] One or more components, blocks, processes or other functional components may be implemented by a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described in terms of their behavior, register transfers, logic components and / or other features using hardware, firmware and / or any number of combinations of data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical storage media, magnetic storage media, or semiconductor storage media.

[0123] In particular, it should be understood that, for purposes of discussion, embodiments may include hardware, software, and electronic components or modules that are illustrated or described as if most components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, will recognize that in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored on a non-transitory computer-readable medium) that is executable by one or more electronic processors, such as a microprocessor and / or an application-specific integrated circuit (“ASIC”). Therefore, it should be noted that embodiments may be implemented using a plurality of software- and hardware-based devices and a plurality of different structural components. For example, the “content activity detector” described herein may include one or more electronic processors, one or more computer-readable media modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0124] Although one or more embodiments have been described by way of example and with respect to specific embodiments, it should be understood that one or more embodiments are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements that are obvious to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation so as to cover all such modifications and similar arrangements.

[0125] Likewise, it should be understood that the phraseology and terminology used herein is for descriptive purposes and should not be regarded as limiting. The use of "including," "comprising," or "having" and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless otherwise specified or limited, the terms "mounted," "connected," "supported," and "coupled" and variations thereof are used broadly and encompass direct and indirect mountings, connections, supports, and couplings.

[0126] Example embodiments of enumeration

[0127] Various aspects and embodiments of the present disclosure may also be understood from the following enumerated example embodiments (EEE), which are not claims.

[0128] EEE1. A method for processing audio data related to user-generated content, the method comprising: obtaining audio data; applying frame-by-frame audio enhancement to the audio data to obtain enhanced audio data; generating metadata for the enhanced audio data based on one or more processing parameters of the frame-by-frame audio enhancement; and outputting the enhanced audio data together with the generated metadata.

[0129] EEE2. A method according to EEE1, wherein applying frame-by-frame audio enhancement to audio data includes applying at least one of the following: noise management; loudness management; peak limiting; and timbre management.

[0130] EEE3. A method according to EEE1 or EEE2, wherein the one or more processing parameters include frequency band gains and / or full-band gains applied during frame-by-frame audio enhancement.

[0131] EEE4. A method according to EEE1 or EEE2, wherein the one or more processing parameters include at least one of the following: a frequency band gain for noise management; a full-band gain for loudness management; a full-band gain for peak limiting; and a frequency band gain for timbre management.

[0132] EEE5. A method according to any one of EEE1 to EEE4, wherein frame-by-frame audio enhancement is applied in real-time.

[0133] EEE6. A method according to any one of EEE1 to EEE5, wherein the metadata is further generated based on the results of analyzing multiple frames of audio data.

[0134] EEE7. A method according to EEE6, wherein analysis of multiple frames of audio data produces long-term statistics of the audio data.

[0135] EEE8. A method according to EEE6 or EEE7, wherein analysis of multiple frames of audio data produces one or more audio features of the audio data.

[0136] EEE9. A method according to EEE8, wherein the audio characteristics of the audio data relate to at least one of the following: a content type of the audio data; an indication of a capture environment of the audio data; a signal-to-noise ratio of the audio data; an overall loudness of the audio data; and a spectral shape of the audio data.

[0137] EEE10. A method according to any one of EEE6 to EEE9, wherein the metadata includes first metadata generated based on one or more processing parameters of frame-by-frame audio enhancement and second metadata generated based on the results of analyzing multiple frames of audio data; and the method also includes compiling the first metadata and the second metadata to obtain compiled metadata as metadata for output.

[0138] EEE11. A method for processing audio data associated with user-generated content, the method comprising: obtaining audio data; obtaining metadata of the audio data, wherein the metadata comprises first metadata indicating one or more processing parameters of previous frame-by-frame audio enhancement of the audio data; applying a recovery process to the audio data using the one or more processing parameters to at least partially reverse the previous frame-by-frame audio enhancement to obtain original audio data; and applying frame-by-frame audio enhancement to the original audio data to obtain enhanced audio data, or applying an editing process to the original audio data to obtain edited audio data.

[0139] EEE12. A method according to EEE11, wherein applying restoration processing to audio data includes applying at least one of the following: background sound restoration; loudness restoration; peak restoration; and timbre restoration.

[0140] EEE13. A method according to EEE11 or EEE12, wherein the one or more processing parameters include frequency band gains and / or full-band gains applied during a previous frame-by-frame audio enhancement.

[0141] EEE14. A method according to EEE11 or EEE12, wherein one or more processing parameters include at least one of the following: a frequency band gain of previous noise management; a full-band gain of previous loudness management; a full-band gain of previous peak limiting; and a frequency band gain of previous timbre management.

[0142] EEE15. A method according to any one of EEE11 to EEE14, wherein the metadata further includes second metadata indicating long-term statistics of the audio data and / or indicating one or more audio features of the audio data. EEE16. A method according to EEE15, wherein the audio features of the audio data relate to at least one of the following: a content type of the audio data; an indication of a capture environment of the audio data; a signal-to-noise ratio of the audio data before a previous frame-by-frame audio enhancement; an overall loudness of the audio data before a previous frame-by-frame audio enhancement; and a spectral shape of the audio data before a previous frame-by-frame audio enhancement.

[0143] EEE17. A method according to EEE15 or EEE16, wherein applying frame-by-frame audio enhancement to the original audio data is based on second metadata.

[0144] EEE18. A method according to any one of EEE11 to EEE17, wherein applying frame-by-frame audio enhancement to the original audio data includes applying at least one of the following: noise management; loudness management; peak limiting; and timbre management.

[0145] EEE19. A device for processing audio data related to user-generated content, the device comprising: a processing module for applying frame-by-frame audio enhancement to the audio data to obtain enhanced audio data, and for outputting the enhanced audio data; and an analysis module for generating metadata of the enhanced audio data based on one or more processing parameters of the frame-by-frame audio enhancement, and for outputting the metadata.

[0146] EEE20. An apparatus according to EEE19, wherein the processing module is configured to apply at least one of the following to the audio data: noise management; loudness management; peak limiting; and timbre management.

[0147] EEE21. An apparatus according to EEE19 or EEE20, wherein the one or more processing parameters include frequency band gains and / or full-band gains applied during frame-by-frame audio enhancement.

[0148] EEE22. An apparatus according to EEE19 or EEE20, wherein the one or more processing parameters include at least one of the following: a frequency band gain for noise management; a full-band gain for loudness management; a full-band gain for peak limiting; and a frequency band gain for timbre management.

[0149] EEE23. An apparatus according to any one of EEE19 to EEE22, wherein the processing module is configured to apply frame-by-frame audio enhancement in real time.

[0150] EEE24. An apparatus according to any one of EEE19 to EEE23, wherein the analysis module is configured to generate metadata further based on the results of analyzing multiple frames of audio data.

[0151] EEE25. An apparatus according to EEE24, wherein analysis of multiple frames of audio data produces long-term statistics of the audio data.

[0152] EEE26. An apparatus according to EEE24 or EEE25, wherein analysis of multiple frames of audio data produces one or more audio features of the audio data.

[0153] EEE27. An apparatus according to EEE26, wherein the audio characteristics of the audio data relate to at least one of: a content type of the audio data; an indication of a capture environment of the audio data; a signal-to-noise ratio of the audio data; an overall loudness of the audio data; and a spectral shape of the audio data. EEE27.

[0154] EEE28. An apparatus according to any one of EEE24 to EEE27, wherein the analysis module is configured to generate first metadata based on one or more processing parameters of frame-by-frame audio enhancement and to generate second metadata based on the results of analyzing multiple frames of audio data; and the analysis module is further configured to compile the first metadata and the second metadata to obtain the compiled metadata as metadata for output.

[0155] EEE29. A device for processing audio data associated with user-generated content, the device comprising: an input module for receiving audio data and metadata of the audio data, wherein the metadata comprises first metadata indicating one or more processing parameters of previous frame-by-frame audio enhancement of the audio data; a processing module for applying a recovery process to the audio data using the one or more processing parameters to at least partially reverse the previous frame-by-frame audio enhancement to obtain original audio data; and at least one of a rendering module and an editing module, wherein the rendering module is a module for applying frame-by-frame audio enhancement to the original audio data to obtain enhanced audio data, and the editing module is a module for applying editing processing to the original audio data to obtain edited audio data.

[0156] EEE30. An apparatus according to EEE29, wherein the processing module is configured to apply at least one of the following to the audio data: background sound restoration; loudness restoration; peak restoration; and timbre restoration.

[0157] EEE31. An apparatus according to EEE29 or EEE30, wherein the one or more processing parameters include frequency band gains and / or full-band gains applied during a previous frame-by-frame audio enhancement.

[0158] EEE32. An apparatus according to EEE29 or EEE30, wherein the one or more processing parameters include at least one of: a frequency band gain of previous noise management; a full-band gain of previous loudness management; a full-band gain of previous peak limiting; and a frequency band gain of previous timbre management.

[0159] EEE33. An apparatus according to any one of EEE29 to EEE32, wherein the metadata further includes second metadata indicating long-term statistics of the audio data and / or indicating one or more audio features of the audio data. EEE34. An apparatus according to EEE33, wherein the audio features of the audio data relate to at least one of the following: a content type of the audio data; an indication of a capture environment of the audio data; a signal-to-noise ratio of the audio data before a previous frame-by-frame audio enhancement; an overall loudness of the audio data before a previous frame-by-frame audio enhancement; and a spectral shape of the audio data before a previous frame-by-frame audio enhancement.

[0160] EEE35. An apparatus according to EEE33 or EEE34, wherein the rendering module is configured to apply frame-by-frame audio enhancement to the original audio data based on the second metadata.

[0161] EEE36. An apparatus according to any one of EEE29 to EEE35, wherein the rendering module is configured to apply at least one of the following to the original audio data: noise management; loudness management; peak limiting; and timbre management. EEE37. An apparatus for processing audio data associated with user-generated content, the apparatus comprising a processor and a memory, the memory being coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of EEE1 to EEE18.

[0162] EEE38. A computer program comprising instructions which, when executed by a computing device, cause the computing device to perform all the steps of the method according to any one of EEE1 to EEE18.

[0163] EEE39. A computer-readable storage medium storing the computer program according to EEE38.

Claims

1. A method for processing audio data associated with user-generated content, the method comprising: Obtaining the audio data; applying frame-by-frame audio enhancement to the audio data to obtain enhanced audio data; generating metadata for the enhanced audio data based on one or more processing parameters of the frame-by-frame audio enhancement; as well as The enhanced audio data is output together with the generated metadata.

2. The method according to claim 1, wherein: Applying the frame-by-frame audio enhancement to the audio data includes applying at least one of: Noise management; Loudness management; Peak limiting; and Tone management.

3. The method according to claim 1 or 2, wherein: The one or more processing parameters include frequency band gains and / or full-band gains applied during the frame-by-frame audio enhancement.

4. The method according to claim 1 or 2, wherein: The one or more processing parameters include at least one of the following: frequency band gain for noise management; full-band gain for loudness management; Full-band gain for peak limiting; and Band gain for tone management.

5. A method according to any one of the preceding claims, wherein: The frame-by-frame audio enhancement is applied in real-time.

6. A method according to any one of the preceding claims, wherein: The metadata is further generated based on results of an analysis of the plurality of frames of the audio data.

7. The method according to claim 6, wherein: The analysis of the plurality of frames of the audio data produces long-term statistics of the audio data.

8. The method according to claim 6 or 7, wherein: The analysis of the plurality of frames of the audio data produces one or more audio features of the audio data.

9. The method according to claim 8, wherein: The audio feature of the audio data relates to at least one of the following: The content type of the audio data; an indication of a capture environment of the audio data; A signal-to-noise ratio of the audio data; The overall loudness of the audio data; and The spectral shape of the audio data.

10. The method according to any one of claims 6 to 9, wherein: The metadata includes first metadata and second metadata, the first metadata is generated based on the one or more processing parameters of the frame-by-frame audio enhancement, and the second metadata is generated based on a result of analyzing a plurality of frames of the audio data; as well as The method also includes compiling the first metadata and the second metadata to obtain compiled metadata as the metadata for output.

11. A method of processing audio data associated with user-generated content, the method comprising: Obtaining the audio data; Obtaining metadata of the audio data, wherein the metadata includes first metadata indicating one or more processing parameters of a previous frame-by-frame audio enhancement of the audio data; applying a restoration process to the audio data using the one or more processing parameters to at least partially reverse the previous frame-by-frame audio enhancement to obtain original audio data; and A frame-by-frame audio enhancement is applied to the original audio data to obtain enhanced audio data, or an editing process is applied to the original audio data to obtain edited audio data.

12. The method according to claim 11, wherein: Applying the restoration process to the audio data includes applying at least one of: Background sound restoration; Loudness restoration; Peak recovery; and The sound is restored.

13. The method according to claim 11 or 12, wherein: The one or more processing parameters include frequency band gains and / or full-band gains applied during the previous frame-by-frame audio enhancement.

14. The method according to claim 11 or 12, wherein: The one or more processing parameters include at least one of the following: Band gain for previous noise management; The previous loudness management full-band gain; Full-band gain from previous peak limiting; and Band gain for previous tone management.

15. The method according to any one of claims 11 to 14, wherein: The metadata further includes second metadata indicating long-term statistics of the audio data and / or indicating one or more audio features of the audio data.

16. The method according to claim 15, wherein: The audio feature of the audio data relates to at least one of the following: The content type of the audio data; an indication of a capture environment of the audio data; a signal-to-noise ratio of the audio data prior to the previous frame-by-frame audio enhancement; an overall loudness of the audio data prior to the previous frame-by-frame audio enhancement; as well as The spectral shape of the audio data prior to the previous frame-by-frame audio enhancement.

17. The method according to claim 15 or 16, wherein: Applying the frame-by-frame audio enhancement to the original audio data is based on the second metadata.

18. The method according to any one of claims 11 to 17, wherein: Applying the frame-by-frame audio enhancement to the original audio data includes applying at least one of the following: Noise management; Loudness management; Peak limiting; and Tone management.

19. An apparatus for processing audio data associated with user-generated content, the apparatus comprising: a processing module for applying frame-by-frame audio enhancement to the audio data to obtain enhanced audio data, and for outputting the enhanced audio data; as well as An analysis module is used to generate metadata of the enhanced audio data based on one or more processing parameters of the frame-by-frame audio enhancement, and to output the metadata.

20. The device according to claim 19, wherein The processing module is configured to apply at least one of the following to the audio data: Noise management; Loudness management; Peak limiting; and Tone management.

21. The device according to claim 19 or 20, wherein: The one or more processing parameters include frequency band gains and / or full-band gains applied during the frame-by-frame audio enhancement.

22. The device according to claim 19 or 20, wherein: The one or more processing parameters include at least one of the following: frequency band gain for noise management; full-band gain for loudness management; Full-band gain for peak limiting; and Band gain for tone management.

23. The device according to any one of claims 19 to 22, wherein: The processing module is configured to apply frame-by-frame audio enhancement in real-time.

24. The device according to any one of claims 19 to 23, wherein: The analysis module is configured to generate the metadata further based on a result of analyzing a plurality of frames of the audio data.

25. The device according to claim 24, wherein: The analysis of the plurality of frames of the audio data produces long-term statistics of the audio data.

26. The device according to claim 24 or 25, wherein: The analysis of the plurality of frames of the audio data produces one or more audio features of the audio data.

27. The device according to claim 26, wherein: The audio feature of the audio data relates to at least one of the following: The content type of the audio data; an indication of a capture environment of the audio data; A signal-to-noise ratio of the audio data; The overall loudness of the audio data; and The spectral shape of the audio data.

28. The device according to any one of claims 24 to 27, wherein The analysis module is configured to generate first metadata based on the one or more processing parameters of the frame-by-frame audio enhancement, and generate second metadata based on a result of analyzing the plurality of frames of the audio data; as well as The analysis module is further configured to compile the first metadata and the second metadata, thereby obtaining compiled metadata as the metadata for output.

29. An apparatus for processing audio data associated with user-generated content, the apparatus comprising: An input module for receiving audio data and metadata of the audio data, wherein the metadata includes first metadata indicating one or more processing parameters of a previous frame-by-frame audio enhancement of the audio data; a processing module for applying a restoration process to the audio data using the one or more processing parameters to at least partially reverse the previous frame-by-frame audio enhancement to obtain original audio data; and at least one of a rendering module and an editing module, The rendering module is a module for applying frame-by-frame audio enhancement to the original audio data to obtain enhanced audio data, and the editing module is a module for applying editing processing to the original audio data to obtain edited audio data.

30. The device according to claim 29, wherein: The processing module is configured to apply at least one of the following to the audio data: Background sound restoration; Loudness restoration; Peak recovery; and The sound is restored.

31. The device according to claim 29 or 30, wherein: The one or more processing parameters include frequency band gains and / or full-band gains applied during the previous frame-by-frame audio enhancement.

32. The device according to claim 29 or 30, wherein: The one or more processing parameters include at least one of the following: Band gain for previous noise management; The previous loudness management full-band gain; Full-band gain from previous peak limiting; and Band gain for previous tone management.

33. The device according to any one of claims 29 to 32, wherein: The metadata further includes second metadata indicating long-term statistics of the audio data and / or indicating one or more audio features of the audio data.

34. The device according to claim 33, wherein The audio feature of the audio data relates to at least one of the following: The content type of the audio data; an indication of a capture environment of the audio data; a signal-to-noise ratio of the audio data prior to the previous frame-by-frame audio enhancement; an overall loudness of the audio data prior to the previous frame-by-frame audio enhancement; as well as The spectral shape of the audio data prior to the previous frame-by-frame audio enhancement.

35. The device according to claim 33 or 34, wherein: The rendering module is configured to apply the frame-by-frame audio enhancement to the original audio data based on the second metadata.

36. The device according to any one of claims 29 to 35, wherein: The rendering module is configured to apply at least one of the following to the raw audio data: Noise management; Loudness management; Peak limiting; and Tone management.

37. An apparatus for processing audio data associated with user-generated content, the apparatus comprising a processor and a memory, the memory being coupled to the processor and storing instructions for the processor, wherein: The processor is configured to perform all the steps of the method according to any one of claims 1 to 18.

38. A computer program comprising instructions which, when executed by a computing device, cause the computing device to perform all the steps of the method according to any one of claims 1 to 18.

39. A computer-readable storage medium storing the computer program according to claim 38.

Images