Audio object separation and audio processing

Abstract

A method for separating audio objects in a mixed audio signal comprising a plurality of audio objects is disclosed. Further disclosed are a computer-implemented method for training a sparse audio object separation model and a method for separating sparse audio objects from a mixed audio signal comprising at least a sparse audio object, a non-sparse audio object and at least one further audio object. Further disclosed is a computer-implemented method for processing audio based on signal-to-noise ratio (SNR) and a computer-implemented method for processing audio based on scene environment classification. A non-transitory computer readable medium and a system configured to perform one or more of the methods are disclosed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to PCT Application No. PCT / CN2022 / 114613 filed on August 24, 2022, U.S. Provisional Application No. 63 / 512,830 filed on July 11, 2023, and U.S. Provisional Application No. 63 / 513,066, all of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to methods for audio object separation, methods for training sparse audio object separation models, methods for processing audio, and non-transitory computer-readable media and systems. Background Art

[0004] In an audio signal, there are usually multiple audio objects. However, in some instances, only a subset of the audio objects in the audio signal is desired. As an example only, in an audio stream that includes speech as an object, there may usually be multiple other objects, such as weather sounds or weather noises (e.g., rain, thunder, wind, etc.), animal sounds (e.g., birds chirping, dogs barking, etc.). In some instances, not all of these audio objects in the audio signal are desired, or it may be desirable to process various audio objects in the audio stream in different ways.

[0005] Audio object separation (i.e., separating objects in an audio signal) is typically used to separate various audio objects in an audio signal and / or to extract certain objects from an audio stream. Some audio object separation methods and systems typically rely on models and may employ neural network architectures, machine learning, deep learning architectures, and the like. Summary of the invention

[0006] The purpose of the present disclosure is to improve the robustness and performance of audio object separation (such as deep learning based audio object separation).

[0007] According to a first aspect of the present disclosure, a method for separating audio objects in a mixed audio signal is provided. The mixed audio signal includes a plurality of audio objects. The method includes: receiving the mixed audio signal by a multi-object separation model; separating one or more audio objects of the plurality of audio objects of the mixed audio signal by the multi-object separation model; and outputting an output signal including the one or more separated audio objects. The model includes a plurality of sub-models, each of the plurality of sub-models being trained to determine and output a corresponding one of the plurality of audio objects.

[0008] Each of the sub-models may be trained by using a corresponding dataset in a plurality of datasets. Each dataset in the plurality of datasets may include a corresponding set of data pairs. Each data pair may include a signal having the corresponding one object and a mixed signal including the corresponding one object and at least one other signal. The plurality of sub-models may include a feature extractor. Each of the plurality of sub-models may include one or more layers configured to map from a feature extraction signal to a corresponding one object. Separating the one or more audio objects may include: receiving the mixed audio signal by the feature extractor; outputting a feature signal including one or more features from the mixed audio signal by the feature extractor; and receiving the feature signal by the one or more layers of each of the plurality of sub-models. The method may also include: adding metadata by the model, the metadata indicating the position of the model layer in the model, the layer type of the model layer and / or the pointer to the backward layer and / or the forward layer of the model; obtaining data indicating that the separation quality of a corresponding one of the multiple objects is lower than a predetermined quality threshold; obtaining a subsequent training data set including a corresponding set of data pairs, each data pair including a signal with the corresponding object, and a mixed signal including the corresponding object and at least one other signal; determining one or more layers of the model for separating the corresponding one object based on the metadata; training the one or more layers of the model using the subsequent training data set, wherein the training includes freezing the remaining layers of the model so that only the determined one or more layers are trained using the subsequent data set.

[0009] According to a second aspect of the present disclosure, a computer-implemented method for training a sparse audio object separation model is provided. The method comprises: obtaining a training audio signal, the training audio signal comprising a sparse audio object, a non-sparse audio object, and at least one other audio object; and training the model to separate a combination of the sparse audio object and the non-sparse audio object from the training audio signal.

[0010] According to a third aspect of the present disclosure, a computer-implemented method for separating sparse audio objects from a mixed audio signal is provided. The mixed audio signal includes at least a sparse audio object, a non-sparse audio object, and at least one other audio object. The method includes: providing a sparse object separation model and training the sparse audio object separation model according to the second aspect of the present disclosure; generating a first separation signal by separating a combination of the sparse audio object and the non-sparse audio object from the mixed audio signal by the sparse audio object separation model; generating a second separation signal by separating the non-sparse audio object from the mixed audio signal by a non-sparse audio object separation model trained to separate non-sparse audio objects from the mixed audio signal; and generating an output signal by subtracting the second separation signal from the first separation signal.

[0011] The sparse audio object separation model of the second aspect and / or the third aspect may be a multi-object separation model (such as the multi-object separation model according to the first aspect) or a portion thereof (such as a sub-model thereof). Alternatively or additionally, the non-sparse audio object separation model may be a multi-object separation model according to the first aspect and / or a portion thereof (such as a sub-model thereof).

[0012] The sparse audio object may have a shorter duration than the non-sparse audio object and / or have a narrower frequency range than the non-sparse audio object. Alternatively or additionally, the sparse audio object may be one or more of an animal sound (such as a bird chirping or a dog barking), thunder, and a percussive sound. Alternatively or additionally, the non-sparse audio object may be one or more of speech, wind noise, and rain.

[0013] According to a fourth aspect of the present disclosure, a computer-implemented method for processing audio based on a signal-to-noise ratio (SNR) is provided. The method comprises: receiving an object-separated input signal including a plurality of audio objects; determining the corresponding SNR of one or more of the plurality of audio objects of the object-separated input signal; determining that one or more of the plurality of audio objects are dominant audio objects in response to determining that one or more of the signal-to-noise ratios are higher than a predetermined first SNR threshold of the corresponding object, and / or determining that one or more of the plurality of audio objects are background audio objects in response to determining that one or more of the signal-to-noise ratios are lower than a predetermined second SNR threshold of the corresponding object; generating a leakage-reduced output signal by reducing leakage from the input signal. Reducing leakage from the input signal comprises reducing one or more non-dominant audio objects and / or reducing the one or more background objects in the input signal.

[0014] The input signal separated by objects can be obtained by a sparse audio object separation model (such as the sparse object separation model according to the second aspect and / or the third aspect) and / or by a multi-object separation model (such as the multi-object separation model according to the first aspect).

[0015] The first SNR threshold and / or the second SNR threshold for each object may be determined based on the type of object and / or based on the characteristics of the object. The first SNR threshold and / or the second SNR threshold for each object may be different thresholds.

[0016] According to a fifth aspect of the present disclosure, a computer-implemented method for processing audio based on scene environment classification is provided. The method comprises: receiving an input signal separated by objects including a plurality of audio objects; determining a scene environment by obtaining a classification of a scene environment in which the audio objects are recorded, wherein the classification of the scene environment comprises classifying the scene environment into a corresponding scene environment of a plurality of scene environments based on audio and / or video information; and outputting a leakage-reduced output signal by reducing leakage from the input signal based on the determined scene environment.

[0017] Reducing leakage may include mixing the audio objects based on the determined scene environment. As an example, mixing the audio objects may include adjusting the weight of one or more audio objects in the plurality of audio objects. The weight may be determined based on the determined scene environment.

[0018] Obtaining the classification of the scene environment may include classifying the scene environment by a scene environment classifier. The scene environment classifier may be trained to determine the scene environment based on an audio signal and / or a visual signal such as a video signal. The input signal may be an output signal of the method according to any one of the first aspect, the second aspect or the third aspect.

[0019] According to a sixth aspect of the present disclosure, a non-transitory computer-readable medium storing instructions is provided, wherein when the instructions are executed by a processing unit, the one or more processing units execute a method according to any one of the first to fifth aspects.

[0020] According to the seventh aspect of the present disclosure, a system is provided, comprising: a processing unit; a non-transitory computer-readable medium, wherein the non-transitory computer-readable medium stores instructions, and when the instructions are executed by the processing unit, the processing unit executes a method according to any one of the first to fifth aspects.

[0021] It should be understood that any advantages described with respect to one aspect may also be applicable to any of the other aspects of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings, in which:

[0023] Figure 1 A schematic flow chart showing an example of a method for separating audio objects in a mixed audio signal according to the present disclosure,

[0024] Figure 2A A schematic block diagram showing an example of a multi-object separation model during a training phase of the multi-object separation model is shown,

[0025] Figure 2B A schematic block diagram showing an example of a multi-object separation model,

[0026] Figure 3 4 is a schematic block diagram showing another example of a multi-object separation model during a training phase of the multi-object separation model,

[0027] Figure 4 A schematic flow chart showing an example of a method for training a sparse audio object separation model,

[0028] Figure 5 A schematic flow chart showing an example of a method for separating sparse audio objects from a mixed audio signal,

[0029] Figure 6 showing an exemplary spectrogram illustrating examples of sparse audio objects and non-sparse audio objects,

[0030] Figure 7 A schematic flow chart showing an example of a method for processing audio based on a signal-to-noise ratio SNR,

[0031] Figure 8 A schematic flow chart showing an example of a computer-implemented method for processing audio based on scene environment classification, and

[0032] Fig. 9 A schematic block diagram of a system configured to perform the method according to the present disclosure is shown. DETAILED DESCRIPTION

[0033] Figure 1 A schematic block diagram of an example of a method 1 for separating audio objects in a mixed audio signal according to the present disclosure is shown. The mixed audio signal includes a plurality of audio objects. The method includes: receiving 10 a mixed audio signal by a multi-object separation model; separating 11 one or more audio objects from the plurality of audio objects in the mixed audio signal by the multi-object separation model; and outputting 12 an output signal including the one or more separated audio objects. Figure 1In the method, the model includes multiple sub-models, and each of the multiple sub-models is trained to determine and output a corresponding one of the multiple audio objects.

[0034] By training each of the sub-models to determine and output a corresponding one of the plurality of objects, the performance and robustness of audio object separation may be improved. For example, improved audio object separation may be provided.

[0035] Throughout this specification, it should be understood that separated objects and signals separated by objects refer to estimated objects and signals including the estimated objects separated by sub-models and / or their elements. Correspondingly, such separated objects and signals separated by objects do not necessarily mean perfect separation of objects, because sub-models and / or their elements cannot always achieve or achieve such perfect separation.

[0036] Throughout this specification, the terms "object" and "audio object" may be used interchangeably, and both terms refer to audio objects.

[0037] As used herein, the term "audio object" may refer to an audio data stream and associated metadata that can be created or "authored" without reference to any particular playback environment. Metadata may indicate the 3D position of an object, rendering constraints, and content type (e.g., dialogue, effects, etc.). Depending on the implementation, metadata may alternatively or additionally include other types of data, such as width data, gain data, trajectory data, object position data, audio object gain data, audio object size data, etc. Some audio objects may be static (i.e., stationary), while other audio objects may be dynamic (i.e., moving). The details of an audio object may be created or rendered based on associated metadata, which may indicate, among other things, the position of an audio object in three-dimensional space at a given point in time. When an audio object is monitored or played back in a reproduction environment, the audio object may be rendered based on position metadata using the reproduction speakers present in the reproduction environment, rather than outputting the audio object to a predetermined physical channel as in a conventional channel-based system (e.g., Dolby 5.1 and Dolby 7.1). The rendering process may involve calculating a set of audio object gain values ​​for each channel in a set of output channels. Each output channel can correspond to one or more reproduction speakers of the reproduction environment. In this way, content placed on the screen can be imaged in substantially the same way as channel-based content, while content placed in surround sound can be rendered to separate speakers as needed.

[0038] In some examples, each audio object contains audio data from a specific audio source or combination of audio sources, i.e., recorded audio data of a specific audio source or combination of audio sources. Examples of such audio sources are speech, wind, animal sounds, musical instruments, rain, etc. For example, an audio object may include audio data from a person speaking, i.e., speech; in the present disclosure, such an audio object may be represented as a speech audio object. Each audio object may be stored as a corresponding audio file.

[0039] In the present disclosure, it should be understood that the term "model" throughout the present disclosure can also refer to a data architecture that is trained to perform the functions of the model. As an example, a multi-object separation model can be implemented by a data architecture (such as a deep learning model or a machine learning data architecture) that is trained to separate objects from a mixed audio signal including multiple objects.

[0040] A submodel may be a machine learning data architecture or a neural network architecture and may include any number of layers or nodes. Throughout this disclosure, a submodel may refer to a part or portion of a multi-object separation model. The part may be smaller than the entire model, i.e., a multi-object separation model may include or be composed of multiple submodels.

[0041] In some embodiments, outputting 12 the output signal may include outputting a plurality of output signals, each output signal including or consisting of a respective separated audio object.

[0042] Throughout this disclosure, it should be understood that the model is generally trained to determine and output a corresponding type or classification of audio objects. For example, such a type or classification of audio objects may be speech, wind noise, rain noise, dog barking, bird chirping, etc. Correspondingly, each of the sub-models may be trained to determine or separate a corresponding type of object from a mixed signal including multiple audio objects (which may be multiple audio objects of different types) and output the corresponding type of object.

[0043] Method 1 may be a computer-implemented method.

[0044] Each sub-model in the sub-model can be trained by using a corresponding data set in a plurality of data sets. Each data set in the plurality of data sets includes a corresponding set of data pairs. Each data pair may include a signal having the corresponding one object and a mixed signal including the corresponding one object and at least one other signal. The set of data pairs may include a plurality of data pairs.

[0045] The corresponding one of the objects in each data pair may be a corresponding one of the same object type or category (i.e., the sub-model is trained to determine or separate and output the object type or category from a mixed signal comprising multiple audio objects (which may be multiple audio objects of different types)).

[0046] Each of the data pairs may include two or more audio files or signals in a common object-based audio file format (such as a Dolby Atmos format). The audio files in the two or more audio files may only have audio corresponding to a corresponding object or a corresponding object type or category that the corresponding sub-model is to be trained to separate. Such an object or object type may be referred to as an expected object or an expected object type for the corresponding sub-model, respectively. Another audio file or signal in the two or more audio files may have audio corresponding to a corresponding object and other objects, noise, etc. Other objects, noise, etc. may be objects that the sub-model in the audio file is to be trained to separate from the corresponding expected object. The data set may include multiple data pairs.

[0047] In one example, each sub-model is trained with a data set comprising at least fifty data pairs or at least one hundred data pairs, each data pair comprising a signal consisting of an object of an expected object or an object type of an expected object, and a mixed signal comprising the object and at least one other object. In an example, the object of each data pair is separated by the sub-model and marked by an operator, and optionally checked. The label provided by the operator can be stored in the metadata of each object and / or the label can be provided to the model for training. In some examples, an operator (such as a listener) provides feedback to the sub-model on the success of separating the desired object from the mixed signal to train the sub-model. In other examples, each sub-model is trained additionally or alternatively using an audio quality metric to separate the corresponding object.

[0048] The term "sub-model" may be used herein to refer to a segment of an object separation model that is or can be trained separately. For example, each sub-model may be and / or may include a corresponding neural network model, a machine learning data architecture, and / or an artificial intelligence model.

[0049] Figure 2A A schematic block diagram of an example of a multi-object separation model 2 during a training phase of the multi-object separation model is shown. Figure 2B The multi-object separation model 2 is shown during operation (ie, after the training phase).

[0050] The multi-object separation model 2 includes a plurality of sub-models, including a first sub-model, a second sub-model, and a third sub-model 20a, 20b, 20c, each of which is trained to determine and output a corresponding one object 21a, 21b, 21c. Each of the sub-models 20a, 20b, 20c can be trained separately using a corresponding first training data set, a second training data set, and a third training data set TDa, TDb, TDc. Each of the training data sets TDa, TDb, TDc can include a corresponding set of data pairs, each of which potentially includes a signal of an object having a corresponding one object or a corresponding object type that the sub-model is to be trained to separate, and a mixed signal including a corresponding one object and at least one other signal.

[0051] For example, the first training data set TDa may include a set of data pairs, each data pair including a signal having a first audio object (such as only having the first audio object), and an audio signal having the first audio object and at least one other audio object or noise (also referred to as a mixed audio signal). Thus, the first sub-model 20a can be trained to separate the first object from the mixed audio signal and output the separated first object 21a. Similarly, the second training data set and the third training data set TDb, TDc can each include a set of data pairs. Each data pair of the second training data set TDb may include a signal having a second audio object and a mixed audio signal having a second audio object and at least one other audio object, which is used to train the second sub-model 20b to separate the second object from the mixed audio signal and output the separated second object 21b. Correspondingly, each data pair of the second training data set TDc may include a signal having a third audio object and a mixed audio signal having a third audio object and at least one other audio object, which is used to train the third sub-model 20c to separate the second object from the mixed audio signal and output the separated second object 21c.

[0052] After training, if Figure 2B As shown, in the example, an input audio signal In including multiple objects is applied to a multi-object separation model 2. The multiple audio objects may include at least one of a first audio object, a second audio object, and a third audio object. In this example, each sub-model 20a, 20b, 20c receives the input audio signal In. Then, Figure 2B In the example illustrated in FIG. 2 , each sub-model 20 a , 20 b , 20 c separates the first audio object, the second audio object, and the third audio object, respectively, and outputs the corresponding separated first audio object 21 a , second audio object 21 b , and third audio object.

[0053] The multi-object separation model 2 may be the multi-object separation model used in method 1.

[0054] exist Figure 2A and Figure 2B In the example of the multi-object separation model 2 illustrated in , three sub-models are illustrated. However, in other examples, the multi-object separation model 2 may include fewer (such as two) or more (such as 5, 8, 10 or more) sub-models, each of which is trained to separate audio objects from mixed audio signals. Correspondingly, a corresponding data set for training each sub-model may be provided during the training phase.

[0055] Alternatively, the plurality of sub-models include a feature extractor and a plurality of layers configured to map from a feature extraction signal to a corresponding one of the objects. Separating the one or more audio objects may include: receiving a mixed audio signal by the feature extractor; outputting a feature signal including one or more features from the mixed audio signal by the feature extractor; and receiving the feature signal by each of the plurality of layers. Separating the one or more audio objects may also include outputting a corresponding object-separated signal including the corresponding separated object by each of the plurality of layers.

[0056] The feature extractor may be a common feature extractor, such as a feature extractor shared by the plurality of sub-models. Correspondingly, the feature extraction signal may be a feature extraction signal shared by the plurality of sub-models. Alternatively or additionally, the common feature extractor may be configured and / or trained to extract features related to the respective object to which each sub-model is configured to map from the feature extraction signal.

[0057] In some examples, the feature extractor is configured to obtain features from the input audio signal and / or combine variables from the input audio signal and / or select variables from the input audio signal and combine these variables into features. Such features may be included in the feature extraction signal and may be determined based on the input audio signal. Alternatively or additionally, the feature may relate to, represent and / or be a reduced-dimensional input signal. The feature may be or may include multiple values ​​and may optionally be a feature set and / or a feature vector. Compared to the input audio signal, the feature extraction signal may have a reduced dimension and / or a reduced amount of data. Correspondingly, the feature extraction signal may be considered as a summary of the input audio signal. The feature extraction signal may be and / or may include features extracted from the input audio signal by the feature extractor. In an example, the feature extractor may be and / or may include an autoencoder or a similar known type of feature extractor configured to extract features from a signal.

[0058] The feature extracted signal may be an output signal from a feature extractor. The feature extracted signal may include a plurality of features from a signal. In some examples, the feature extractor is configured to extract features from the signal that are indicative of an audio object in an input audio signal.

[0059] In some examples, the plurality of sub-models are composed of a feature extractor and a plurality of additional layers, each additional layer being configured to map from a feature extraction signal to a corresponding audio object. The feature extractor and the additional layers may be trained together or may be trained separately.

[0060] Where the multiple sub-models include and / or are implemented as a feature extractor and multiple additional layers, a simple implementation may be provided to reduce complexity and therefore processing time, since the feature extractor may be common to all objects to be separated and only one or more additional layers may be required to obtain each object. Secondly, this allows the object separation method to be implemented in devices with limited processing power and / or low power devices. Examples of such devices may be head mounted devices, such as wireless head mounted devices, ear buds, etc.

[0061] In some examples, each of the plurality of sub-models includes one or more layers configured to map from a feature extraction signal to a corresponding one of the objects.

[0062] In other examples, the multi-object separation model may include a feature extractor configured to extract features from an input audio signal and output a feature extraction signal. Each of the sub-models of the multi-object separation model may be one or more layers configured to map from the feature extraction signal to a corresponding object. Separating the one or more audio objects may include: receiving a mixed audio signal by the feature extractor; outputting a feature signal including one or more features from the mixed audio signal by the feature extractor; and receiving the feature signal by the one or more layers of each of the multiple sub-models.

[0063] The one or more layers configured to map from the feature extraction signal to the corresponding one of the objects may be represented as additional layers. A layer may be understood herein in a well-known manner as a layer in a machine learning, neural network or artificial intelligence data structure or model. Thus, a layer may, for example, refer to a node layer in a neural network data structure or model.

[0064] Figure 3 A schematic block diagram of a multi-object separation model 2′ during the training phase of the multi-object separation model 2′ is shown. The multi-object separation model 2′ includes a plurality of sub-models having a common feature extractor 22. The feature extractor 22 is configured to extract features from an input signal input into the feature extractor 22 and output a feature extracted signal 23, i.e., a signal including features extracted by the feature extractor 22 from the input signal.

[0065] Each of the plurality of sub-models further comprises an additional layer 24a, 24b, 24c for each object to be separated. Each of the first additional layer 24a, the second additional layer 24b, and the third additional layer 24c is trained and / or configured to separate the corresponding first audio object, the second audio object, and the third audio object from the feature extraction signal 23. Each separated corresponding audio object is output from each additional layer 24a, 24b, 24c as an output signal 21a, 21b, 21c, respectively.

[0066] exist Figure 3 , the multi-object separation model 2′ is illustrated in a training phase in which a training dataset TD provides an input signal to a common feature extractor 22. When in operation (i.e., not in training mode), the training dataset TD may be replaced by an input audio signal (such as the input audio signal In) comprising a plurality of objects.

[0067] The training dataset TD may include multiple datasets for each of the objects to be separated by the multi-object separation model 2′ as described with respect to method 1. Alternatively or additionally, the training dataset may include a first training dataset, a second training dataset, and a third training dataset TDa, TDb, TDc and / or datasets or data pairs similar thereto.

[0068] Although Figure 3 The multi-object separation model 2 ′ is illustrated as having three additional layers 24 a , 24 b , 24 c , but it should be understood that the multi-object separation model 2 ′ may include any number of additional layers 24 a , 24 b , 24 c and / or sub-models. The multi-object separation model 2 ′ may be used in method 1 .

[0069] In some examples, the method also includes: adding metadata by the model, the metadata indicating the location of a model layer in the model, the layer type of the model layer and / or a pointer to a backward layer and / or a forward layer of the model; obtaining data indicating that the separation quality of a corresponding one of the multiple objects is lower than a predetermined quality threshold; obtaining a subsequent training data set including a corresponding set of data pairs, each data pair including a signal with the corresponding object, and a mixed signal including the corresponding object and at least one other signal; determining one or more layers of the model for separating the corresponding one object based on the metadata; training the one or more layers of the model using the subsequent training data set, wherein the training includes freezing the remaining layers of the model so that only the determined one or more layers are trained using the subsequent data set.

[0070] Freezing a layer may be understood herein as maintaining the layer, the layer structure and / or the coefficients determined for each layer during, for example, a training phase. In other words, such a frozen layer is not changed and, for example, is neither trained nor retrained when frozen. In contrast, unfrozen layers may be trained and / or retrained, respectively.

[0071] Correspondingly, by using metadata, retraining can be performed on certain layers where improved performance is desired or required. This can further allow for improved object audio separation performance of the model, as the separation of each object can be improved.

[0072] Metadata may be associated with an object. Alternatively or additionally, metadata may indicate the layers used to separate a particular object. In some examples, multiple objects (e.g., all objects) are each provided with metadata indicating the layers of the model used to separate the corresponding objects in the multiple objects. Alternatively or additionally, metadata may contain information about the layer location in the model of the entire architecture, layer type, head pointers to backward and forward layers, and / or related objects. Metadata may be output with the object or stored in a multi-object separation model and / or sub-model.

[0073] The one or more layers of the model for separating the corresponding one object may include or be only trained and / or used to separate the layers of the corresponding object.Alternatively or additionally, the one or more layers of the model for separating the corresponding one object may include or be trained and / or used to separate the layers of the corresponding object and one or more additional objects.

[0074] In case that the plurality of sub-models include and / or are implemented as a feature extractor and a plurality of additional layers, the frozen layer may be an additional layer for separating the corresponding objects.

[0075] The data indicating the quality may be provided by the listener or user via a user interface. The data indicating the quality may be provided by the user in the form of a quality rating on a scale (such as a digital scale), or may be provided in the form of a binary input (e.g., sufficient quality or insufficient quality). The separation quality may relate to whether other objects are audible in the output signal with the separated object, the signal energy of other objects in the output signal, etc. Alternatively or additionally, the data indicating the quality may be evaluated for the corresponding object based on a quality metric (such as a sound-to-noise ratio SNR, a correlation between the object separated from the output signal and the signal composed of the object, a speech quality metric, or a combination thereof). For example, such data may include an indication of quality (such as a quality value on a scale) or may be a binary value indicating sufficient quality or insufficient quality (i.e., whether the quality is below a predetermined quality threshold).

[0076] The predetermined quality threshold may be a value or a subjective threshold for a listener or user.

[0077] The subsequent training data set may be different from the training data set used to train the audio object separation model and / or the training data set used to previously train the sub-model. For example, compared to the training data set used to train the audio object separation model and / or the sub-model, the subsequent training data set may include different data pairs, i.e., different signals including audio objects or composed of audio objects, and / or different mixed audio signals. It should be understood that in the present disclosure, different signals including audio objects refer to different signals including audio objects of the same type.

[0078] A corresponding one of the objects in each data pair of a subsequent training data set may be a corresponding one of the same object type or category (i.e., the sub-model is trained to determine or separate and output the object type or category from a mixed signal comprising multiple audio objects (which may be multiple audio objects of different types)).

[0079] For example, in the case where the audio object is a speech recording, the data set for initial training may include multiple data pairs, each data pair including a signal consisting of the recorded speech, and a mixed audio signal including the recorded speech and at least one other object (such as recorded wind noise, rain noise, dog barking, bird chirping, knocking sound, etc.). The subsequent training data set may include data pairs, each data pair including a signal consisting of a different recorded speech, and a mixed audio signal including the different recorded speech and at least one other object. The at least one other object may be different from the at least one other object in the data set for initial training.

[0080] In the example, metadata can be added by a multi-object separation model 2′, such as Figure 3 . In the case of obtaining data indicating that the separation quality of the first separated object 21a output is lower than a predetermined threshold, a subsequent training data set including data pairs can be obtained, each data pair including an object containing the first object or the same object type of the first object or a signal consisting of it, and a mixed audio signal including the object and at least one other object. In the example, only the first additional layer 24a is used to separate the first audio object or the first type of audio object. In other examples, other or additional layers can also be used for this purpose. In this example, the second additional layer and the third additional layer 24b, 24c are frozen, and only the first additional layer 24a is retrained based on the subsequent training data set.

[0081] Figure 4 A schematic flowchart of an example of method 3 for training a sparse audio object separation model is shown.

[0082] Method 3 is a computer-implemented method for training a sparse audio object separation model. Method 3 comprises: obtaining 30 a training audio signal, the training audio signal comprising a sparse audio object, a non-sparse audio object, and at least one further audio object; and training 31 the model to separate a combination of sparse audio objects and non-sparse audio objects from the training audio signal.

[0083] In some examples, a plurality of training audio signals may be obtained in the step of obtaining the training audio signal 30. Alternatively or additionally, the training audio signal may be or include a training data set, the training data set including one or more data pairs, each data pair including a signal including a sparse audio object, a non-sparse audio object, and a mixed signal including a sparse audio object, a non-sparse audio object, and at least one additional audio object.

[0084] Typically, audio objects are classified as sparse objects or non-sparse objects based on their characteristics, such as the duration and / or spectrum of the object. Sparse audio objects can generally be defined as being narrower in frequency or time span than non-sparse objects.

[0085] A sparse audio object may be an audio object that appears in a relatively short time compared to a non-sparse audio object and / or has substantially all or most of its power in a relatively narrow frequency range compared to a non-sparse audio object. For example, a sparse audio object may be an audio object that has a duration of less than 1 second (e.g., less than 800 ms, less than 500 ms, less than 400 ms, less than 300 ms, less than 200 ms, less than 100 ms, less than 80 ms, or less than 50 ms). Alternatively or additionally, a sparse audio object may have a portion of its power, such as greater than 50%, greater than 60%, greater than 70%, greater than 80%, greater than 90%, greater than 92%, greater than 94%, greater than 96%, or greater than 98%, in a frequency band narrower than the human hearing range (e.g., less than 10 kHz, less than 5 kHz, less than 4 kHz, less than 3 kHz, less than 2 kHz, less than 1 kHz, less than 500 Hz, or less than 250 Hz).

[0086] Thus, a sparse object may appear during a portion of a non-sparse object's occurrence time range or duration, the portion being less than the entire occurrence time range or duration. Alternatively or additionally, the duration of a sparse object may be shorter than the duration of a non-sparse object. Alternatively or additionally, a sparse object may have a narrower frequency range (i.e., a frequency range in which a sparse object has frequency components) than a non-sparse object and / or may have a frequency range that is a portion of a non-sparse object's frequency range, the portion being less than the entire frequency range.

[0087] By training a sparse object separation model to separate a combination of sparse audio objects and non-sparse audio objects from a training audio signal, the model can be trained to improve separation performance. For example, the combination can allow for more robust separation because the risk of the model attempting to learn an all-zero mask for object separation is reduced, thereby improving the performance and robustness of audio object separation performed using the corresponding trained model.

[0088] The combination of sparse audio objects and non-sparse audio objects can be a signal, an audio file and / or a separated audio object including or consisting of sparse audio objects and non-sparse audio objects and / or audio data of sparse audio objects and non-sparse audio objects. Potentially, the model can be trained to separate a signal, an audio file and / or a separated audio object including a combination of sparse audio objects and non-sparse audio objects from a training signal.

[0089] The sparse audio object may be one or more of an animal sound (such as a bird chirping or a dog barking), thunder, and a percussion sound. Alternatively or additionally, the non-sparse audio object may be one or more of a speech, wind noise, and rain.

[0090] Figure 6 An exemplary spectrum illustrating examples of sparse audio objects and non-sparse audio objects is shown. Figure 5 .

[0091] exist Figure 6 Spectrum Figure 5 In FIG. 1 , a time axis TA and a frequency axis FA are provided. On the frequency axis FA, a first frequency to an eighth frequency f1-f8 are indicated. Similarly, a first time point to a tenth time point t1-t10 are indicated on the time axis.

[0092] The non-sparse object 50 is illustrated as having a duration from time point t1 to time point t9 and a frequency range from frequency f1 to frequency f8 on the time axis. It should be understood that the non-sparse object 50 does not need to include frequency components in each frequency bin from the first frequency f1 to the eighth frequency f8 at any time between the first time point t1 and the tenth time point t10; rather, the non-sparse object 50 generally includes signal energy within the time span from the first time point t1 to the tenth time point t10 and frequency components within the frequency range from the first frequency f1 to the eighth frequency f8. In an example, the non-sparse object may be speech.

[0093] exist Figure 6In the figure, the first group of sparse objects 51a, 51b and the second group of sparse objects 52a, 52b are also illustrated. As shown, the sparse object 51a appears from the second time point t2 to the third time point t3 and in the frequency range from the third frequency f3 to the fifth frequency f5. Similarly, the sparse object 51b appears from the sixth time point t6 to the seventh time point t7 and in the frequency range from the third frequency f3 to the fifth frequency f5. The first group of sparse objects 51a, 51b have the same object type and can be the same object, such as the same object. In the example, the sparse objects 51a, 51b can both be bird calls.

[0094] The sparse objects 52a, 52b in the second group of sparse objects similarly appear from the fourth time point t4 to the fifth time point t5 and in the frequency range from the first frequency f1 to the seventh frequency f7 for the sparse object 52a, and appear from the eighth time point t8 to the ninth time point t9 and in the frequency range from the first frequency f1 to the second frequency f2 for the sparse object 52b. The second group of sparse objects 52a, 52b have the same object type. In this example, the second group of sparse objects 52a, 52b are different from the first group of sparse objects 51a, 51b. In the example, the sparse objects 52a, 52b can all be knocking sounds.

[0095] like Figure 6 As shown in FIG. 5 , the frequency range of each of the sparse objects 51a, 51b, 52a, 52b is narrower than the frequency range of the non-sparse object 50. Similarly, the sparse objects 51a, 51b, 52a, 52b appear in a time interval shorter than the time interval of the non-sparse objects. In other examples, the frequency range of the sparse objects may be the same as the frequency range of the non-sparse objects. In this example, the time interval of appearance may be shorter than the time interval of the non-sparse objects.

[0096] like Figure 6As shown, non-sparse objects occur during the occurrence of non-sparse objects. Specifically, all time points from the second time point t2 to the ninth time point t9 are located in the span from the first time point t1 to the tenth time point t10. Similarly, the second frequency f2 to the seventh frequency f7 are located in the frequency range from the first frequency f1 to the eighth frequency f8. Correspondingly, each sparse object 51a, 51b, 52a, 52b in the first group of sparse objects and the second group of sparse objects occurs during a part (the part is less than the entire time range or duration) of the occurrence time range of non-sparse objects 50. In addition, the duration of sparse objects can be shorter than the duration of non-sparse objects. Alternatively or additionally, each of sparse objects 51a, 51b, 52a, 52b has a frequency range narrower than non-sparse objects 50, that is, each of sparse objects 51a, 51b, 52a, 52b has a frequency range of frequency components. Each of the sparse objects 51a, 51b, 52a, 52b has a frequency range that is a portion of the frequency range of the non-sparse objects, the portion being smaller than the entire frequency range.

[0097] Figure 5 A schematic flow chart of an example of method 4 for separating sparse audio objects from a mixed audio signal is shown. Method 4 is a computer-implemented method for separating sparse audio objects from a mixed audio signal. The mixed audio signal includes at least a sparse audio object, a non-sparse audio object, and at least one additional audio object. Method 4 includes: providing 40 a sparse object separation model and training the sparse audio object separation model according to the second aspect of the present disclosure; generating 41 a first separation signal by separating a combination of sparse audio objects and non-sparse audio objects from the mixed audio signal by the sparse audio object separation model; generating 42 a second separation signal by separating non-sparse audio objects from the mixed audio signal by a non-sparse audio object separation model trained to separate non-sparse audio objects from the mixed audio signal; generating 43 an output signal by subtracting the second separation signal from the first separation signal.

[0098] Thus, improved performance and robustness of audio object separation can be provided. By using a model trained according to the second aspect of the present disclosure (such as method 3) to separate a combination of sparse audio objects and non-sparse audio objects, for example to obtain a first separation signal, and then subtracting the non-sparse audio objects (e.g., the second separation signal) from the first separation signal, the sparse audio objects can be obtained. As mentioned with respect to method 3, by using a model trained according to the second aspect, therefore, when the non-sparse audio objects are subtracted from the combination of sparse audio objects and non-sparse audio objects, a more accurate and robust separation of sparse audio objects can be obtained.

[0099] In some examples, the non-sparse audio objects of the second separated signal are the same non-sparse audio objects and / or non-sparse audio objects corresponding to the first separated signal. Alternatively or additionally, the combination of sparse audio objects and non-sparse audio objects can be a signal, an audio file and / or a separated audio object including or consisting of sparse audio objects and non-sparse audio objects and / or audio data of sparse audio objects and non-sparse audio objects.

[0100] The sparse audio object separation model of the second aspect and / or the third aspect may be a multi-object separation model (such as the multi-object separation model according to the first aspect) or a portion thereof (such as a sub-model thereof). Alternatively or additionally, the non-sparse audio object separation model may be a multi-object separation model according to the first aspect and / or a portion thereof (such as a sub-model thereof).

[0101] In some examples, non-sparse audio objects can be separated from sparse audio objects by traditional non-sparse audio object separation models because these models are generally focused on and suitable for separating non-sparse audio objects from multi-object audio.

[0102] Subtracting the second separated signal from the first separated signal can be understood as determining the difference between the second separated signal and the first separated signal, the difference being the subtracted sparse audio object. Correspondingly, in the present disclosure, subtracting the two separated signals should not be understood in a strict mathematical sense, but refers to determining the difference between the two separated signals. In other examples, an output signal can be generated 43 by determining the difference between the second separated signal and the first separated signal, the output signal including or consisting of the difference.

[0103] In some examples, Method 4 may be introduced, incorporated into, or otherwise used in conjunction with Method 1.

[0104] Figure 7A schematic flow chart of an example of a method 6 for processing audio based on a signal-to-noise ratio (SNR) is shown. The method 6 is a computer-implemented method for processing audio based on a signal-to-noise ratio (SNR). The method 6 comprises: receiving 60 an input signal separated by objects comprising a plurality of audio objects; determining 61 a corresponding SNR of one or more of the plurality of audio objects of the input signal separated by objects; determining 62 that one or more of the plurality of audio objects are dominant audio objects in response to determining that one or more of the signal-to-noise ratios are above a predetermined first SNR threshold for the corresponding object, and / or determining that one or more of the plurality of audio objects are background audio objects in response to determining that one or more of the signal-to-noise ratios are below a predetermined second SNR threshold for the corresponding object; generating 63 a leakage-reduced output signal by reducing leakage from the input signal. Reducing leakage from the input signal comprises reducing one or more non-dominant audio objects and / or reducing the one or more background objects in the input signal.

[0105] Thereby, dominant objects (such as may be considered objects of interest in the signal) may be extracted more accurately and robustly, since leakage from non-dominant or background objects in the audio may be reduced or even removed from the signal.

[0106] Method 6 may be a post-processing method.

[0107] The first SNR threshold and / or the second SNR threshold for each object may be determined based on the type of the object and / or based on the characteristics of the object. The first SNR threshold and / or the second SNR threshold for each object may be different thresholds.

[0108] The first and / or SNR thresholds for each object may be pre-set or pre-determined for each object type and / or for each object. For example, a first SNR threshold and / or a second SNR threshold may be determined for each object type, and upon determining that the object belongs to the object type, the first SNR threshold and / or the second SNR threshold may be set for a specific extraction object. Alternatively or additionally, the object-separated input signal may include an indication of the object type for each of the objects and / or metadata indicating the object type for each of the separated objects of the object-separated signal.

[0109] In other examples, the method includes: determining an object type for each object in the objects of the signal separated by objects; and determining a first SNR threshold and / or a second SNR threshold for each corresponding object based on the object type and / or one or more characteristics of the corresponding object. The characteristics of the object may include duration, frequency range, average power, spectral power distribution in a frequency band, loudness estimation, or a combination thereof.

[0110] Reducing leakage may include applying a gain value (eg, a reduced gain value or a gain value less than 1) to each of the one or more non-dominant audio objects and / or the one or more background objects in the input signal.

[0111] The input signal separated by objects can be obtained by a sparse audio object separation model (such as the sparse object separation model second trained according to method 3 and / or used in method 4) and / or by a multi-object separation model (such as any one of the multi-object separation models 2, 2' or the multi-object separation model used in method 1). Alternatively or additionally, the input signal separated by objects can be obtained by method 1 or method 4.

[0112] An example of method 6 utilizing an input signal will be described below. However, it should be understood that this is only an example of method 6 utilizing this particular input signal, and other input signals may be applied. Correspondingly, it should be understood that method 6 is not limited to the following example.

[0113] In the example, the method receives 60 an input signal separated by an object, the input signal separated by an object includes a speech object, a bird chirping object, an animal sound object, a wind object, a rain and thunder object (represented as a rain object hereinafter for simplicity) and another background object. In this example and other examples, the rain and thunder object can alternatively be a rain object without thunder. The rain object including thunder and / or the rain and thunder object can be considered as an object. In this example, 61 SNR is determined for each object in the object. In this example, for each object in the object, the SNR is named as follows: the SNR of the speech object is represented as SNR_s, the SNR of the bird chirping object is represented as SNR_b, the SNR of the animal sound object is represented as SNR_a, the SNR of the wind object is represented as SNR_w, the SNR of the rain object is represented as SNR_r, and the SNR of another background object is represented as SNR_n.

[0114] In this example, the method determines 62 a first threshold and a second threshold for each object. In this example, the first threshold for each object is named as follows: the first threshold for the speech object is denoted as Th_s, the first threshold for the bird chirp object is denoted as Th_b, the first threshold for the animal sound object is denoted as Th_a, the first threshold for the wind object is denoted as Th_w, the first threshold for the rain object is denoted as Th_r, and the first threshold for another background object is denoted as Th_n. Similarly, the second threshold for each object is named as follows: the second threshold for the speech object is denoted as ε_s, the second threshold for the bird chirp object is denoted as ε_b, the second threshold for the animal sound object is denoted as ε_a, the second threshold for the wind object is denoted as ε_w, the second threshold for the rain object is denoted as ε_r, and the second threshold for another background object is denoted as ε_n.

[0115] In this example, a leakage-reduced output signal is generated 63 based on the determined SNR and the first and second thresholds by reducing leakage according to the following exemplary cases. There may be many cases, for which reason the following exemplary cases may represent only a subset of cases to which method 6 may be applied:

[0116] Case 1: For SNR_b>Th_b or SNR_a>Th_a: The bird chirping sound object or the animal sound object is determined to be the dominant object. When generating 63 the output signal, the rain object is determined to be a background object and is then cleared from the output signal or reduced or removed in the output signal so that the rain object is substantially absent from the output signal. In some examples, the rain object may be mixed with other background objects and then cleared, reduced or removed.

[0117] Case 2: For SNR_w>Th_w: The wind object is determined to be dominant. In this example, wind noise typically has high signal energy at 0-5kHz, which may mask a large amount of bird chirping. When generating 63 the output signal, the bird chirping object, the animal sound object, and the rain object are cleared from the output signal or reduced or removed in the output signal, so that the bird chirping object, the animal sound object, and the rain object are substantially absent from the output signal.

[0118] Case 3: For SNR_r>Th_r: The rain object is determined to be the dominant object. Based on the rain object as the dominant object, the bird chirping object, animal sound objects, and other background objects are considered unlikely to be objects of interest, and therefore, they are determined to be background objects and cleared from the output signal or reduced or removed from the output signal, so that the bird chirping object, animal sound objects, and other background objects are substantially absent from the output signal. In some examples, other backgrounds may be mixed with the rain object and subsequently cleared or reduced or removed from the output signal. In addition, if SNR_w<ε_w (a second threshold value for the lowest wind object SNR, e.g., -10dB), the wind is further determined to be a background object and cleared from the output signal or reduced or removed from the output signal, optionally mixed with the rain object and subsequently cleared, reduced, or removed.

[0119] Case 4: For SNR_n>Th_n: It is determined that speech is the only dominant audio object. In this case, for example, a rain object can be determined as a background object and cleared from the output signal or reduced or removed in the output signal, optionally mixed with other background objects and then cleared, reduced or removed. In addition, if SNR_b<ε_b (a second threshold of the lowest bird song object SNR, such as -70dB), the bird song object can be determined as a background object and cleared from the output signal or removed in the output signal, optionally mixed with other background objects and then cleared, reduced or removed.

[0120] Case 5: For SNR_w<ε_w: The wind object is determined to be a background object and is cleared from the output signal or reduced or removed in the output signal, optionally mixed with other background objects and subsequently cleared, reduced or removed.

[0121] Case 6: For SNR_r < ε_r (second threshold for minimum rain object SNR, e.g. -10 dB): the rain object is determined to be a background object and is cleared from, reduced or removed in the output signal, optionally mixed with other background objects and subsequently cleared, reduced or removed.

[0122] Case 7: For SNR_a<ε_a (second threshold of animal object SNR, eg -20 dB): the animal sound object is determined as a background object and cleared or removed from the output signal, optionally mixed with other background objects and subsequently cleared, reduced or removed.

[0123] In some examples, the input audio signal separated by object may be provided by method 1, method 4, or any combination thereof. Alternatively or additionally, the input audio signal separated by object may be provided by any one or both of the object separation models 2, 2'. In some examples, method 6 may be applied as a post-processing method to the signal separated by object separated by method 1, method 4, or any combination thereof.

[0124] A method is further disclosed, the method comprising the steps of method 1 and any additional features disclosed optionally in combination therewith, and for example the steps of subsequent method 6 and any additional features disclosed optionally in combination therewith, wherein the input audio signal separated by object received in step 60 is the output signal output in step 12. A method is further disclosed, the method comprising the steps of method 4 and any additional features disclosed optionally in combination therewith, and for example the steps of subsequent method 6 and any additional features disclosed optionally in combination therewith, wherein the input audio signal separated by object received in step 60 is the output signal output in step 43.

[0125] Figure 8A schematic flow chart of an example of a computer-implemented method 7 for processing audio based on scene environment classification is shown. The method 7 is a computer-implemented method for processing audio based on scene environment classification. The method 7 comprises: receiving 70 an object-separated input signal comprising a plurality of audio objects; determining 71 a scene environment by obtaining a classification of a scene environment in which the audio objects are recorded, the classification of the scene environment comprising classifying the scene environment into a respective scene environment of a plurality of scene environments based on audio and / or video information; outputting 72 a leakage-reduced output signal by reducing leakage from the input signal based on the determined scene environment.

[0126] Thereby, improved robustness and increasingly accurate separation can be achieved, since information about the scene context can help the method to reduce leakage.For example, the method can perform different leakage reductions or can reduce different objects depending on the classification of the scene context.

[0127] In some examples, as an alternative or in addition to determining step 71, method 7 may include determining an object type of one or more of the plurality of audio objects by obtaining a classification of the one or more of the plurality of audio objects, including classifying the one or more of the plurality of audio objects into corresponding object types of a plurality of object types based on audio and / or video information. The step of generating a leakage-reduced output signal may include reducing leakage from the input signal based on the determined scene environment, based on the determined object type (or classification) of the one or more audio objects, the determined scene environment, or a combination thereof, respectively.

[0128] Determining 71 the scene environment and / or determining the object type where relevant may include obtaining a visual image of the scene (such as one or more photographs and / or one or more videos of the scene), and / or obtaining audio from the location of the scene. The visual image and / or audio may be obtained from a user device (such as a mobile phone). For example, the input signal separated by object may be a signal separated by object originating from a video call, a conference call with video, or a call from a mobile phone with an accessible camera. Alternatively, the signal separated by object may originate from a recorded video recorded, for example, by a camera or camera phone. The visual image may be obtained from a data stream of a recorded video, a video / conference call, or a camera of a mobile phone. The audio may be obtained from an audio stream of such a call or may be obtained from the input signal separated by object (such as obtained from the input signal separated by object).

[0129] Determining 71 the scene environment may include analyzing and classifying the scene based on, for example, visual images and / or audio using a machine learning data architecture trained to classify the scene environment. The machine learning data architecture may be trained to classify the scene environment of a photo, video, and / or audio. The machine learning data architecture may be a classification model. The machine learning data architecture may be trained by including a data set of images, videos, and / or audio obtained in various scene environments. In one example, a training data set of at least 200 images, videos, and / or audio clips may be provided, and an operator may classify the scene environment for each image, video, and / or audio clip. The machine learning data architecture may be trained based thereon.

[0130] In some examples, scene environment categories may include classifications such as indoor, outdoor, and traffic.

[0131] Determining the object type may include analyzing and classifying the one or more objects based on, for example, visual images and / or audio using a machine learning data architecture trained to classify the scene environment. The machine learning data architecture may be trained to classify the object type of the audio object based on one or more of a photo, video, and / or audio (such as audio and one or more of a video and a photo). The machine learning data architecture may be an object classification model. The machine learning data architecture may be trained by a data set, the data set including the audio of each audio object type in the multiple audio object types and one or more of an image and a video obtained during recording the audio of the audio object. In one example, a training data set of at least 200 audio clips of at least two different object types and corresponding images and / or videos obtained during recording the audio of the audio object. An operator may classify the audio object type of each object. The machine learning data architecture may be trained based thereon.

[0132] In some examples, object type classifications may include classifications such as birds, dogs, cats, ocean, other animals, music, tree sounds, etc.

[0133] In some examples, an object type or classification for each of the plurality of objects of the input signal may be provided or obtained (e.g., in a separate step or obtained during receiving step 70) by method 7. In some examples, the object type may be provided in the input signal (e.g., in metadata of the input signal). In these and other examples, method 7 may include object type classification to classify the object into a subcategory or subtype of an audio object type. For example, where the metadata of the input signal indicates that one of the plurality of objects is an animal sound, method 7 may include classifying the object as a sound of a cat or a dog or another animal.

[0134] An example of method 7 utilizing an input signal will be described below. However, it should be understood that this is only an example of method 7 utilizing this particular input signal, and other input signals may be applied. Correspondingly, it should be understood that method 7 is not limited to the following example. In this example of method 7, the method further includes determining an object type of one or more audio objects in the plurality of audio objects.

[0135] In this example, audio and video are provided in an object-separated input signal comprising a plurality of object-separated audio objects received 70 by the method. The input audio signal comprises a speech object, a bird chirping object, an animal sound object, a wind object, a rain and thunder object, and another background object. The input audio data signal also comprises metadata indicating the object type of each of these objects.

[0136] In this example, the method determines 71 a scene classification based on the audio and video. The scene categories into which the scene is classified include indoor, outdoor and traffic. The scene environment may alternatively or additionally be the surrounding environment in which the audio in the plurality of audio objects is recorded.

[0137] In this example, the method determines the object type of each of the plurality of audio objects before, simultaneously with, or after determining 71 the scene classification. There may be many cases, for which reason the following exemplary cases may represent only a subset of cases in which the method 7 may be applied to generate 72 a leakage-reduced output signal:

[0138] Case 1: Scene classified as indoor: Wind objects and rain objects are cleaned from the output signal or reduced or removed in the output signal. Any bird chirping sound objects may not be rendered into the height channels.

[0139] Case 2: Scene classified as traffic: Any bird chirping objects, animal objects, wind objects and rain objects are cleaned from the output signal or reduced or removed in the output signal. Other background objects can be redefined as noisy speech and traffic noise.

[0140] Case 3: The audio object is classified as an ocean object, and optionally the scene environment is classified as outdoor: Bird chirping and wind objects are considered objects of interest. These objects can be enhanced or suppressed, and / or the remaining objects of the plurality of objects can be cleaned, reduced, or removed from the output signal.

[0141] Case 4: Audio object is classified as a cat or dog audio object: The speech object and the animal object are considered to be objects of interest. The rain object can be cleared from the output signal or reduced or removed in the output signal. Alternatively or additionally, other objects (such as the remaining objects in the plurality of objects other than the speech and animal objects) can be suppressed, such as cleared from the output signal or reduced or removed in the output signal.

[0142] Case 5: Audio objects are classified as other animals or trees, and optionally, the scene environment is classified as outdoor: Any wind objects are considered objects of interest, and potential bird chirping objects are similarly considered objects of interest. Any rain objects may be suppressed, such as cleared from the output signal or reduced or removed in the output signal. The remaining objects in the plurality of objects may be suppressed, such as cleared from the output signal or reduced or removed in the output signal. Another background object may be redefined as chatter noise.

[0143] When generating 72 a leakage-reduced output signal, further cases may be provided; in the above-mentioned example of method 7 with an exemplary object-separated input signal, cases 1-5 are mentioned only as exemplary cases.

[0144] The input signal separated by object can be the output signal of method 6 (optionally having any features disclosed in connection with it), method 4 (optionally having any features disclosed in connection with it) or method 1 (optionally having any features disclosed in connection with it).

[0145] Method 7 may be a post-processing method.

[0146] It should be understood that method 7 can be used in combination with method 6, such as for simultaneous processing or for processing before or after method 6. In some examples, methods 6 and 7 can be interrelated and / or can be provided as one method.

[0147] Reducing leakage may include mixing the audio objects based on the determined scene environment. As an example, mixing the audio objects may include adjusting the weight of one or more audio objects in the plurality of audio objects. The weight may be determined based on the determined scene environment.

[0148] Alternatively or additionally, reducing leakage from the input signal may be performed as described with respect to method 6, such as by applying a gain value having a value less than 1.

[0149] Where object classification is performed, a weight or gain value may be determined based on the determined object type of one or more objects of the plurality of objects.

[0150] Obtaining the classification of the scene environment may include classifying the scene environment by a scene environment classifier.The scene environment classifier may be trained to determine the scene environment based on an audio signal and / or a visual signal such as a video signal.

[0151] The input signal may be an output signal of the method according to any one of the first, second or third aspects.

[0152] Fig. 9 A schematic block diagram of a system 8 configured to perform a method according to the present disclosure is shown. The system 8 comprises: a processing unit 81; a non-transitory computer-readable medium 82, wherein the non-transitory computer-readable medium stores instructions, and when the instructions are executed by the processing unit 81, the processing unit performs the method according to any one of the first aspect to the fifth aspect, such as any one or more of methods 1, 3, 4, 6 or 7.

[0153] The processing unit 81 may be any type of processing unit, such as a central processing unit CPU, a microcontroller unit MCU, a field programmable gate array FPGA, a digital signal processor DSP, etc. The non-transitory computer readable medium 82 may be any type of non-transitory computer readable medium, such as a computer memory, a random access memory RAM, a read-only memory ROM, a flash memory, etc.

[0154] In some examples, system 8 may be incorporated into and / or may be a part of a computer device or a portable audio device, such as, but not limited to, a head-mounted device, earbuds, an audio processing system of a speaker system, a wireless portable speaker, a mobile phone, a tablet computer, a personal computer, a server, etc.

[0155] Conclusion

[0156] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe common objects indicates merely that different instances of similar objects are referred to and is not intended to imply that the objects so described must be in a given order temporally, spatially, in ranking, or in any other manner.

[0157] In the claims below and the description herein, any of the terms comprising, including, or which comprises is an open term, which means at least including the elements / features that follow, but not excluding other elements / features. Therefore, when the term comprising is used in a claim, the term should not be interpreted as being limited to the devices or elements or steps listed thereafter. For example, the scope of the expression "a device comprising A and B" should not be limited to a device comprising only elements A and B. As used herein, the term including, or any of which includes, or that includes, is also an open term, which also means at least including the elements / features that follow the term, but not excluding other elements / features. Therefore, including is synonymous with comprising and means comprising.

[0158] As used herein, the term "exemplary" is used in the sense of providing an example, rather than indicating a quality. That is, an "exemplary embodiment" is an embodiment provided as an example, but not necessarily an embodiment of exemplary quality.

[0159] It should be understood that in the above description of exemplary embodiments of the present invention, various features are sometimes grouped together in a single embodiment, figure, or description thereof in order to simplify the present disclosure and aid in understanding one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention to require more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, inventive aspects lie in less than all the features of a single preceding disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, wherein each claim stands on its own as a separate embodiment of the present invention.

[0160] In addition, although some embodiments described herein include some features included in other embodiments and do not include other features included in other embodiments, as will be understood by those skilled in the art, combinations of features of different embodiments are intended to be covered and combinations of features of different embodiments form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0161] In addition, some embodiments herein are described as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform functions. Therefore, a processor with instructions required to perform such methods or method elements forms a device for performing the methods or method elements. In addition, the elements of the device embodiments described herein are examples of devices that perform the functions performed by the elements.

[0162] In the description provided herein, many specific details are set forth. However, it should be understood that embodiments of the present invention can be practiced without these specific details. In other instances, well-known methods, structures, and techniques are not shown in detail in order to avoid blurring the understanding of this specification.

[0163] Thus, while specific embodiments of the invention have been described, those skilled in the art will recognize that other and further modifications may be made, and it is intended that all such changes and modifications be claimed. For example, any formula given above merely represents a procedure that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged among functional blocks. The described methods may have steps added or deleted.

[0164] The systems, devices and methods disclosed above may be implemented as software, firmware, hardware or a combination thereof. For example, aspects of the present application may be at least partially embodied in a device, a system including more than one device, a method, a computer program product, etc. In a hardware implementation, the division of tasks between the functional units mentioned in the above description does not necessarily correspond to the division of physical units; on the contrary, a physical component may have multiple functions, and a task may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a digital signal processor or a microprocessor or as hardware or an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transient medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. Further, it is well known to those skilled in the art that communication media typically embodies computer-readable instructions, data structures, program modules or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery media.

[0165] Project List

[0166] Item 1. A method for processing audio, the method comprising:

[0167] Receive audio;

[0168] extracting objects from the audio using a model trained based on a signal-to-noise ratio (SNR), the model having a plurality of sub-models or a plurality of layers, the model being configured to process sparse objects;

[0169] reducing or removing leakage from the object by processing the extracted object to generate output audio based on at least one of SNR or audio-visual context; and

[0170] The output audio is provided for transmission, processing, or storage.

[0171] Item 2. A method for training a model based on a signal-to-noise ratio (SNR), the method comprising:

[0172] receiving an input signal as training data;

[0173] The input signal is processed by a model including a plurality of sub-models, each sub-model is trained by a corresponding data set, each data set includes a corresponding set of data pairs, each data pair includes a corresponding object signal and a corresponding mixed signal, each mixed signal includes a mixture of the corresponding object signal and the inference signal at a corresponding signal-to-noise ratio (SNR), the processing generates a corresponding mask or a corresponding cleaned spectral magnitude from each sub-model; and

[0174] A result of the process for processing audio is provided, the result comprising the model trained based on the signal-to-noise ratio.

[0175] Item 3. A method for training a model having multiple layers, the method comprising:

[0176] receiving an input signal as training data;

[0177] extracting shared features and individual objects from the input signal;

[0178] processing each object through a corresponding layer, each layer learning a mapping between the shared features and an object target corresponding to the layer; and

[0179] A result of the processing is output, the result including the model having a plurality of layers, the model being configured to process the audio including extracting features from the audio and separating objects from the audio using the layers.

[0180] Item 4. The method as described in Item 3, comprising:

[0181] each layer is represented by corresponding metadata, the metadata including information about: a location of a corresponding layer of an architecture of the layer, a layer type, head pointers to backward and forward layers, and one or more objects corresponding to the layer; and

[0182] Using a new training dataset to train specific layers identified as bad performers, including locating the bad performers based on metadata after freezing other layers.

[0183] Item 5. A method for processing audio containing sparse objects, the method comprising:

[0184] Training deep learning-based models, including:

[0185] Add a non-sparse object to a sparse object;

[0186] adjusting the ratio of the non-sparse objects to the sparse objects to generate a plurality of mixed objects; and

[0187] Using the hybrid object to train the deep learning based model; and

[0188] The trained model is used to separate the sparse audio objects from other objects of the input, including subtracting the non-sparse objects.

[0189] Item 6. A method as described in Item 5, wherein the non-sparse objects include speech, and the sparse objects include at least one of bird chirping or knocking sounds.

[0190] Item 7. A method for processing audio based on a signal-to-noise ratio (SNR), the method comprising:

[0191] receiving an input including an audio object separation result, the input including one or more audio objects; and

[0192] Reducing leakage from the input by applying (1) a plurality of source separation models, each corresponding to a respective audio type, and (2) a plurality of maximum and minimum signal-to-noise ratios (SNRs), each corresponding to a respective audio type, wherein reducing the leakage comprises:

[0193] For a first object having a first audio type, in response to determining that the SNR satisfies a first threshold, designating the first object as misclassified; and

[0194] For a second object having a second audio type, in response to determining that the SNR satisfies a second threshold, a portion of the second object is removed or reclassified as leakage.

[0195] Item 8. A method for processing audio based on an audiovisual context, the method comprising:

[0196] receiving an input including an audio object separation result, the input including one or more audio objects; and

[0197] Reducing leakage from the input based on a context determined based on audio-visual information includes applying a classifier derived based on the audio-visual information, wherein reducing the leakage includes adjusting a weight of the audio object based on a weight assigned to the object type of the audio object and a specific audio-visual context corresponding to the object type.

[0198] Item 9. A system comprising:

[0199] one or more processors; and

[0200] A non-transitory computer-readable medium storing instructions which, when executed by the one or more processors, cause the one or more processors to perform the operations of claims 1 to 8.

[0201] Item 10. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of claims 1 to 8.

Claims

1. A method for separating audio objects in a mixed audio signal, the mixed audio signal comprising a plurality of audio objects, wherein: The method comprises: Receiving the mixed audio signal by a multi-object separation model; separating one or more audio objects among the plurality of audio objects of the mixed audio signal by the multi-object separation model; outputting an output signal comprising the one or more separated audio objects, The model includes multiple sub-models, and each of the multiple sub-models is trained to determine and output a corresponding one of the multiple audio objects.

2. The method according to claim 1, wherein: Each of the sub-models is trained by using a corresponding dataset among a plurality of datasets, each of the plurality of datasets comprising a corresponding set of data pairs, each data pair comprising a signal having the corresponding one audio object, and a mixed signal comprising the corresponding one audio object and at least one other signal.

3. The method according to claim 1 or 2, wherein: The plurality of sub-models include a feature extractor, and wherein each of the plurality of sub-models includes one or more layers configured to map from a feature extracted signal to a corresponding one of the audio objects, and wherein separating the one or more audio objects includes: Receiving the mixed audio signal by the feature extractor; Outputting, by the feature extractor, a feature signal comprising one or more features from the mixed audio signal; The characteristic signal is received by the one or more layers of each sub-model of the plurality of sub-models.

4. A method according to any one of the preceding claims, wherein: The method further comprises: adding metadata by the model, the metadata indicating the location of model layers in the model, the layer types of model layers and / or pointers to backward layers and / or forward layers of the model; obtaining data indicating that a separation quality of a corresponding one of the plurality of audio objects is below a predetermined quality threshold; obtaining a subsequent training data set comprising a corresponding set of data pairs, each data pair comprising a signal having the corresponding audio object and a mixed signal comprising the corresponding audio object and at least one further signal; determining, based on the metadata, one or more layers of the model for separating the corresponding one audio object; The one or more layers of the model are trained using the subsequent training dataset, wherein the training includes freezing the remaining layers of the model such that only the determined one or more layers are trained using the subsequent dataset.

5. A computer-implemented method for training a sparse audio object separation model, the method comprising Obtaining a training audio signal, the training audio signal comprising a sparse audio object, a non-sparse audio object, and at least one other audio object; The model is trained to separate the sparse audio objects and the combination of the non-sparse audio objects from the training audio signal.

6. A computer-implemented method for separating a sparse audio object from a mixed audio signal, the mixed audio signal comprising at least a sparse audio object, a non-sparse audio object and at least one further audio object, the method comprising: Providing a sparse object separation model and training the sparse audio object separation model according to claim 5; generating a first separated signal by separating the combination of the sparse audio object and the non-sparse audio object from the mixed audio signal by the sparse audio object separation model; generating a second separated signal by separating the non-sparse audio objects from the mixed audio signal by a non-sparse audio object separation model trained to separate the non-sparse audio objects from the mixed audio signal; An output signal is generated by subtracting the second separated signal from the first separated signal.

7. The method according to claim 5 or 6, wherein: The sparse audio object has a shorter duration than the non-sparse audio object and / or has a narrower frequency range than the non-sparse audio object, optionally wherein the sparse audio object is one or more of animal sounds, thunder and percussion sounds, and / or wherein the non-sparse audio object is one or more of speech, wind noise and rain.

8. A computer-implemented method for processing audio based on a signal-to-noise ratio (SNR), the method comprising: receiving an object-separated input signal including a plurality of audio objects; determining a corresponding SNR of one or more audio objects of the plurality of audio objects of the input signal separated by objects; determining that one or more of the plurality of audio objects are dominant audio objects in response to determining that one or more of the signal-to-noise ratios are above a predetermined first SNR threshold for the corresponding audio object, and / or determining that one or more of the plurality of audio objects are background audio objects in response to determining that one or more of the signal-to-noise ratios are below a predetermined second SNR threshold for the corresponding audio object; A leakage-reduced output signal is generated by reducing leakage from the input signal, wherein reducing leakage from the input signal includes reducing one or more non-dominant audio objects and / or reducing the one or more background audio objects in the input signal.

9. The method according to claim 8, wherein: The first SNR threshold and / or the second SNR threshold for each audio object is determined based on a type of audio object and / or based on a characteristic of the audio object, and / or wherein, The first SNR threshold and / or the second SNR threshold for each audio object are different thresholds.

10. A computer-implemented method for processing audio based on scene environment classification, the method comprising: receiving an object-separated input signal including a plurality of audio objects; determining a scene environment by obtaining a classification of the scene environment in which the audio object was recorded, the classification of the scene environment comprising classifying the scene environment into a corresponding scene environment from a plurality of scene environments based on audio and / or video information; A leakage-reduced output signal is output by reducing leakage from the input signal based on the determined scene environment.

11. The method according to claim 10, wherein: Reducing the leakage includes mixing the audio objects based on the determined scene environment, such as adjusting a weight of one or more audio objects of the plurality of audio objects, the weight being determined based on the determined scene environment.

12. The method according to claim 10 or 11, wherein: Obtaining the classification of the scene environment includes classifying the scene environment by a scene environment classifier, the scene environment classifier being trained to determine the scene environment based on an audio signal and / or a visual signal such as a video signal.

13. The method according to any one of claims 8 to 12, wherein: The input signal is the output signal of a method according to any one of claims 1 to 4 or claims 5 to 7.

14. A non-transitory computer readable medium storing instructions which, when executed by a processing unit, cause the one or more processing units to perform the method according to any one of claims 1 to 13.

15. A system comprising: Processing unit; A non-transitory computer readable medium storing instructions which, when executed by the processing unit, cause the processing unit to perform the method according to any one of claims 1 to 13.

Images