Systems and methods for hierarchical audio source separation

Abstract

An audio processing system includes a memory storing a neural network trained to process audio mixtures to output an estimate of at least a subset of a set of audio sources present in the audio mixture. The audio sources are subject to a hierarchical constraint of a parent-child hierarchy applied to the set of audio sources, such that a parent audio source includes a mixture of one or more of its child audio sources. The subset includes the parent audio source and at least one of its child audio sources. The system also includes a processor to process the received input audio mixture using the neural network to estimate the subset of audio sources and their relationships according to the parent-child hierarchy. The system further includes an output interface configured to present the extracted audio sources and their relationships.

Description

Technical Field

[0001] This disclosure relates generally to audio processing, and more specifically to systems and methods for hierarchical audio source separation. Background Technology

[0002] With the introduction of deep learning technology, performance in the field of source separation has been significantly improved, especially in speech enhancement, speech separation, and music separation. These techniques succeed when the concept of the source is well-defined; in the case of speech enhancement or separation, the target is always defined as the speech of a single speaker. However, real-world scenarios may have more complex source definitions because, in real-world scenarios, audio source separation needs to consider multiple audio sources, which may not always be well-defined.

[0003] For example, in music separation tasks, determining what constitutes the target source is not well-defined. Even in well-studied problems such as vocal separation (where the singer is separated from non-vocal background music), the definition of "vocal" is somewhat ambiguous. Many popular songs typically contain a lead vocal part, possibly several additional background vocal parts, and sometimes additional sound effects tracks. This is a simple example, and determining which specific instrumental part to isolate becomes even more difficult when considering the variety of instruments with more possible timbres (such as synthesizers or guitars). One might want to go further and separate each instrument into unique instances of notes or chords.

[0004] Currently, audio source separation systems can separate audio sources existing in a given audio mix based on a single concept constituting the source. However, in many applications, this specific audio separation is insufficient. Summary of the Invention

[0005] Technical issues

[0006] Some embodiments aim to achieve sound separation of audio sources in an audio mix. Additionally or alternatively, some embodiments aim to separate audio sources at multiple granularity levels. Additionally or alternatively, some embodiments aim to determine the interrelationships between audio sources included in an audio mix. For example, some embodiments aim to estimate multiple audio sources within an auditory scene and simultaneously define the interrelationships between the estimated multiple audio sources. Additionally or alternatively, some embodiments aim to provide a neural network trained to achieve both of these objectives: extracting different audio sources included in an audio mix and providing the interrelationships between the extracted audio sources.

[0007] Solution to the problem

[0008] Some implementations are based on the understanding that classifying the extracted sound sources is beneficial for many applications. However, classification can be not only absolute but also relative, and in many practical applications, relative classification can complement or be as important as absolute classification. For example, in some applications, classifying a sound as originating from a guitar is just as important as classifying it as originating from a principle similar to that of a violin. However, there are countless different ways to define interrelationships, and not all of them can be used to train a neural network.

[0009] Some implementations are based on the understanding that many sounds encountered by humans are inherently hierarchical. For example, the hi-hat sound is one of many such sounds; it is one of several parts of a complete drum kit, one of many instruments in a band, and may be played in a bar, accompanied by other sounds. Therefore, the source separation problem can be restructured as hierarchical, grouping similar sound sources together at some levels and separating them at others. Restructuring the source separation problem as hierarchical allows users to deconstruct the same mixture in multiple ways depending on the appropriate level of the hierarchy of a given application.

[0010] It is worth noting that hierarchical classification of sources provides a natural and intuitive way to define the relationships between them. In fact, the relationship between any two audio sources belonging to the same hierarchical ontology can be defined by the shortest path connecting them in the graph representing that hierarchical ontology. Furthermore, hierarchical relationships between different audio sources can be defined in a manner suitable for training neural networks for joint extraction and mutual classification. For example, some implementations are based on the understanding that the estimated relationships between multiple audio sources can be defined in the form of a hierarchical structure of audio sources based on audio characteristics. The audio characteristics used to define the hierarchical system may depend on the application. For example, different harmonic structures can be used to determine the hierarchical system of musical instruments.

[0011] To this end, some implementations train neural networks to extract multiple audio sources from an audio mix subject to hierarchical constraints on audio characteristics, which enforce a parent-child hierarchy of the extracted audio sources. As used herein, the hierarchical constraints on the audio characteristics enforced in the parent-child hierarchy of the audio characteristics include the sets of audio characteristics of the parent audio source and the sets of audio characteristics of its child audio sources. Therefore, the set of audio characteristics of a parent audio source is such that, for each audio characteristic, its set of attributes is the union of the sets of attributes of the child audio sources for that audio characteristic, where the set of attributes is the set of values ​​that the audio characteristic can take. For example, electricity and acoustics are attributes of a sound production characteristic. Audio characteristics can vary depending on the application. Examples of audio characteristics include one or a combination of physical characteristics that produce sound, signal features (e.g., whether the sound has a harmonic frequency structure), or environmental information indicating the simultaneous occurrence of the sound.

[0012] Defining a hierarchical ontology using hierarchical constraints on audio features allows ensuring that these constraints are met during neural network training. In fact, hierarchical constraints, expressed as including the audio features of a sub-audio source within the audio features of a parent audio source, can be mathematically defined in a differentiable manner and thus used during training via backpropagation. In this way, a neural network trained to extract audio sources subject to hierarchical constraints can simultaneously perform two desired tasks.

[0013] Furthermore, some implementations are based on the surprising observation that training neural networks with dual objectives—specifically, hierarchical constraints on audio characteristics—offers additional technical advantages. For example, such training may require less training data to achieve dual-objective source separation, which is necessary for training neural networks with only a single source separation objective. Moreover, training neural networks for multi-level source separation achieves synergy in performance quality across the separation task at each level. Some implementations are based on the intuition that this synergy is possible due to the reusability of the same training data samples across different levels of the hierarchical ontology. These are training data samples used to train the ontology at a specific level, which indirectly participate in the training of shared parameters that also affect the separation at other levels of the ontology.

[0014] In some exemplary embodiments, the graded instrument separation process is based on separating a specific instrument (such as a guitar). In some additional or alternative embodiments, the graded instrument separation process is based on determining what to separate based on user-provided audio examples. In some embodiments, the graded instrument separation process is based on separation across all grade levels, even if the training data is limited to a fine-grained level of the hierarchical system. It is important to note that the instrument separation examples are provided for clarity and are not intended to limit the scope of different implementations. Various implementations can be applied to different kinds of audio mixing, audio source separation, and cross-classification.

[0015] Some implementations are based on the idea that audio sources separated from a given audio mix can be arranged in a hierarchical order of sound-producing objects, where the top level of the hierarchy corresponds to a broad group (e.g., alto string instruments), while lower levels are more specific (e.g., acoustic guitar). Regarding source separation, an audio hierarchy can be defined such that sources at higher levels in the hierarchy consist of mixtures of sources at lower levels. Each source node may be further separated into sub-sources and combined with its siblings to create a parent source.

[0016] Some implementations are based on the understanding that source-specific separation (SSS) networks can be implemented in a single network to estimate multiple sources from a given auditory mixture. The mask inference network attempts to estimate a real-valued mask M for a single target source c by minimizing some distortion metric between the source estimates obtained from the mask and the reference spectrogram. To this end, in some implementations, the SSS network uses a truncated phase-sensitive approximation (tPSA). The SSS network outputs a set of masks (e.g., N masks), one mask for each target source.

[0017] Therefore, in some implementations, multiple sources are arranged in a hierarchical order. To this end, some implementations implement a multi-level strategy, where the network is trained to output multiple levels of a hierarchical system at a time. For example, a single network can output N masks, one mask for each of the N levels of the hierarchical system. This allows multiple layers of the network to be trained jointly, and the network to be trained along the entire path of the hierarchical system (e.g., [string / keyboard instruments] → [guitar] → [unlit guitar]). Thus, a single network may be able to leverage learned knowledge about the masks to enhance and improve its estimation of another mask along the hierarchical system.

[0018] Some implementations are based on the understanding that it is impossible to capture many instruments in an instrument hierarchy using only an SSS network for hierarchical source separation. To capture the relationships between different instruments in the hierarchy, it may be necessary to use a single network to separate multiple instruments at different levels. For this purpose, a hierarchical audio separation algorithm can be used, employing a query-by-example (QBE) network. The QBE network takes examples of the mix and desired source as input to separate from the mix. By extending this to the hierarchical case, the entire source of the audio (e.g., instrument) hierarchy used for source separation can be modeled. Alternatively, separation conditions can be set on leaf node labels instead of on the query, thus achieving a hierarchical extension of conditional source separation methods. In some implementations, the hierarchical QBE includes two networks: one to compute the query anchor A for a given input query Q. q The query network, and the combination of query anchors A q As input, a masking net is generated for the target source in the same node as query Q in the hierarchical system, which is also used to generate a mask M.

[0019] Some implementations are based on the understanding that a hierarchical QBE system can be trained to minimize a truncated phase-sensitive approximation (tPSA) objective function based on a target source, where the target source used to train the network is determined by both the query and a given level in the hierarchical system. The audio samples used for training and validation can depend on the type of sound to be hierarchically separated. For example, for hierarchical separation of musical instruments, the network can be trained on audio samples of different instruments (such as guitars, pianos, etc.).

[0020] Some implementations are based on the understanding that the mixed components exist within a hierarchical system, and therefore, knowledge of the structure of the mixture can be used to constrain the network. For example, the relationship between different instruments can be predetermined according to the hierarchical order of the instruments. In one implementation, this relationship is captured such that the set of masks for audio separation generated by the multi-level hierarchical network follows the same structure as the hierarchical system, i.e., higher-level masks consist of lower-level masks. However, this could lead to a situation where the network outputs a mask for every node in the hierarchical system, which may be infeasible for sufficiently realistic hierarchical systems. Therefore, hierarchical constraints that do not require output for every node in the hierarchical system can be imposed. In some implementations, this hierarchical constraint requires that higher-level masks in the hierarchical system must allocate at least the same amount of energy as lower-level masks. Thus, a parent audio source in the hierarchical system is a mixture of its child audio sources.

[0021] Therefore, one embodiment discloses an audio processing system. The audio processing system includes a memory configured to store a neural network trained to process audio mixtures to output an estimate of at least a subset of a set of audio sources present in the audio mixture, wherein the audio sources are subject to hierarchical constraints that enforce a parent-child hierarchy on the set of audio sources, such that a parent audio source in the parent-child hierarchy includes a mixture of one or more of its child audio sources, and wherein the subset includes at least one parent audio source and at least one of its child audio sources; an input interface configured to receive an input audio mixture; a processor configured to process the input audio mixture using the neural network to extract an estimate of the subset of audio sources and their interrelationships according to the parent-child hierarchy; and an output interface configured to render the extracted audio sources and their interrelationships.

[0022] Another embodiment discloses an audio processing method, wherein the method uses a processor coupled to a memory storing a neural network trained to process audio mixtures to output an estimate of at least a subset of a set of audio sources present in the audio mixture, wherein the audio sources are subject to a hierarchical constraint of a parent-child hierarchy on the set of audio sources, such that a parent audio source in the parent-child hierarchy includes a mixture of one or more of its child audio sources, and wherein the subset includes at least one parent audio source and at least one of its child audio sources, wherein the processor is coupled to stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising: receiving an input audio mixture; processing the input audio mixture using the neural network to extract an estimate of the subset of audio sources and their interrelationships according to the parent-child hierarchy; and presenting the extracted audio sources and their interrelationships.

[0023] definition

[0024] As used in this article, audio characteristics are a set of properties that describe a sound signal. Sound signals with common properties for a given characteristic can be grouped together. Examples of audio characteristics include: signal features, such as frequency content with properties such as high-frequency dominance, mid-frequency dominance, and harmonic frequency structure; physical mechanisms that produce sounds with properties such as stringed instruments, acoustics, and electronics; or environmental information indicating where the sound typically occurs, such as indoors or outdoors.

[0025] As used in this paper, the audio hierarchy is a set of audio sources with a parent-child relationship. The parent audio source has the following characteristics: for each attribute, its attribute set is the union of the attribute sets of the child audio sources for that attribute. The attribute set is the set of values ​​that the attribute can take.

[0026] As used in this article, an audio source is an audio signal composed of vibrations from one or more sound-producing objects that can be meaningfully combined for a given application. Examples of audio sources could be a single string on a guitar, an entire guitar, or all the guitars played in a song.

[0027] As used in this paper, audio mixing is an audio signal composed of vibrations from multiple sound-producing objects, some of which may be irrelevant to a given task.

[0028] As used in this article, audio separation is the process of extracting signals from one or more audio sources from an audio mix.

[0029] As used in this paper, a mask is, for example, a nonlinear time-frequency filter estimated from an audio mix via a neural network, and is used by an audio separation system to extract audio sources from the audio mix. Typically, the mask estimates the probability that a source exists in each time-frequency interval (bin) of the spectrogram.

[0030] As used in this paper, hierarchical constraints apply a parent-child hierarchy to audio sources, such that a parent audio source in the hierarchy includes one or more of its child audio sources. Hierarchical constraints can be applied during source separation, training of the neural network performing the separation, or both. Separated audio sources are associated with nodes in the parent-child hierarchy. In this way, the extracted audio sources are correlated with each other according to the hierarchy. It is worth noting that although a parent-child hierarchy is predefined for the set of audio sources, during audio source extraction, only estimates of a subset of the audio sources of interest present in the audio mix are extracted.

[0031] There are many ways to implement this relationship, and therefore, hierarchical constraints vary in different implementations. Hierarchical constraints can be implemented directly, indirectly, or in combination thereof in a parent-child hierarchy. Examples of directly implementing hierarchical constraints include defining the hierarchy of audio sources as leaf audio sources corresponding to leaf nodes of the hierarchy and non-leaf audio sources corresponding to non-leaf nodes of the hierarchy, and directly matching the output of the separated audio sources to one of the nodes. For example, one implementation separates only the leaf audio sources and forms non-leaf audio sources by combining their child leaf audio sources. In this way, it is guaranteed that the parent audio source is a mixture of its child audio sources. Alternatively or additionally, one implementation separates the audio source sequence according to the path in the parent-child hierarchy, such that each subsequent audio source includes all previous audio sources.

[0032] Examples of indirectly implementing hierarchical constraints include establishing relationships between the energy and / or values ​​of audio characteristics of audio sources based on a parent-child hierarchy. For instance, one implementation estimates a mask for extracting a corresponding audio source when applied to audio mixing. In this implementation, the hierarchical constraint ensures that the energy of the mask corresponding to a parent audio source is not less than the energy of the mask corresponding to any of its child audio sources. Alternatively or additionally, one implementation ensures that the value of an audio characteristic of a parent node includes the values ​​of the corresponding audio characteristics of its child nodes. For example, in some implementations, the audio characteristics of a parent node are such that, for each characteristic, its attribute set is the union of the attribute sets of its child nodes, where the attribute set is the set of values ​​that the characteristic can take.

[0033] In some implementations, hierarchical constraints are applied to the training data used to train the network, wherein the network is trained to estimate at least a parent audio source and at least one of its child audio sources, and the ground truth training data of the parent audio source is such that it is a mixture of the ground truth training data of its child audio sources. Attached Figure Description

[0034] [ Figure 1A ]

[0035] Figure 1A A block diagram of an audio processing system for hierarchical audio separation according to an exemplary embodiment is shown.

[0036] [ Figure 1B ]

[0037] Figure 1B The relationship between a parent source and its corresponding child source according to an exemplary implementation is shown.

[0038] [ Figure 1C ]

[0039] Figure 1C Multiple paths extracted from a predetermined hierarchical structure by an audio processing system according to an exemplary embodiment are shown.

[0040] [ Figure 1D ]

[0041] Figure 1D An audio processing system is shown according to an exemplary embodiment, which utilizes a network to extract audio sources at each level in a parent-child hierarchy.

[0042] [ Figure 1E ]

[0043] Figure 1E An audio processing system is shown according to an exemplary embodiment, which utilizes a network to extract audio sources at all levels in a parent-child hierarchy.

[0044] [ Figure 1F ]

[0045] Figure 1F This is a schematic diagram illustrating different components of a neural network in an audio processing system according to an exemplary embodiment, wherein the network is trained for source-specific separation (SSS).

[0046] [ Figure 1G ]

[0047] Figure 1G This is a schematic diagram illustrating different components of a neural network in an audio processing system according to an exemplary embodiment, wherein network 105 is trained to separate by example query (QBE).

[0048] [ Figure 1H ]

[0049] Figure 1HThis is a schematic diagram illustrating different components of a neural network trained for example query (QBE) separation according to an exemplary embodiment, wherein the query is provided as an input label.

[0050] [ Figure 1I ]

[0051] Figure 1I The extraction of multiple musical instruments by a QBE network according to an exemplary embodiment is illustrated, as well as the interrelationships between the extracted musical instruments.

[0052] [ Figure 2A ]

[0053] Figure 2A This is a flowchart illustrating the training of an audio processing system for extracting multiple source-specific audio sources from an input audio mixture signal according to an exemplary embodiment of the present disclosure.

[0054] [ Figure 2B ]

[0055] Figure 2B This is a flowchart illustrating the training of an audio processing system for extracting multiple audio sources from an input audio mixture signal according to an exemplary embodiment of the present disclosure, wherein the sources to be extracted are based on the content of a second query audio signal, i.e., a query-by-example (QBE) query.

[0056] [ Figure 3 ]

[0057] Figure 3 This is a block diagram illustrating a single-channel mask inference network architecture according to an exemplary embodiment of the present disclosure.

[0058] [ Figure 4A ]

[0059] Figure 4A Table 1, according to an exemplary embodiment, illustrates the contents of the grading levels used for training and testing a grading single instrument source-specific separation (SSS) network.

[0060] [ Figure 4B ]

[0061] Figure 4B Table 2, based on an exemplary embodiment, illustrates improvements in the scale-invariant source distortion ratio or SI-SDR (dB) for graded SSS (guitar) and QBE models.

[0062] [ Figure 4C ]

[0063] Figure 4C Table 3, based on an exemplary implementation, shows the results of non-hierarchical and hierarchical SSS and QBE models according to SI-SDR (dB).

[0064] [ Figure 4D ]

[0065] Figure 4D Table 4, based on an exemplary embodiment, illustrates the SI-SDR improvement (dB) of graded SSS and QBE models (separated by thick dashed lines) for an unprocessed audio mix (“mix”).

[0066] [ Figure 5 ]

[0067] Figure 5 An anomaly detection method for an audio processing system according to an exemplary embodiment is illustrated.

[0068] [ Figure 6A ]

[0069] Figure 6A This illustrates hierarchical audio separation used in an audio zooming application according to an exemplary implementation.

[0070] [ Figure 6B ]

[0071] Figure 6B A scaling control knob 601 according to an exemplary embodiment is shown, wherein, according to the exemplary embodiment, the position of the scaling control knob 601 is converted into a blending weight for audio scaling.

[0072] [ Figure 7 ]

[0073] Figure 7 A block diagram of a computer-based audio processing system according to some embodiments is shown.

[0074] [ Figure 8 ]

[0075] Figure 8 The steps of a method for extracting multiple audio sources from an input audio mix, implemented according to an exemplary embodiment, are shown. Detailed Implementation

[0076] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown only in block diagram form to avoid obscuring this disclosure.

[0077] As used in this specification and claims, the terms “for example,” “like,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and their other verb forms, when used with a list of one or more components or other items, shall be interpreted as open-ended, meaning that the list shall not be construed as excluding other, additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be construed as limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0078] Figure 1A A block diagram 100 of an audio processing system 101 for hierarchical audio separation according to an exemplary embodiment is shown. Figure 1A A high-level view of the proposed audio processing system 101 for audio source separation is provided. The audio processing system 101 obtains an input audio mix. The input audio mix includes multiple audio signals from multiple different audio sources. For example, the input audio mix may include the sounds of multiple musical instruments, such as guitar, piano, drums, etc. In another embodiment, the input audio mix may include the sounds of multiple engines, multiple human voices, multiple animal voices, multiple natural sounds, etc. Some embodiments of this disclosure aim to extract multiple audio sources 113a, 113b, 113c, and 113d (also referred to as multiple audio sources 113a-113d) from the input audio mix and simultaneously determine the interrelationships between the extracted multiple audio sources 113a-113d.

[0079] To this end, the audio processing system 101 includes a memory 103 configured to store a neural network 105 trained to separate audio sources from an audio mix, the audio mix being subject to a hierarchical constraint of a parent-child hierarchy for the audio sources, such that a parent audio source in the parent-child hierarchy is a mix of one or more of its child audio sources. Therefore, the neural network 105 extracts multiple audio source signals included in the input audio mix from the input audio mix. Furthermore, the input audio mix is ​​subject to a hierarchical constraint that implements a parent-child hierarchy 111 of multiple audio sources 113a-113d corresponding to the extracted multiple audio source signals. The extracted multiple audio sources 113a-113d may correspond only to a portion of the audio sources included in a predetermined hierarchical hierarchy of audio sources 107. The multiple audio sources 113a-113d are extracted in a hierarchical tree structure as defined in the hierarchical hierarchy of audio sources 107, wherein the hierarchical tree structure may include at least one grandparent, at least one parent, and at least one child (or leaf node). In another embodiment, the audio processing system 101 further extracts a hierarchical path including at least one grandfather or at least one father and at least one child.

[0080] Hierarchical constraints can impose constraints on the extracted multiple audio sources 113a-113d, such that each sub-audio source includes at least one set of audio features included by its parent audio source. Therefore, hierarchical constraints implement a hierarchy of audio features of the extracted audio sources 113a-113d, such that the audio features of the parent audio source 113a are the union of the values ​​of the audio features of one or more of its sub-audio sources 113b-113d. In another embodiment, hierarchical constraints can define how sub-audio sources are mixed to determine their parent audio source. In other words, the parent audio source includes a mixture of its sub-audio sources.

[0081] The hierarchical structure of the audio sources 107 determines the parent-child hierarchy 111 of the extracted audio sources 113a-113d. In some implementations, the hierarchical structure of the audio sources 107 is predetermined and includes a set of all possible audio sources associated with the input audio mixing and the relationships between the audio sources in that set. The set of audio sources depends on the type of audio source to be extracted. For example, in music source separation applications, the set of audio sources includes different instruments, such as acoustic guitar, electric guitar, drums, etc.

[0082] In response to receiving an input audio mix, the audio processing system 101 submits the input audio mix to the neural network 105. The neural network 105 extracts audio sources from the input audio mix that correspond to a parent-child hierarchy 111 of extracted multiple audio sources, and presents at least a portion of the extracted multiple audio sources 113a-113d and the relationships between them based on the parent-child hierarchy 113a-113d, where 113a is a grandparent source node with two child audio sources 113b and 113c. 113b and 113c are sibling nodes. Furthermore, 113c is a parent source node, where parent source node 113c has one child audio source 113d. Child source nodes 113b and 113d are also referred to as leaf nodes because they are the last source nodes in the parent-child hierarchy 111. Hierarchical constraints ensure that a parent audio source (e.g., 113a) comprises the audio characteristics or energy of all its child audio sources (e.g., 113b and 113c) and grandchild audio sources (e.g., 113d). Therefore, a parent audio source in a parent-child hierarchy is a mixture of one or more of its child audio sources.

[0083] In some implementations, any time-frequency region in the spectrogram of the hierarchical constraint-implementing neural network 105 assigned to one or more sub-sources (e.g., an acoustic guitar source) must also be assigned to the parent source (e.g., all guitars). Specifically, for each time-frequency point in the spectrogram, the output of the hierarchical constraint-implementing parent source is the maximum value of the outputs from the parent and sub-sources. The set of audio sources depends on the type of application. For example, in applications where music sources are separated, the set of audio sources includes different instruments.

[0084] In one exemplary embodiment, the set of audio characteristics for extracting multiple audio sources may include one or a combination of the physical characteristics of the multiple audio sources, signal characteristics of the input audio mixture generated by the multiple audio sources, and environmental information indicating the co-occurrence of audio in the input audio mixture. In another embodiment, the set of audio characteristics may include multiple pitch components, pitch-to-noise ratio, and cepstral peak amplitude in the input audio mixture.

[0085] Figure 1B The diagram illustrates the relationship between a parent source 115 and its corresponding child source 117 according to an exemplary embodiment. Hierarchical constraints ensure a hierarchical relationship between the parent source 115 and the child source 117, such that the parent source 115 includes a set of audio features, wherein the set of audio features of the parent source 115 includes the set of audio features of its children or child sources 117. The set of audio features of the parent source 115 is such that, for each audio feature, its attribute set is the union of the attribute sets of its children 117, where the attribute set is the set of values ​​that the audio feature can take. In such a way... Figure 1B In the exemplary embodiment shown, parent source 115 may correspond to the audio signals of all guitars. The parent source "guitar" may include a set of attributes corresponding to an electric guitar (sub-1) and an acoustic guitar (sub-2). Furthermore, in order to extract different audio sources from the input audio mix, the audio processing system 101 may initially use the neural network 105 to extract only the sub-audio sources corresponding to the leaf nodes of the parent-child hierarchy 111 from the input audio mix. Additionally, the system 101 reconstructs at least one parent audio source (e.g., ...) of the parent node of the parent-child hierarchy 111 by mixing the sub-audio sources according to one or more paths leading to the parent node on the parent-child hierarchy. Figure 1B (As shown). Furthermore, the audio processing system 101 can also be configured to separate the sub-source from the parent source using different masks (or filters). See below. Figure 1C Provide a detailed description of one or more paths.

[0086] In another embodiment, the interrelationships between the extracted multiple audio sources also allow the user to combine two or more audio sources from the extracted multiple audio sources and generate a new audio signal together. For this purpose, the audio processing system 101 can interpolate different audio sources by using different weights applied to the sub-sources to mix the sub-audio sources. (Refer to below...) Figure 6A Provides a detailed description of interpolating sub-audio sources using different weights.

[0087] In some embodiments, the audio processing system 101 extracts at least one path from a predetermined hierarchical system 107 based on the input audio mix, such that the end of the extracted path includes at least one target audio source, wherein the audio signal from the target audio source is present in the input audio mix. (Refer to below) Figure 1C Provide a detailed description of the path.

[0088] Figure 1C Multiple paths (121, 123) extracted from a predetermined hierarchical system by the audio processing system 101 according to an exemplary embodiment are shown. Figure 1C A predetermined hierarchy associated with musical instruments is shown (similar to predetermined hierarchy 107). In this example, the predetermined hierarchy includes four levels (level 1, level 2, level 3, and level 4). The instrument hierarchy is set from wide levels to narrow (or specific) levels, where level 1 corresponds to the widest level and level 4 corresponds to the narrowest level. Furthermore, the audio source nodes at the wide levels include mixed audio signals from multiple instruments, while the audio source nodes at the lowest level (in this case, level 4) include audio signals corresponding to a specific instrument (such as a guitar, piano, etc.). Initially, the audio processing system 101 receives an input audio mix 119. The input audio mix 119 includes a musical performance generated using multiple instruments.

[0089] Consider a musical performance that includes audio signals from guitars and trumpets, as well as audio sources from other instruments. Upon receiving an input audio mix 119, the audio processing system 101 filters the input audio mix 119 at each level that forms a path from the widest level to the narrowest level in order to extract specific audio sources. The audio processing system 101 can then extract the audio sources along the extracted path.

[0090] For example, audio processing system 101 can extract path 121 such that the leaf node at the end of path 121 corresponds to a guitar. Similarly, path 123 can be extracted such that the leaf node at the end of path 123 corresponds to a trumpet. Audio processing system 101 can indicate the entire path to the user, wherein each source node in the path can be indicated by the audio signal included in that source node. In an exemplary embodiment, system 101 can provide different identifiers to parent source nodes, sibling source nodes, and child source nodes, which can enable the user to easily analyze the different audio sources present in the input audio mix 119 and enable the user to combine different audio sources to generate different audio signals together.

[0091] Audio hierarchy system

[0092] As used in this paper, hierarchical constraints apply a parent-child hierarchy to audio sources, such that a parent audio source in the hierarchy includes one or more of its child audio sources. Hierarchical constraints can be applied during source separation, training of the neural network performing the separation, or both. Separated audio sources are associated with nodes in the parent-child hierarchy. In this way, the extracted audio sources are interconnected according to the hierarchical structure.

[0093] There are many ways to implement this relationship, and therefore, hierarchical constraints differ in different implementations. Hierarchical constraints can be implemented directly, indirectly, or in combination thereof to establish a parent-child hierarchy. An example of directly implementing hierarchical constraints includes defining the hierarchy of audio sources as leaf audio sources corresponding to leaf nodes of the hierarchy and non-leaf audio sources corresponding to non-leaf nodes of the hierarchy, and directly matching the output of the separated audio sources to one of the nodes. For example, one implementation separates only the leaf audio sources and forms non-leaf audio sources by combining their child leaf audio sources. In this way, it is guaranteed that the parent audio source is a mixture of its child audio sources. Alternatively or additionally, one implementation separates the audio source sequence according to the path in the parent-child hierarchy, such that each subsequent audio source includes all previous audio sources.

[0094] Examples of indirectly implementing hierarchical constraints include implementing relationships between the energy and / or values ​​of audio characteristics of audio sources based on a parent-child hierarchy. For example, one implementation estimates a mask for extracting a corresponding audio source when applied to audio mixing. In this implementation, the hierarchical constraint ensures that the energy of the mask corresponding to the parent audio source is not less than the energy of the mask corresponding to any of its child audio sources. Alternatively or additionally, one implementation ensures that the value of an audio characteristic of a parent node includes the values ​​of the corresponding audio characteristics of its child nodes. For example, in some implementations, the audio characteristics of a parent node are such that, for each characteristic, its attribute set is the union of the attribute sets of its child nodes, where the attribute set is the set of values ​​that the characteristic can take.

[0095] It is worth noting that although a parent-child hierarchy is predefined for the set of audio sources, during audio source extraction, only estimates of a subset of the audio sources of interest present in the audio mix are extracted. The subset of audio sources can be predefined or specified at execution time. In some implementations, the subset of audio sources corresponds to a path in the parent-child hierarchy of the audio source set, starting from the root node and extending to a leaf node. Alternatively or additionally, the subset of audio sources may skip one or more audio sources on a path and / or include audio sources from other paths in the hierarchy.

[0096] During source separation, a trained neural network is used to estimate the audio sources. While the neural network is trained on training data of varying quality, the estimates of the extracted audio sources can vary for different implementations based on the content of the input audio mix. However, to implement hierarchical constraints, the estimation of a parent audio source includes estimates of its child audio sources within the provided input audio mix. This also includes implementing hierarchical relationships of audio characteristics that are useful for many applications. For example, in some implementations, the audio characteristics of at least one extracted parent audio source comprise the union of the audio characteristics of at least one extracted child audio source. Examples of audio characteristics of audio sources include one or a combination of spectral features, ambient information, pitch components, pitch-to-noise ratio, and cepstral peak amplitude.

[0097] In some implementations, hierarchical constraints are applied to the training data used to train the network, wherein the network is trained to estimate at least one parent audio source and at least one of its child audio sources, and the training data of the parent audio source is such that it is a mixture of the training data of the child audio sources. In this way, the execution of the trained neural network produces outputs with hierarchical relationships. These hierarchical relationships (e.g., cross-relationships) can be utilized by subsequent processing (e.g., anomaly detection, audio scaling, etc.).

[0098] In some embodiments of this disclosure, the relationships between the extracted audio sources 113a-113d are obtained by arranging each audio source among the extracted audio sources 113a-113d in a hierarchical order of the sound-producing objects. In the hierarchical system of the sound-producing objects, the top level corresponds to a wide group, while lower levels are more specific. For example, in a hierarchical system of musical instruments, the top level may include mid-range string instruments, while lower levels may include acoustic guitars. Furthermore, regarding source separation, an audio hierarchy can be defined such that sources at higher levels in the hierarchy consist of a mixture of sources at lower levels. Each source node can potentially be further divided into one or more sub-sources and combined with its sibling nodes to create a parent source. Consider a hierarchy with L levels, S l，c This represents the c-th source type node at level l in the hierarchical system, where l = 1, ..., L, and assuming the leaf source type set S. 1，c It cannot be decomposed into other source types, and S L，1 It is the only source type at the top of the hierarchy and includes all source types. Furthermore, C... l，c S represents l，c The set of indices of the sub-sources at level l-1. The hierarchical system can be defined as:

[0099]

[0100] The path along the hierarchical system can be defined as starting from the source type node S. a To the lower-level destination source type node S b The source type sequence.

[0101] In some implementations, the hierarchical system defined by equation (1) above can be used to decompose audio mixes x. Let s l，c This indicates that the source type in x is S. l，c The corresponding source components, where submixtures of all signals of the same type are considered a single component. By definition, S L，1 =x. Each c-th source component s at level l. l，c It can be decomposed into source components s l-1，c′ This makes S l-1，c′ Is it related to subsource type S l-1，c′ Signals corresponding to all sources:

[0102]

[0103] Where l = 2, ..., L. For simplicity, the summation operator represents mixing, but mixing is usually more complex than a simple summation of signals.

[0104] In some embodiments of this disclosure, an audio hierarchy composed of instrumental mixes is analyzed. Some embodiments are based on the understanding that the hierarchy formula can be applied to mixes with source content of any type.

[0105] Graded source separation

[0106] In a non-hierarchical setting, source separation can be formulated as estimating C complex spectra S. c ∈C F×T Where c = 1, ..., C, it represents the spectrogram X ∈ C of the input audio mixture. F×T The expected source set within. In this general formula, the source S is... c No need to be with source S d There is no relation, because c≠d. In some implementations, for a given input audio mix X, the proposed audio processing system 101 is implemented as a hierarchical source separation system that uses a hierarchical separation algorithm under a given hierarchy and can attempt to extract one or more submixes, each submixe corresponding to a source type S at level l. l，c All sources. For example, separating all guitars (acoustic and electric guitars) from a mix that includes electric guitars, acoustic guitars, pianos, and drums (such as...). Figure 1I (As shown).

[0107] Figure 1DAn audio processing system 101, according to an exemplary embodiment, is shown that utilizes a network to extract audio sources at each level of a parent-child hierarchical system 107. Figure 1D In this context, neural network 105 includes multiple neural networks 105a, 105b, and 105c (also referred to as multiple neural networks 105a-105c), wherein each of the multiple neural networks 105a-105b extracts audio source nodes 113a, 113c, and 113d at its corresponding level, such that the extracted audio sources are arranged hierarchically. In this case, each of the multiple neural networks 105a-105c is trained individually to extract a specific audio source. The extracted audio sources 113a, 113c, and 113d are based on a predetermined parent-child hierarchical system 107 (…). Figure 1D (Not shown in the image) are arranged in a hierarchical manner.

[0108] Figure 1E An audio processing system 101 is shown according to an exemplary embodiment, which utilizes a network to extract audio sources of all levels in a parent-child hierarchy 107. In this case, a neural network 105 is jointly trained to extract all audio sources 113a, 113c, and 113d from an input audio mix in a hierarchical manner. Furthermore, the extracted audio sources 113a, 113c, and 113d are based on a predetermined parent-child hierarchy 107 (…). Figure 1E (Not shown in the image) are arranged hierarchically. Because only one neural network is used to extract multiple audio sources, the aforementioned multi-level network can increase the scalability of the audio processing system.

[0109] It is worth noting that, compared to a combination of different neural networks 105a, 105b and 105c trained for a specific audio source, a neural network 105 jointly trained to extract multiple audio sources based on their hierarchical relationships can be more efficient.

[0110] In addition, some implementations use source-specific separation (SSS) networks or query-by-example (QBE) networks to implement neural network 105 for hierarchical audio source separation.

[0111] Hierarchical source-specific separation

[0112] Source-Specific Separation (SSS) networks based on mask inference typically attempt to make the source estimate obtained from the mask consistent with the reference S. c To estimate the real-valued mask of a single target source c, some distortion metrics are minimized. The objective function that minimizes distortion can be achieved using the truncated phase-sensitive approximation (tPSA) objective:

[0113]

[0114] Where ⊙ represents element-wise product, and |Y| and ∠Y represent the amplitude and phase of spectrum Y. It is a truncation function that ensures the target can be achieved using a sigmoid activation function. The estimated mask... Element-wise multiplication with the original mixed spectrum X yields the target source S. c The estimate.

[0115] Some implementations construct hierarchical SSS networks by training a single network to output a single node S at a given level of the hierarchy. n，c Each such single-level network can be trained to minimize the tPSA objective above, where the target source is S. n，c The components correspond to the target source type in the hierarchical system within the hybrid X. Each of these networks outputs a mask for its target source type. And they are trained independently of each other.

[0116] Some implementations employ a multi-level strategy, in which the neural network 105 is trained to output multiple levels of a hierarchical system at a time. A potential advantage of this strategy is that the neural network 105 can utilize information about the mask. Learning knowledge to enhance and improve its understanding of another mask in the hierarchical system. The estimation. In some implementations, a single network outputs N masks of N levels along a single path in the hierarchical system (e.g., [orchestral / keyboard instruments] → [guitar] → [undriven guitar], where "undriven" indicates an acoustic or electric guitar without any drive or distortion). In other implementations, multiple paths in the hierarchical system are extracted simultaneously. In other implementations, all nodes in the hierarchical system are considered and extracted. In some implementations, the network outputs a mask for each leaf node in the hierarchical system and recombines submixes corresponding to parent nodes from leaf sources as it traverses the hierarchical system, training the neural network 105 by combining the loss function of all nodes in the hierarchical system.

[0117] During training, the goal of network 105 is to simultaneously reconstruct the extracted multiple audio sources 113a-113d at all levels L of the parent-child hierarchical system 111. That is, each level in the parent-child hierarchical system 111 has a tPSA objective term (Equation 3), and the total loss is the sum of levels L across the parent-child hierarchical system 111. Neural network 105 can be trained to simultaneously output masks of audio sources at different levels of the hierarchical parent-child hierarchical system 111 subject to hierarchical constraints, while minimizing the objective function tPSA at each level of the parent-child hierarchical system 111.

[0118] In one exemplary embodiment, assuming network 105 outputs one parent source node and two child source nodes, network 105 is trained in such a way that the first output always corresponds to the parent source node, the second output always corresponds to the first child source node, and the third output always corresponds to the last child source node. That is, during the training of network 105, the true child mixture of the parent source will be used as the target of the first output in network 105, the true signal of the first child source node will be used as the target of the second output, and so on. In the exemplary embodiment, SSS network 105 can be configured with one grandparent -> one parent -> one child output.

[0119] Queries by Example (QBE)

[0120] To capture the interrelationships between different audio sources in an audio source hierarchy, some implementations use a single network to separate multiple instruments at different levels. A hierarchical SSS network using one network per level only allows the network to learn one node in the hierarchy at a time. A hierarchical SSS network using a multi-level network along a single path in the instrument hierarchy cannot model other instruments. A hierarchical SSS network that models all nodes in the hierarchy can extract the interrelationships between different audio sources in the audio source hierarchy, but any sufficiently realistic hierarchy may contain dozens of leaf nodes, which can lead to memory and computational problems, as well as difficulties in balancing the contributions of all losses. To mitigate these potential problems, in some implementations, the audio processing system 101 uses a query-by-example (QBE) network.

[0121] The QBE network takes a mixture of desired sources and an example (or query audio) as input to separate from the input audio mixture. The QBE network can be used to model an entire hierarchy of audio sources (e.g., an instrument hierarchy) for source separation. In an exemplary implementation, instead of setting a condition on the query audio, separation can be conditional on leaf node labels, leading to a hierarchical extension of the conditional source separation method. The query audio can be considered a generalization of class labels, and QBE can further provide the ability to interpolate to unseen source types during inference.

[0122] In some implementations, the audio processing system 101 accepts an input query audio and an input audio mix, and outputs a portion of a plurality of extracted audio sources 113a-1113d corresponding to a path along a parent-child hierarchy 111, the path terminating at an audio source of the same source type as the input query audio. The path along the parent-child hierarchy 111 includes a plurality of source nodes, each corresponding to a type of audio source. Each source node includes at least one child audio source, and each source node combines with sibling source nodes to create a parent source node.

[0123] Some implementations achieve a QBE network (or hierarchical QBE network) that relies on both a query network and a shield network. The query network calculates certain input query audio. Query anchor point A q ∈R k V is a k-dimensional query embedding generated by the network at each time-frequency interval i = (f, t) in the query spectral space. q，i Weighted sum:

[0124]

[0125] in, The query existence vector for query Q is defined as P if the magnitude at interval i = (f, t) is higher than a threshold (set from the maximum value to -60dB). q，i =1, otherwise 0. Query anchor point A q At each frame t, with the blended X t The frequency vectors are concatenated and used as input to a shielding network that generates target sources s for each hierarchical layer n of interest. n，c mask Its node S in the hierarchical system is the same as the query Q. n，c In China. This architecture is... Figure 1I As shown in the image.

[0126] The QBE system is trained based on the target source s n，c Minimize the tPSA objective in equation (3), where the target source for training the neural network 105 is determined by both the query and a given level in the hierarchical system. In an exemplary embodiment, the loss function includes terms directly defined on the query embedding space.

[0127] In an exemplary embodiment, the QBE system can be trained using query audio (e.g., acoustic guitar). The acoustic guitar query is used to train neural network 105, which attempts to extract the corresponding source at the leaf node level, where the objective would consist of submixes of all unaccented guitars in the mix. Furthermore, the same acoustic guitar query can also be used to train neural network 105, which attempts to extract the corresponding source at the previous level, where the objective would consist of submixes of all guitars in the mix, regardless of whether they are unaccented guitars. Additionally, the QBE system is trained to output silence when the source node associated with the input audio query is absent among multiple source nodes in the hierarchical hierarchy of audio sources 107. The notification can be sent to the user in the form of a buzzer or a message informing them that no audio source node is associated with the input query audio.

[0128] In another implementation, a single-level QBE network can be implemented for a specific level in the hierarchy, with a single instrument implemented only at that level. Alternatively, a hierarchical multilevel QBE network returning to multiple (or even all) levels of the hierarchy can be implemented. To this end, some implementations further impose hierarchical constraints on the network output.

[0129] Constraints on hierarchical masks

[0130] Some implementations are based on the understanding that by assuming the mixed components exist in a hierarchical system, knowledge related to the structure of the components can be used to constrain the neural network 105. For example, the relationship that requires the set of masks generated by a multi-level hierarchical network as defined in equation (2) can be used to follow the same structure as the hierarchical system, i.e., higher-level masks are composed of lower-level masks.

[0131] However, this would require outputting a mask for every node in the hierarchy, which may be infeasible for large-scale hierarchical systems encountered in practice. Therefore, a hierarchical constraint that does not depend on knowledge of the entire hierarchy can be used. This hierarchical constraint requires that masks of higher levels in the hierarchy must allocate at least the same amount of energy as masks of lower levels. More precisely, the mask for level l is set as follows:

[0132]

[0133] Here, max is applied element-wise to each TF interval, and This is a mask estimation for level l output by the network. In some embodiments, the audio processing system 101 is configured to receive input data indicating a target audio source, and wherein the audio processing system 101 extracts the target audio source and its parent audio source from the input audio mix according to a parent-child hierarchy 107. The input data includes labels of nodes in the parent-child hierarchy, wherein the labels may include information indicating the target audio source, such as text data describing the name of the target audio source to be extracted from the input audio mix. In an exemplary embodiment, the labels may include a user's voice signal, wherein the voice signal includes the name of the target audio source that the user wishes to extract from the input audio mix.

[0134] In some implementations, the audio processing system 101 receives input data including a query audio signal, such that the neural network 105 is trained to separate the child audio source closest to the query audio signal and the parent audio source of the child audio source according to a parent-child hierarchy 107. The query audio signal may include an audio signal associated with a target audio source. Reference is made below. Figures 1F to 1H Provides a detailed description of how the audio processing system 101 extracts the target audio source from the input audio mix based on different types of input data indicating the target audio source.

[0135] Figure 1F This is a schematic diagram illustrating different components of a neural network 105 in an audio processing system 101 according to an exemplary embodiment, wherein the network 105 is trained for source-specific separation (SSS). An input audio mixture signal 125 is provided to a spectrogram extractor module 127, which is configured to generate a spectrogram of a frame sequence of the input audio mixture 127. The input audio mixture spectrogram is processed by a separation neural network 129, which consists of layers of learned weights that can process the input in a memoryless manner via matrix multiplication or convolution operations, or recursively connected to memory. Each layer also has one or more nonlinear operations that allow the separation neural network 129 to learn a mapping from the spectrogram of the input audio mixture signal 125 to a time-frequency mask, which, when multiplied with the input mixture spectrogram, recovers an independent spectrogram. The output of the separation neural network 129 is provided to a mask determination module 133, which outputs multiple masks, one mask for each relevant node in the hierarchical hierarchy. The mask determination module 133 is configured to form the output of the neural network 105 into a set of masks corresponding to the audio sources extracted at different levels of the parent-child hierarchy 111, such that the mask-to-spectrum application extracts the corresponding audio sources. Furthermore, hierarchical constraints are applied to ensure that the mask assignment at higher levels in the parent-child hierarchy 111 has at least the same amount of energy as the mask at lower levels in the parent-child hierarchy 111. In an exemplary embodiment, the separating neural network 129 itself may include the mask determination module 133.

[0136] Furthermore, the SSS network 131 is pre-specified to have a fixed number of outputs, one output for each relevant level in the parent-child hierarchy 111. To this end, the SSS network 131 is configured to generate multiple N masks, each mask corresponding to one of the N levels along the path in the parent-child hierarchy 111. The SSS network 131 is configured to output source nodes corresponding to the multiple masks based on the parent-child hierarchy 111 outputs. Therefore, the outputs of the SSS network 131 have an order that matches the parent-child hierarchy 111.

[0137] Furthermore, each mask output by the mask determination module 133 is provided to the source selector / interpolator module 135. The source selector / interpolator module 135 is configured to select a single mask, such as a mask corresponding to a child or parent node, or a subset of hierarchical masks to be output. The source selector / interpolator module 135 selects a mask from the mask set based on the target output (or target audio source) to be separated. In some embodiments, the source selector / interpolator module 135 selects a mask formed by the mask determination module 133 that corresponds to a node on a single path in a parent-child hierarchy.

[0138] Furthermore, the source selector / interpolator module 135 can interpolate between multiple masks to obtain, for example, 80% of the sub-sources mixed with 20% of the parent audio source. Each mask output by the source selector / interpolator module 135 is then multiplied by the complex spectrogram of the input audio mixture signal 125 to obtain a spectrogram associated with the target source signal, where the target source signal is associated with a hierarchical system trained by the SSS system 131 for its target source type (e.g., guitar).

[0139] The output of the source selector / interpolator module 135 is provided to the spectrum inverter module 137, whereby the audio source is extracted from the complex spectrum of the input audio mixture 125 and provided as the output signal 139. The spectrum inverter module 137 is configured to apply a mask selected by the source selector / interpolator module 135 to the spectrum to extract the audio source from the input audio mixture. In an exemplary embodiment, the source selector / interpolator module 135 interpolates at least two masks such that the spectrum inverter module 137 applies the interpolated mask to the spectrum to extract the audio source. In another exemplary embodiment, the spectrum inverter module 137 performs an inverse short-time Fourier transform to extract the audio source. This process is repeated for the mask corresponding to each level of the hierarchical system, or for any combination of sources returned by the source selector / interpolator module 135.

[0140] Figure 1GThis is a schematic diagram illustrating different components of a neural network 105 in an audio processing system 101 according to an exemplary embodiment, wherein the network 105 is trained for query-by-example (QBE) separation. A QBE network 147 obtains an input audio mixture 125 and an input query audio 141. The input audio mixture 125 is provided to a spectrogram extractor module 127, in which a complex spectrogram of the input audio mixture 125 can be determined. Furthermore, the input query audio 141 is provided to a query extractor 143, wherein the query extractor 143 processes the query audio signal to obtain anchor nodes. Anchor nodes are vectors used by the QBE network 147 to encode information in the input query audio signal 141 in a compact manner. Furthermore, anchor nodes (or anchor vectors) and complex spectrograms of the input audio mix 125 are provided to a cascade module 145, in which anchor nodes are cascaded with each frame of the input audio mix spectrogram and provided to a separation neural network 129 to dynamically extract sound sources from the input audio mix data 125 most similar to the query audio signal 141 provided to system 101. The separation neural network 129 produces an output representing a mask of audio sources most similar to the query audio signal in the parent-child hierarchy 111. The separation neural network 129 is similar to that used in... Figure 1F The SSS model in the model is a separate neural network, but with a larger input feature vector to accommodate the cascaded query anchor vector.

[0141] Figure 1H This is a schematic diagram illustrating different components of a neural network 105 trained for query-by-example (QBE) separation according to an exemplary embodiment, wherein the query is provided as an input label 149. The QBE network 147 receives an input audio mix 125 and a query in the form of an input label 149. The input label 149 may be text indicating a target audio source to be separated from the input audio mix 125. In another embodiment, the input label 149 may be a speech signal, wherein the speech signal may correspond to a target audio source to be separated from the input audio mix 125.

[0142] The input audio mixture signal 125 is provided to the spectrogram extractor module 127, where a complex spectrogram of the input audio mixture signal 125 can be determined. Furthermore, input tags 149 are provided to the tag extractor 151, where the tag extractor 152 processes the input tags 149 and determines the anchor nodes corresponding to the input tags 149. In an exemplary embodiment, anchor nodes for different audio sources can be predetermined and stored in memory. Therefore, upon receiving the input tag 149, the tag extractor 151 can extract the anchor nodes corresponding to the input tag 149. Anchor nodes are vectors used in the QBE network 147 to encode the information in the input tags 149 in a compact manner. Furthermore, anchor nodes (or anchor vectors) and complex spectrograms of the input audio mix 125 are provided to a cascade module 145, in which anchor nodes are cascaded with each frame of the input audio mix spectrogram and provided to a separation neural network 129 to dynamically extract sound sources from the input audio mix 125 most similar to the input label 149 already provided to system 101. The separation neural network 129 produces an output mask representing the audio source in the parent-child hierarchy 111 most similar to the input label 149. The separation neural network 129 is similar to... Figure 1G The separate neural network used in the SSS model.

[0143] Figure 1I The extraction of multiple musical instruments and the interrelationships between the extracted instruments are illustrated using a QBE network 147 according to an exemplary embodiment. In some embodiments, the QBE network 147 obtains an input query audio 141 and an input audio mix 125. It is assumed that the input audio mix 125 contains a hierarchical structure of instruments (lower left), and the query audio 141 is used to separate instruments at multiple levels of the hierarchical structure 159, where the closest target is located at the lowest level (e.g., a guitar). Based on the input query audio 141, the query network 147 generates anchor nodes. The query network 147 consists of two bidirectional long short-term memory (BLSTM) layers with 600 nodes in each direction and a discard rate of 0.3, followed by a fully connected layer with linear activation that maps each time-frequency interval to an embedding space with 20 dimensions. Furthermore, the anchor nodes and the input audio mix 125 are cascaded and provided to a masking network 155.

[0144] The masking network 155 is identical to the SSS model, but with a larger input feature vector to accommodate cascaded query anchors. The masking network 155 consists of four bidirectional long short-term memory (BLSTM) layers with 600 hidden units in each direction and a discard rate of 0.3, followed by a fully connected layer with sigmoid activation, which outputs a mask. The masking network 155 generates multiple hierarchical masks 157. These hierarchical masks 157 can be multiplied with the spectrogram of the input audio mix 125 to extract multiple instruments and the relationships between them, as shown in the multi-level hierarchical target source 159.

[0145] Figure 2A This is a flowchart illustrating the training of an audio processing system 200A for extracting multiple source-specific audio sources from an input audio mixture signal according to an exemplary embodiment. System 101 illustrates a general source separation scenario, wherein system 101 estimates multiple target audio signals from a mixture of target audio signals and potential other non-target sources (such as noise). The input audio mixture signal 205 includes the sum of multiple overlapping sources and is sampled from a training set 203 containing the original audio mixture signal and corresponding independent sources constituting the mixture 205. The source-specific independent sources 201 are independent source signals corresponding to the relevant hierarchical category (e.g., guitar) for which the SSS network 131 was trained.

[0146] The input audio mix 205 is processed by the spectrum estimator 207 to compute a time-frequency representation of the original mix. The spectrum is then input to the SSS network 131 using stored network parameters 209. The SSS network 131 makes a decision about the presence of each relevant source class in the hierarchical hierarchy at each time-frequency interval of the spectrum and estimates an amplitude mask set 211. An amplitude mask exists for each source in the relevant path of the hierarchical hierarchy, and an enhanced spectrum set 213 is computed by multiplying each mask by the complex time-frequency representation of the input audio mix 205. A set 215 of estimated audio signal waveforms is obtained by passing each of the enhanced spectrum 213 through a signal reconstruction process 217, where the signal reconstruction process inverts the time-frequency representation computed by the spectrum estimator 207.

[0147] The enhanced spectrogram 213 is then passed to the objective function 219, which can be used by the network training module 221 to update the network parameters 209. The network parameters are updated by comparing the enhanced spectrogram 213 with the target source using the tPSA objective in equation (3) and an update rule based on stochastic gradients, such as the ADAM algorithm. After passing through the hierarchical mixing module 223, the target source is obtained from the source-specific independent source 201. The hierarchical mixing module 223 combines independent audio sources from child nodes in the hierarchical system to generate a target audio signal corresponding to the parent audio source. For example, independent acoustic guitar audio signals and electric guitar audio signals are combined to create a complete guitar audio signal (parent), and then this complete guitar audio signal is further combined with a piano audio signal to create a mid-frequency string instrument audio signal (grandfather).

[0148] Figure 2B This is a flowchart illustrating the training of an audio processing system 200B for extracting multiple audio sources from an input audio mixture signal according to an exemplary embodiment of this disclosure, wherein the sources to be extracted are based on the content of a second query audio signal, i.e., a Query by Example (QBE). The training process is similar to... Figure 2A In the case of SSS (Sound Segmentation), however, from the training set of audio mix and independent sources 203, we sample not only the audio mix signal and the audio signals of the independent sources constituting the mix, but also the audio queries 225 corresponding to the independent source signals that are not among the source signals constituting the audio mix 205. In an alternative embodiment, the audio queries can be replaced by class labels. The spectrogram estimator 207 calculates the time-frequency representations of both the original mix and the original queries.

[0149] The QBE network 147 then uses the spectra from both the original mixing and the query to make a decision about the presence of each relevant source class in the hierarchical hierarchy at each time-frequency interval of the spectra, and estimates the amplitude mask set 211. For the query of the hierarchical hierarchy, an amplitude mask is determined for each source in the relevant path, and the enhanced spectra set 213 is computed by multiplying each mask by a complex time-frequency representation of the input audio mixing 205. The objective function 219 compares the enhanced spectra set 213 with the target source obtained from the hierarchical mixing module 223 using the query-related independent sources 227.

[0150] Figure 3This is a block diagram illustrating a single-channel mask inference network architecture 300 according to an exemplary embodiment of the present disclosure. A sequence of feature vectors obtained from the input audio mixture signal 205 (e.g., the logarithmic size of the short-time Fourier transform of the input audio mixture signal 205) is used as input to a hybrid encoder 301. For example, the dimension of the input vectors in the sequence can be F. The hybrid encoder 301 consists of multiple bidirectional long short-term memory (BLSTM) neural network layers, from the first BLSTM layer 303 to the last BLSTM layer 305. Each BLSTM layer consists of a forward long short-term memory (LSTM) layer and a backward LSTM layer, whose outputs are combined and used as input to the next layer. For example, the dimension of the output of each LSTM in the first BLSTM layer 303 can be N, and the input and output dimensions of each LSTM in all other BLSTM layers, including the last BLSTM 305, can both be N. The output of the last BLSTM layer 305 is used as input to an SSS network 131, which includes linear neural network layers 307 and nonlinear layers 309. For each time frame and each frequency in the time-frequency domain (e.g., the short-time Fourier transform domain), linear layer 307 uses the output of the last BLSTM layer 305 to output the number of C, where C is the number of target sources to be separated. Nonlinearity 309 is applied to the set of C values ​​for each time frame and each frequency to obtain a mask value indicating the dominance of each target source in the input audio mixture signal 205 for each time frame, each frequency, and each target source. Separation coding estimation from masking module 311 uses these masks and a representation of the input mixture in the time-frequency domain (e.g., the amplitude short-time Fourier transform domain) of the estimated mask to output the separation code for each target source. For example, separation coding estimation from masking module 311 can multiply the mask of the target source by the complex short-time Fourier transform of the input mixture to obtain an estimate of the complex short-time Fourier transform of the separated signal of that target source as if it had been observed independently.

[0151] train:

[0152] In an exemplary embodiment, in order to train the audio processing system 101 to extract multiple audio sources (instruments) from an audio mix of multiple instruments and further determine the relationships between the extracted multiple audio sources, the audio processing system 101 needs to be trained with a large dataset containing independent sources of many instruments that can be combined in a hierarchical manner. The dataset should include a wide variety of granular source labels, i.e., not only "guitar," but also "acoustic guitar," "electric guitar," and "effects guitar," etc., for each instrument in the dataset.

[0153] In some implementations, the Slakh2100 dataset is used to train the audio processing system 101. The Slakh2100 dataset includes 2100 music mixes and independent sources. This dataset contains 145 hours of mix data divided into 34 instrument categories. Furthermore, to enable the neural network 105 to learn a hierarchical system from the Slakh2100 dataset, the dataset is further modified to include an instrument hierarchy from the instrument categories included in Slakh. The hierarchy can be defined with three levels (excluding a general level consisting of mixes from all sources). The top level contains four categories: mid-range string and keyboard instruments (guitar, keyboard, and orchestral instruments), bass instruments (acoustic and electric bass), wind instruments (flute, reed instruments, and brass), and percussion instruments (drum kit and chromatic percussion). The middle level has seven categories (e.g., from mid-range string instruments: orchestral instruments, guitar, keyboard, and electric keyboard), and the lowest level has eighteen categories (e.g., from guitar: clean guitar and effects guitar).

[0154] In addition, in order to select examples from the dataset, such as Figure 4A As shown, multiple instantaneous submixes corresponding to different levels of the hierarchical system can be created for each orbital.

[0155] Figure 4A Table 1, according to an exemplary implementation, illustrates the contents of the hierarchical levels used for training and testing the hierarchical single-instrument source-specific separation (SSS) network. At the highest level, all pianos, guitars, and stringed instruments are considered a single source; at the next level, all guitars are considered a single source; and at the lowest level, only unaccented guitars are considered a single source. For each mix in the dataset, the salience of each hierarchical submixture within a 10-second chunk with a jump size of 2.5 seconds is calculated. A source in a submixture is considered salience if it has energy greater than -30 dB within that chunk. For experiments involving multiple levels, it can be ensured that for a given node, its parent source node (or grandparent source node) has energy from its child source nodes, not itself. In other words, it can be ensured that parent source nodes are not identical to child source nodes, meaning that some of the child source node's sibling or same-level nodes are also salience.

[0156] In an exemplary implementation, the Slakh200-split2 layering method is used, and the audio is downsampled to 16 kHz. Mixing audio sources (here, instruments) to generate an audio mix is ​​done on the fly, with blocks randomly selected from a pool of prominent examples for a specific experiment. For training, neural network 105 uses 20,000 examples per epoch (≈55.5 h), and an additional 3,000 examples (approximately 8.3 h) are used for validation and test sets. To ensure that a sufficient number of examples and a sufficiently rich hierarchy are used for training for the SSS network 131, audio sources are separated along the paths of the parent-child hierarchy 111, as shown in Table 1. In another implementation, the proposed method can be extended to other paths along other hierarchical structures. Furthermore, for the QBE network 147, each instrument type in the parent-child hierarchy 111 can be separated. Query audio blocks are selected from the pool of prominent blocks such that they always follow leaf nodes along the same path as the target (regardless of the target level), but originate from different tracks. In addition, to determine the output quality of our model, the scale-invariant source-to-distortion ratio (SI-SDR) was used. For reference, the SI-SDR is reported when the mixing is not processed.

[0157] In an exemplary implementation, four types of hierarchical source separation models can be evaluated. The source separation models vary along two dimensions: whether they are single-instrument (i.e., source-specific separation or SSS) or multi-instrument (i.e., per-example query or QBE), and whether they output a single rank or multiple ranks. Each configuration is described below:

[0158] Single instrument, single level: Three instrument-specific SSS models, each model corresponding to a level along a hierarchical path in the hierarchy.

[0159] Single instrument, multiple levels: An SSS model whose output mask is a set of levels, while all levels are separated in a single level path.

[0160] Multi-instrument, single-level: Three multi-instrument QBE models that output a mask at one level of a hierarchical system determined by the input query audio.

[0161] Multi-instrument, multi-level: A QBE model that outputs a hierarchical set of masks for each level of the hierarchy along a path determined by the input query.

[0162] For the single-instrument model, audio sources are separated along a single path in the hierarchical system, as shown in Table 1. The multi-instrument multi-level model is trained to separate sources based on queries, thus learning the entire hierarchical system (i.e., all instruments), instead of just learning one path as in the single-instrument multi-level case. For the multi-level model, the effect of constraints is tested. Furthermore, the extent to which the multi-level model utilizes limited data about leaf sources is tested. To do this, the three-level SSS 131 and QBE147 models are trained under the assumption that real leaf data is unavailable for 50% or 90% of the time, in which case only the upper levels directly participate in the objective function. For comparison, models with all nodes missing for 50% or 90% of the time during training are evaluated. These experiments illustrate how multi-level networks can utilize higher (i.e., coarser) levels of the parent-child hierarchical system at leaf nodes. This capability will be particularly advantageous because collecting data from fine-grained real data sources is generally more difficult than utilizing mixed and only broadly collected source component data.

[0163] All single-stage and multi-stage networks share the same architecture. The SSS model consists of four bidirectional Long Short-Term Memory (BLSTM) layers with 600 hidden units in each direction and a dropout rate of 0.3, followed by a fully connected layer with sigmoid activation that outputs a mask. The QBE model consists of two sub-networks: a query network and a mask network. The query network consists of two BLSTM layers with 600 nodes in each direction and a dropout rate of 0.3, followed by a fully connected layer with linear activation that maps each time-frequency interval to an embedding space with 20 dimensions. The mask network is identical to the SSS model, but with a larger input feature vector to accommodate the linked query anchors.

[0164] All models were trained for 100 epochs using the Adam optimizer at a learning rate of 1e-4 and a batch size of 25. The learning rate was halved if the loss of the validation set did not decrease within 5 direct epochs. If the norm of the mini-batch was higher than this value, the gradients were constrained to the 10th percentile of the historical gradient norm.

[0165] result:

[0166] Figure 4BTable 2, illustrating improvements in SI-SDR (dB) for hierarchical SSS (guitar) and QBE models according to an exemplary implementation, shows the improvements. Each model is trained either using a hierarchical constraint (HC) or without constraints on the masks generated for sources at different granularity levels. Table 2 investigates the impact of the hierarchical constraint (HC) on multi-level hierarchical networks. Furthermore, it is observed that for the Source-Specific Separation (SSS) network 131 (which separates only guitars in this case), HC slightly degrades performance across all levels, suggesting that the SSS model is able to learn specific hierarchical relationships for individual sources (guitars in this case) at different levels without additional assistance. For the By-Example Query (QBE) network 147 (which separates all types of instruments), HC slightly hinders performance at level 2 but greatly helps at leaf nodes (level 1). Furthermore, it is assumed that QBE network 147 provides more benefit because QBE network 146 cannot learn the specific mask "shape" for any individual source and therefore requires the additional assistance provided by HC. Therefore, in all subsequent experiments, HC was included for QBE network 147, but omitted for SSS network 131.

[0167] Figure 4C Table 3, based on an exemplary implementation, shows the results of the non-hierarchical and hierarchical SSS and QBE models according to SI-SDR (dB). In Table 3, Δ represents the improvement in noise mixing. The SSS network 131 was trained only to separate sources in the parent-child hierarchical system 111 containing unaccompanied guitars (see Table 1), while the QBE network 147 separates any source in the hierarchical system. Here, single-level networks (denoted by "1") are compared with multi-level networks (denoted by "3"). There is only one multi-level network for all three levels, but three single-level networks (one for each level). Table 3 expands on the results of Table 2 and compares the results of the single-level hierarchical model and the multi-level hierarchical model from both the SSS and QBE separation models. In both cases, the multi-level hierarchical network improves over the single-level model, gaining the greatest benefit at lower hierarchical levels. This means that the networks can leverage their shared knowledge of the hierarchical system to help themselves at lower levels where individual instruments are more difficult to identify in the mixing.

[0168] As can be seen from the results for Level 1 in Table 3, separating sources at such fine-grained levels of detail (e.g., clean electric guitar versus distorted electric guitar) is extremely difficult, especially for MIDI synthesis datasets such as Slakh2100, where several different instrument types may sound similar. In fact, a single network fails when attempting to train only these fine-grained sources, as indicated by the negative SI-SDR improvement in the #lvls=1 (single level) row for Level 1 sources. Training the network simultaneously at multiple levels alleviates this to some extent, although it has been noted that multi-level networks sometimes output nearly identical separated sources between Level 1 and Level 2. Furthermore, the highest output SI-SDR value is obtained when separating Level 3 sources in Table 3. Therefore, separating sources at finer-grained levels (1 and 2) is more difficult than it is typically attempted in music source separation.

[0169] Figure 4D Table 4, illustrating the SI-SDR improvement (dB) of the hierarchical SSS and QBE models (separated by thick dashed lines) relative to the unprocessed audio mix (“mix”), is shown according to an exemplary implementation. Each model was trained while removing only a specific percentage of the data leaves (“leaves”) or the entire example (“all”). Reducing only the leaf nodes to 90% shows a 0.3 dB decrease for the SSS network 131 and a 0.8 dB decrease for the QBE network 147 compared to using all leaves. Furthermore, the effect of removing leaf sources (level 1 sources, see the guitar example in Table 1) from the training set can be observed in Table 4. The performance degradation of reducing only the leaves is very small compared to reducing all data by 50% or 90%. Hierarchical multi-level networks can do well in separating lower levels when data is abundant at higher levels but sparse at lower levels. Similarly, for the SSS network 131 and the QBE network 147, even a small amount of leaf data helps avoid a significant performance degradation.

[0170] Therefore, this disclosure demonstrates the benefits of learning to separate audio sources simultaneously at different granular levels by reframing the audio source separation problem as a hierarchical process.

[0171] Figure 5 Anomaly detection is illustrated in an audio processing system 101 according to an exemplary embodiment. Figure 5The diagram illustrates a scenario 500 including a manufacturing production line 501, a training data pool 503, a machine learning model 505, and an audio processing system 101. The manufacturing production line 501 comprises multiple components, such as multiple engines working together to manufacture products. Furthermore, the production line 501 uses sensors to collect data. These sensors can be digital sensors, analog sensors, or combinations thereof. The collected data serves two purposes: some data is stored in the training data pool 503 and used as training data to train the machine learning model 503, while some data is used by the audio processing system 101 as operational time data to detect anomalies. Both the machine learning model 503 and the audio processing system 101 can use the same data.

[0172] To detect anomalies in manufacturing line 501, training data is collected. Machine learning model 505 uses training data from training data pool 503 to train neural network 105. Training data pool 503 may include labeled or unlabeled data. Labeled data is marked with labels, such as abnormal or normal, while unlabeled data has no label. Based on the type of training data, machine learning model 505 applies different training methods to detect anomalies. For labeled training data, supervised learning is typically used, while for unlabeled training data, unsupervised learning is typically applied. In this way, different implementations can handle different types of data. Furthermore, detecting anomalies in manufacturing line 501 includes detecting anomalies in each of the multiple engines included in manufacturing line 501.

[0173] Machine learning model 503 learns the features and patterns of training data, including normal and anomalous data patterns associated with audio sources. Audio processing system 101 uses a trained neural network 105 and collected operation time data 507 to perform anomaly detection, wherein operation time data 507 may include multiple audio source signals associated with multiple engines. Furthermore, system 101 includes a predetermined hierarchical relationship or order of different types of engines stored in a hierarchy of audio sources 107. The hierarchical order of different types of engines is predetermined in the hierarchy of audio sources 107, and the hierarchical order may be determined based on different audio characteristics of the engines (e.g., based on the frequency range of the sound produced by a particular engine, etc.).

[0174] Upon receiving operation time data 507, system 101 uses neural network 105, which is configured to subject the operation time data 507 to a hierarchical constraint of a parent-child hierarchy for the different audio sources included in the operation time data 507, such that a parent audio source in the parent-child hierarchy is a mixture of one or more of its child audio sources. Audio processing can extract different audio sources and their interrelationships from the operation time data 507 based on the parent-child hierarchy. Separating the engine sounds in a hierarchical order provides the relationships between the different engines included in manufacturing line 501. This allows the user to analyze the sound of each of the multiple engines, enabling the user to analyze manufacturing line 501 at a granular level.

[0175] Furthermore, the output of the audio processing system 101 is provided to an anomaly detector 523, which is configured to detect anomalies based on the extracted audio sources and the relationships between them. The anomaly detector 523 identifies the operation time data 507 as normal or abnormal. For example, using normal data patterns 509 and 511, the trained neural network 105 can classify the operation time data into normal data 513 and abnormal data 515. For example, operation time data X1 517 and X2 519 are classified as normal, and operation time data X3 521 is classified as abnormal. Once an anomaly is detected, the controller 525 can be used to take necessary actions 527 based on the detected anomaly.

[0176] Figure 6A A hierarchical audio separation method used in an audio scaling application, according to an exemplary embodiment, is illustrated. Audio scaling applications allow for greater focus on a specific sound source while using a hierarchical structure of the sound source to differentiate the ambient level from the audio mix. For example, audio separation system 101 receives a song as input, which includes the singer's voice and various instruments such as guitar, drums, piano, cello, and violin. Upon receiving the song, audio separation system 101 can extract the ontology of the different sounds from a predetermined hierarchical order (e.g., parent-child hierarchy 111).

[0177] In this example, the most specific extracted audio source 113c is a guitar, its parent 113b contains all mid-range string instruments—that is, guitar, piano, and violin—and its grandparent 113a contains all sources throughout the song. The hierarchical separation of sound allows the user to use audio scaling control 601 and focus more on the sound from the selected instrument (e.g., the guitar), reduce the focus to also hear sounds from similar instruments (i.e., the next level in the hierarchy, in this example, mid-range string instruments), or further reduce the focus to hear the original audio mix. Based on the user's scaling control 601 settings, a set of mix weights 603 is generated and fed to an audio mixer 605, which uses the mix weights 603 to combine each extracted audio source from the hierarchy 111 to generate an output signal for listening.

[0178] Figure 6B A scaling control knob 601 according to an exemplary embodiment is shown, wherein the position of the scaling control knob 601 is converted into a blending weight for audio scaling. The scaling control knob 601 includes a ratio indicator 609, which indicates the weights used to blend different audio sources extracted from the input audio mix at that particular level, wherein the audio sources are extracted using the audio processing system 101. For example, when the knob is set to position A, the blending weights are adjusted such that the audio mixer outputs only the child source (e.g., a guitar), as shown by blending weight 607a. In this case, weights are assigned only to the child audio source, and no weights are assigned to the grandparent and parent audio sources. Similarly, at position B, blending weight 607b is a blend of the child source (e.g., a guitar) and the parent source (e.g., a mid-range string instrument). Midpoint position C outputs only the parent source 607c, while position D 607d interpolates between the parent and grandparent sources, and position E 607e outputs only the grandparent source (e.g., the original audio mix). In this way, the audio processing system 101 outputs the extracted audio source at a specific level and proportionally to the indicated ratio.

[0179] Exemplary Implementation

[0180] Figure 7A block diagram of a computer-based audio processing system 700 according to some embodiments is shown. The computer-based audio processing system 700 includes multiple interfaces connecting the system 700 to other systems and devices. The system 700 includes an input interface 701 configured to receive an input audio mix 125 and an input query audio signal 141. The input device 705 may be a microphone. Alternatively or additionally, the computer-based audio processing system 700 may receive the input audio mix 125 and the input query audio signal 141 from various other types of input interfaces. In some embodiments, the system 700 includes an audio interface configured to obtain the input audio mix 125 and the input query audio signal 141 from the input device 705. In some other embodiments, the input interface 701 includes a network interface controller (NIC) 703 configured to obtain the input audio mix 125 and the input query audio signal 141 via a network 705, which may be one or a combination of a wired network and a wireless network.

[0181] A network interface controller (NIC) 703 is adapted to connect system 700 to network 705 via bus 707. Network 705 connects system 700 to sensing devices (e.g., input devices such as microphones). Alternatively or additionally, system 700 may include a human machine interface (HMI) 709. The HMI 709 within system 700 connects system 700 to a keyboard 711 and a pointing device / medium 713, wherein the pointing device / medium 713 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, etc.

[0182] System 700 includes a processor 109 configured to execute instructions stored in memory 103. Processor 109 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. Memory 103 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Processor 109 may be connected to one or more input and output devices via bus 707.

[0183] The instructions can implement a method for audio processing, wherein the method extracts multiple audio sources from an input audio mix 125 based on an input query audio 141. The extracted multiple audio sources produce sounds similar to the input query audio 141. For this purpose, computer memory 103 stores a neural network 105 trained to extract the multiple audio sources included in the input audio mix 125 from the input audio mix. The extracted multiple audio sources are subject to hierarchical constraints of audio characteristics, which enforce a parent-child hierarchy 111 of the extracted multiple audio sources. The parent-child hierarchy 111 of the extracted multiple audio sources is determined based on the hierarchy of audio sources 107, wherein the hierarchy of audio sources 107 is predetermined and includes a set of all possible audio sources associated with the input audio mix and the relationships between the audio sources in the set.

[0184] The audio processing system 700 includes an output interface 715 configured to present at least a portion of extracted audio sources. In another embodiment, the output interface outputs a notification when the neural network 105 determines that no audio source similar to the input query audio 141 exists in the hierarchy of audio sources 107. The notification may be a message displayed on a display device 717. In another embodiment, a user may be notified by generating a beep tone. In some other embodiments, the audio processing system 700 may not output audio to indicate silence, where such silence corresponds to an indication of no audio source, similar to the input query audio 141.

[0185] Examples of display devices 717 include computer monitors, televisions, projectors, or mobile devices. System 700 may also be connected to application interface 719, which is adapted to connect system 700 to external devices 721 for performing various tasks.

[0186] Figure 8 The steps of a method 800 for extracting multiple audio sources from an input audio mix, according to an exemplary embodiment, are shown. In step 801, an audio mix signal can be obtained. The input audio mix includes multiple audio signals from multiple different audio sources. For example, the input audio mix may include the sounds of various musical instruments, such as guitar, piano, drums, etc. The input audio mix signal can be provided to a neural network 105.

[0187] In step 803, multiple audio sources present in the input audio mix and the relationships between the extracted audio sources can be extracted, wherein the relationships are based on a parent-child hierarchy of the multiple audio sources (e.g., parent-child hierarchy 111). For this purpose, neural network 105 is trained to separate the audio sources from the audio mix, which is subject to a hierarchical constraint on the audio sources, such that a parent audio source in the parent-child hierarchy is a mix of one or more of its child audio sources. Therefore, neural network 105 extracts multiple audio source signals from the input audio mix. The input audio mix is ​​subject to a hierarchical constraint that implements a parent-child hierarchy of the multiple audio sources corresponding to the extracted multiple audio source signals. Therefore, in response to receiving the input audio mix in step 801, neural network 105 extracts multiple audio sources and presents at least a portion of the extracted multiple audio sources.

[0188] In step 805, at least a portion of the extracted audio sources can be presented. This portion can then be used to analyze each extracted audio source in more detail. This may help identify anomalies within the audio source itself.

[0189] Implementation

[0190] This specification provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing one or more of the exemplary embodiments. Various changes to the function and arrangement of the elements are contemplated without departing from the spirit and scope of the subject matter set forth in the appended claims.

[0191] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, the same reference numerals and names in the different figures indicate the same elements.

[0192] Furthermore, the various implementations can be described as processes using flowcharts, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations can be executed in parallel or simultaneously. Additionally, the order of operations can be rearranged. A process may terminate upon completion of its operations, but may have additional steps not discussed or included in the diagram. Furthermore, not all operations in any particular described process may occur in all implementations. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or the main function.

[0193] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. Manual or automatic implementation can be performed, or at least assisted by, the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. The processor can then perform the necessary tasks.

[0194] Furthermore, the embodiments of this disclosure and the functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in one or more combinations thereof. Additionally, some embodiments of this disclosure can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing device or for controlling the operation of a data processing device. Furthermore, program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or one or more combinations thereof.

[0195] According to embodiments of this disclosure, the term "data processing apparatus" can include various means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0196] Computer programs (which may also be referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program may, but must not, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, in a markup language document, a single file dedicated to the program in question, or in one or more scripts in a plurality of collaborative files, such as a file storing one or more modules, subroutines, or portions of code. A computer program can be deployed to execute on one or more computers located at a single site or distributed across multiple sites and interconnected by a communication network. Computers suitable for executing computer programs include, for example, central processing units that may be based on general-purpose or special-purpose microprocessors or both, or any other type. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic elements of a computer are the central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., hard disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0197] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0198] The embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or web browser through which a user interacts with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0199] Computing systems can include clients and servers. Clients and servers are typically geographically separated and usually interact via communication networks. The client-server relationship arises from computer programs running on their respective computers and having a client-server relationship with each other.

[0200] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adjustments and modifications may be made within the spirit and scope of this disclosure. Therefore, one aspect of the appended claims is to cover all such variations and modifications within the true spirit and scope of this disclosure.

Claims

1. An audio processing system, the audio processing system comprising: A memory configured to store a neural network trained to process audio mixtures to output an estimate of at least a subset of a set of audio sources present in the audio mixture, wherein the audio sources are subject to a hierarchical constraint of a parent-child hierarchy on the set of audio sources, such that a parent audio source in the parent-child hierarchy includes a mixture of one or more child audio sources of the parent audio source, and wherein the subset includes at least one parent audio source and at least one child audio source of the at least one parent audio source, the subset corresponding to a path from the root node to a leaf node on the parent-child hierarchy of the set of audio sources; An input interface configured to receive input audio mixes; A processor configured to process the input audio mixture using the neural network to extract estimates of subsets of the audio sources and their relationships according to the parent-child hierarchy; and An output interface configured to present the extracted audio sources and their interrelationships. The processor is linked to stored instructions for a module implementing the audio processing system, the audio processing system comprising: A spectrogram extractor module, configured to generate a spectrogram of the frame sequence of the input audio mixture; A mask determination module is configured to form the output of the neural network into a set of masks corresponding to audio sources extracted at different levels in the parent-child hierarchy, such that the mask-to-spectrum application extracts the corresponding audio sources, wherein the hierarchical constraint implements a mask assignment at a higher level in the parent-child hierarchy with at least the same amount of energy as the mask at a lower level in the parent-child hierarchy. A source selector module, configured to select a mask from the mask set based on the target output; and A spectrum inverter module configured to apply a selected mask to a spectrum to extract the audio source from the input audio mix. The hierarchical constraint implementation neural network assigns any time-frequency region in the spectrum of one or more sub-audio sources to the parent audio source.

2. The audio processing system of claim 1, wherein, The audio characteristics of at least one extracted parent audio source include the union of the audio characteristics of at least one extracted child audio source, wherein the audio characteristics of the audio source include one or a combination of spectral features, environmental information, pitch components, pitch-to-noise ratio, and cepstral peak amplitude.

3. The audio processing system of claim 1, wherein, The processor is configured to: The neural network is used to extract only the sub-audio sources corresponding to the leaf nodes of the parent-child hierarchy from the input audio mix; Extract one or more paths associated with the extracted child audio sources from the parent-child hierarchical system; as well as By mixing the child audio sources according to one or more paths leading to the parent node in the parent-child hierarchy, at least one parent audio source of the parent node in the parent-child hierarchy is reconstructed.

4. The audio processing system of claim 1, wherein, The processor is also configured to interpolate different audio sources by mixing the sub-audio sources using different weights applied to different sub-audio sources.

5. The audio processing system of claim 1, wherein, The input interface is configured to receive input data indicating a target audio source, and the processor is configured to extract the target audio source and the parent audio source of the target audio source from the input audio mix according to the parent-child hierarchy.

6. The audio processing system of claim 5, wherein, The input data includes the labels of the nodes in the parent-child hierarchy.

7. The audio processing system of claim 5, wherein, The input data includes a query audio signal, which enables the neural network to be trained to separate the child audio source closest to the query audio signal and the parent audio source of the child audio source according to the parent-child hierarchical system.

8. The audio processing system of claim 1, wherein, The source selector module selects a mask formed by the mask determination module, and the mask corresponds to a node on a single path in the parent-child hierarchy.

9. The audio processing system of claim 1, wherein, The source selector interpolates at least two masks such that the spectrogram inverter applies the interpolated mask to the spectrogram to extract the audio source.

10. The audio processing system of claim 1, wherein, The audio processing system also includes the following modules: A query extractor configured to process a query audio signal to obtain an anchor vector; and A cascaded module is configured to combine the anchor vector with each frame of the input audio mixing spectrogram, wherein the neural network is configured to receive the spectrogram combined with the anchor vector as input to produce an output representing a mask of the audio source most similar to the query audio signal in the parent-child hierarchy.

11. The audio processing system according to claim 1, wherein, The neural network is trained to simultaneously output masks of audio sources at different levels of the parent-child hierarchy that are subject to the hierarchical constraints, while minimizing the objective function of each level of the parent-child hierarchy.

12. The audio processing system according to claim 11, wherein, The objective function is the truncated phase-sensitive approximation tPSA objective function.

13. An anomaly detection system for controlling a machine, said anomaly detection system include: The audio processing system according to claim 1, wherein the audio processing system is configured to: Receives input audio mixes generated by the machine's components; as well as Extract multiple audio sources and their interrelationships based on the parent-child hierarchical system; An anomaly detector is configured to detect anomalies based on extracted audio sources and the relationships between them. as well as A controller configured to control the machine based on detected anomalies.

14. An audio scaling system, the audio scaling system include ： The audio processing system according to claim 1, and The parent-child hierarchy is indicated by a ratio indicator for different levels, wherein the audio processing system outputs the extracted audio source in proportion to the ratio indicated at a specific level.

15. An audio processing method, wherein, The audio processing method uses a processor connected to a memory storing a neural network trained to process audio mixtures to output an estimate of at least a subset of a set of audio sources present in the audio mixture, wherein the audio sources are subject to a hierarchical constraint of a parent-child hierarchy on the set of audio sources, such that a parent audio source in the parent-child hierarchy includes a mixture of one or more child audio sources of the parent audio source, and wherein the subset includes at least one parent audio source and at least one child audio source of the at least one parent audio source, the subset corresponding to a path from the root node to a leaf node in the parent-child hierarchy of the set of audio sources, wherein the processor is connected to stored instructions implementing the audio processing method, wherein the instructions, when executed by the processor, perform steps of the audio processing method, the audio processing method including the following steps: Receive and mix input audio; The neural network is used to process the input audio mixture to extract subsets of the audio sources and estimates of their relationships based on the parent-child hierarchical system; and Presenting the extracted audio sources and their interrelationships. The method further includes: Generate a spectrogram of the frame sequence of the input audio mixture; The output of the neural network is formed into a set of masks corresponding to the audio sources extracted at different levels in the parent-child hierarchy, such that the application of mask to spectrogram extracts the corresponding audio sources, wherein the hierarchical constraint implements a mask assignment at a higher level in the parent-child hierarchy with at least the same amount of energy as the mask at a lower level in the parent-child hierarchy. Select a mask from the mask set based on the target output; and The selected mask is applied to the spectrogram to extract the audio source from the input audio mix. The hierarchical constraint implementation neural network assigns any time-frequency region in the spectrum of one or more sub-audio sources to the parent audio source.

16. The audio processing method according to claim 15, wherein, The audio processing method further includes the following steps: The neural network is used to extract only the sub-audio sources corresponding to the leaf nodes of the parent-child hierarchy from the input audio mix; and By mixing the child audio sources according to one or more paths leading to the parent node in the parent-child hierarchy, at least one parent audio source of the parent node in the parent-child hierarchy is reconstructed.

17. The audio processing method according to claim 15, wherein, The audio processing method further includes the following steps: Receive input data indicating the target audio source, and Based on the parent-child hierarchy, the target audio source and the parent audio source of the target audio source are extracted from the input audio mix.

18. The audio processing method according to claim 17, wherein, The input data includes the labels of the nodes in the parent-child hierarchy.

Images