Method and system for target source separation - Patents.com

Abstract

An embodiment of the present disclosure discloses a system and method for extraction of a target acoustic signal. The system collects a mixture of acoustic signals. The system selects a query that identifies a target acoustic signal to be extracted from the mixture of acoustic signals, the query including one or more identifiers. Each identifier is present in a predetermined set of one or more identifiers and defines at least one of mutually inclusive and mutually exclusive characteristics of the mixture of acoustic signals. The system determines one or more logical operators that connect the extracted one or more identifiers. The system converts the one or more identifiers and the extracted logical operators into a digital representation. The system executes a neural network trained to extract the target acoustic signal by blending the digital representation with an intermediate output of an intermediate layer of the neural network.

Description

[Technical field]

[0001] The present disclosure relates generally to target sound source separation, and more particularly to an audio processing system for extracting a target sound from a mixture of sound signals. [Background technology]

[0002] Conventional source separation systems for extracting target acoustic signals are typically intended to isolate only certain types of sounds, such as for speech enhancement or instrument separation, where the target is determined by a training scheme and cannot be changed during testing. Conventional source separation approaches typically separate a speech mixture into only fixed sources (e.g., voices from background music) or separate all sources in the mixture without differentiating factors (e.g., each speaker in a conference room) and then use post-processing to find the target signal. In recent years, conditioning-based approaches have emerged as a promising alternative, where auxiliary inputs such as class labels can be used to indicate the desired sources, but the available sets of conditions are typically mutually exclusive and lack flexibility.

[0003] For example, in the cocktail party problem, humans have a supernatural ability to focus on sources of interest in complex acoustic scenes, and may change their focus target depending on the situation, relying on attention mechanisms that modulate cortical responses to auditory stimuli. The field of source separation has made great strides in replicating such abilities in machines, especially with the advent of deep learning approaches, but there is still a gap in terms of flexibility in allowing target source determination. As mentioned above, early work developed "special list" models that were intended to isolate only certain types of sounds. More recent work, such as deep clustering and permutation invariant training (PIT), has focused on separating all sources in a mixture without any discriminatory factors. However, in this case, the problem of determining which of the extracted sources is the unresolved source of interest still remains.

[0004] Therefore, there is a need for an advanced system that overcomes the above-mentioned drawbacks. To that end, there is a need for a technical solution to overcome the above-mentioned challenges. More specifically, there is a need for such a system for extracting a target acoustic signal that has better performance than conventional acoustic processing systems. Summary of the Invention

[0005] The present disclosure provides an improved sound processing system for identifying and extracting a target sound signal from a mixture of sounds. More specifically, the present disclosure provides a sound processing and training system configured to identify a target sound signal from a mixture of sounds based on mutually inclusive concepts such as loudness, gender, language, spatial location, etc.

[0006] To that end, some embodiments provide a conditional model configured to mimic human flexibility in selecting a target acoustic signal by focusing on extracting acoustics based on semantic concepts and criteria of different natures, i.e., heterogeneous, such as whether the speaker is close to or far from the microphone, speaks softly or loudly, or speaks in a particular language. Some embodiments are based on the recognition that a mixture of acoustic signals is collected from multiple sources. In addition, a query is collected that identifies a target acoustic signal to be extracted from the mixture of acoustic signals. The query is associated with one or more identifiers that indicate mutually inclusive characteristics of the target acoustic signals.

[0007] To that end, a mixture of acoustic signals is collected with the aid of one or more microphones from multiple sound sources, the multiple sound sources corresponding to at least one of one or more speakers, people or individuals, industrial equipment, and vehicles.

[0008] Further, each identifier present in the query having one or more identifiers belongs to a predetermined set of one or more identifiers and is extracted from the query, each extracted identifier defines at least one of mutually inclusive and mutually exclusive features of the target acoustic signal, and one or more logical operators are used to connect the extracted one or more identifiers.

[0009] Some embodiments are based on the recognition that the extracted one or more identifiers and one or more logical operators are converted into a digital representation, the digital representation of the one or more identifiers being selected from a set of predetermined digital representations of a plurality of combinations of the one or more identifiers.

[0010] To that end, the digital representation corresponds to a conditioning input, which may be represented in any manner, such as by a text input, a voice input, by a one-hot or multi-hot conditional vector, etc. The conditioning input comprises one or more of the mutually inclusive features of the target acoustic signal.

[0011] Some embodiments are based on the realization that a trained neural network is implemented to extract a target acoustic signal from a mixture of acoustic signals by mixing a digital representation with an intermediate output of an intermediate layer of the neural network. The neural network is trained for each set of predetermined digital representations of a plurality of combinations of one or more identifiers for extracting the target acoustic signal from the mixture of acoustic signals. To that end, during training, the extraction model is configured to generate one or more queries associated with one or more identifiers from the predetermined set of one or more identifiers.

[0012] To that end, in some embodiments, the neural network is based on an architecture comprising one or more interrelated blocks, where each block comprises at least a feature encoder, a conditioning network, a separation network, and a feature decoder. The conditioning network comprises a feature-invariant linear modulation (FiLM) layer that takes a mixture of acoustic signals as input and modulates the input to a conditioning input. The FiLM layer processes the conditioning input and sends the processed conditioning input to the separation network.

[0013] Accordingly, one embodiment discloses a computer-implemented method for extracting a target acoustic signal. The method includes collecting a mixture of acoustic signals from a plurality of acoustic sources. The method further includes selecting a query that identifies a target acoustic signal to be extracted from the mixture of acoustic signals. The method includes extracting each identifier present in a predetermined set of one or more identifiers from the query. The method includes determining one or more logical operators connecting the extracted one or more identifiers. The method further includes converting the extracted one or more identifiers and the one or more logical operators into a predetermined digital representation for querying the mixture of acoustic signals. The method includes executing a neural network trained to extract the target acoustic signal identified by the digital representation from the mixture of acoustic signals by combining the digital representation with an intermediate output of an intermediate layer of the neural network that processes the mixture of acoustic signals. The neural network is trained with machine learning to extract various acoustic signals identified in the set of predetermined digital representations. The method further includes outputting the extracted target acoustic signal.

[0014] Some embodiments provide an audio processing system configured to extract a target audio signal from a mixture of audio signals. The audio processing system comprises at least one processor and a memory having instructions stored thereon forming executable modules of the audio processing system. The at least one processor is configured to collect the mixture of audio signals. Additionally, the at least one processor is configured to collect a query that identifies a target audio signal to be extracted from the mixture of audio signals. The query includes one or more identifiers. The at least one processor is further configured to extract each identifier of the one or more identifiers from the query, each identifier being present in a predetermined set of one or more identifiers. Each identifier defines at least one of mutually inclusive and mutually exclusive characteristics of the mixture of audio signals. The at least one processor is configured to determine one or more logical operators connecting the extracted one or more identifiers. Additionally, the at least one processor is configured to convert the extracted one or more identifiers and the one or more logical operators into a predetermined digital representation for querying the mixture of audio signals. The at least one processor is further configured to execute a neural network trained to extract a target acoustic signal identified by the digital representation from the mixture of acoustic signals by combining the digital representation with an intermediate output of an intermediate layer of the neural network, and the at least one processor is further configured to output the extracted target acoustic signal.

[0015] Various embodiments disclosed herein provide an acoustic processing system that can extract a target acoustic signal from a mixture of acoustic signals more accurately, efficiently, and in less time. Additionally, various embodiments provide an acoustic processing system based on a neural network that can be trained to extract a target acoustic signal based on mutually inclusive and / or mutually exclusive features of the target acoustic signal. The neural network can be trained with a combination of mutually inclusive and / or mutually exclusive feature data sets in the form of a predefined set of one or more identifiers in a manner that is superior to existing neural networks.

[0016] Further features and advantages will become more readily apparent from the detailed description when taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram illustrating an environment for extraction of a target acoustic signal, according to some embodiments of the present disclosure. [Figure 2A] FIG. 1 is a block diagram illustrating an acoustic processing system for extracting a target acoustic signal, according to some embodiments of the present disclosure. [Figure 2B] FIG. 1 is a functional block diagram illustrating an acoustic processing system for extracting a target acoustic signal, according to some embodiments of the present disclosure. [Figure 2C] 1 is a block diagram illustrating a query interface of an audio processing system according to some embodiments of the present disclosure. [Figure 2D] 1A and 1B illustrate examples of a query interface of an audio processing system according to some embodiments of the present disclosure. [Figure 3A] FIG. 2 is a block diagram illustrating a method for generating a digital representation according to some embodiments of the present disclosure. [Figure 3B] 1 is a block diagram illustrating multiple combinations of one or more identifiers according to some embodiments of the present disclosure. [Figure 3C] 1 is a block diagram illustrating various types of digital representations in accordance with some embodiments of the present disclosure. [Figure 4] FIG. 2 is a block diagram illustrating a neural network in accordance with some embodiments of the present disclosure. [Figure 5A] FIG. 2 is a block diagram illustrating training of a neural network in accordance with some embodiments of the present disclosure. [Figure 5B] FIG. 2 is a block diagram illustrating training of a bridge-conditioned neural network in accordance with some embodiments of the present disclosure. [Figure 6] FIG. 2 is a block diagram illustrating an implementation of a neural network for extracting a target acoustic signal, according to some embodiments of the present disclosure. [Figure 7] FIG. 1 is a flow diagram illustrating training of a neural network, according to some embodiments of the present disclosure. [Figure 8] FIG. 2 is a flow diagram illustrating a method performed by an audio processing system to perform signal processing according to some embodiments of the present disclosure. [Figure 9] FIG. 1 is a block diagram illustrating an acoustic processing system for extracting a target acoustic signal, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] [Description of the embodiment] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, the apparatus and methods are shown only in block diagram form to avoid obscuring the present disclosure. It is contemplated that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0019] As used herein and in the claims, the words "for example," "for example," and "such as," as well as "comprising," "having," "including," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should be construed as open-ended. This means that the list should not be considered to exclude additional components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the terms and terminology used herein are for purposes of description and should not be considered as limiting. Any headings used herein are for convenience only and should not be construed as having any legal or limiting effect.

[0020] In the following description, specific details are given to enable a thorough understanding of the embodiments. However, those skilled in the art will appreciate that the embodiments may be practiced without these specific details. For example, systems, processes and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Additionally, like reference numbers and names in the various drawings refer to like elements.

[0021] The present disclosure provides an audio processing system configured to identify a target audio signal from a mixture of audio based on concepts, including mutually inclusive concepts such as loudness, gender, language, spatial location, etc. That is, the same target audio signal may be identified using multiple different such concepts. The audio processing system collects a mixture of audio signals and selects a query that identifies a target audio signal to be extracted from the mixture of audio signals. Further, the audio processing system extracts one or more identifiers associated with the target audio signal from the query. The one or more identifiers are indicative of features of the target audio signal, including mutually inclusive and mutually exclusive features of the target audio signal. The one or more identifiers are used as conditioning inputs and converted into a digital representation in the form of at least one of a one-hot conditional vector, a multi-hot conditional vector, a text input, or a voice input. Further, the digital representation of the conditioning input is utilized as an input to a neural network to extract the target audio signal from the mixture of audio signals. The neural network is trained to extract the target audio signal identified by the digital representation from the mixture of audio signals by combining the digital representation with intermediate outputs of an intermediate layer of the neural network that processes the mixture of audio signals. The neural network is trained with machine learning to extract a target acoustic signal identified in a set of predetermined digital representations. In addition, the neural network is trained based on an architecture having one or more associated blocks. The one or more associated blocks include at least one of a feature encoder, a conditioning network, a separation network, and a feature decoder. The conditioning network comprises a feature-invariant linear modulation (FiLM) layer that takes an encoded feature representation of a mixture of acoustic signals as an input and modulates the input based on a conditioning input in the form of a digital representation. The FiLM layer processes the conditioning input and sends the processed conditioning input to the separation network. Here, the target acoustic signal is separated from the mixture of acoustic signals.In addition, the FiLM layer repeats the process of sending conditioned inputs to the separation network to separate the target acoustic signal from the mixture of acoustic signals.

[0022] System Overview

[0023] 1 illustrates an environment 100 for extracting a target acoustic signal according to some embodiments of the present disclosure. The environment 100 includes multiple sound sources 102, one or more identifiers 104, one or more microphones 106, a mixture of acoustic signals 108, a network 110, and an audio processing system 112.

[0024] The multiple sound sources 102 may correspond to at least one of one or more speakers, such as a person or individual, industrial equipment, and vehicles. A mixture of sound signals 108 is collected from the multiple sound sources 102 with the aid of one or more microphones 106. Each sound signal in the mixture of sound signals 108 is associated with criteria or one or more identifiers 104 that define some characteristic of that sound signal in the mixture of sound signals 108. For example, the one or more identifiers 104 may be used to mimic human flexibility in selecting which sound sources to process by focusing on extracting sounds from the mixture of sound signals 108 based on semantic concepts and criteria of different nature, i.e., heterogeneity. These heterogeneity criteria include, in one example, whether the speaker is close or far from the one or more microphones 106, speaks softly or loudly, speaks in a particular language, etc. In this way, one or more identifiers 104 are associated with the multiple sound sources 102. Other examples of the one or more identifiers 104 include at least one of the loudest source, the quietest source, the furthest source, the nearest source, a female speaker, a male speaker, and a language-specific source, and the like.

[0025] A mixture 108 of acoustic signals associated with one or more of these identifiers 104 may be transmitted over a network 110 to an audio processing system 112 .

[0026] In one embodiment of the present disclosure, the network 110 is the Internet. In another embodiment of the present disclosure, the network 110 is a wireless mobile network. The network 110 includes a set of channels. Each channel in the set of channels supports a finite bandwidth. The finite bandwidth of each channel in the set of channels is based on the capacity of the network 110. Furthermore, the one or more microphones 106 are arranged in a pattern such that the acoustic signal of each of the multiple sound sources 102 is captured. The pattern of arrangement of the one or more microphones 106 enables the sound processing system 112 to estimate localization information of the multiple sound sources 102 using the relative time difference between the microphones. The localization information may be provided in the form of a direction of arrival of the sound or a distance from the one or more microphones 106 to the sound source.

[0027] In operation, the sound processing system 112 is configured to collect a mixture of sound signals 108 from the multiple sound sources 102. Additionally, the sound processing system 112 is configured to collect a query that identifies a target sound signal to be extracted from the mixture of sound signals 108. Furthermore, the sound processing system 112 is configured to extract from the query each identifier present in a predetermined set of one or more identifiers that define mutually inclusive and mutually exclusive features of the mixture of sound signals 108. The sound processing system 112 is further configured to determine one or more logical operators connecting the extracted one or more identifiers. The sound processing system 112 is further configured to convert the extracted one or more identifiers and the one or more logical operators into a predetermined digital representation for querying the mixture of sound signals 108. Furthermore, the sound processing system 112 is configured to execute the trained neural network to extract the target sound signal identified by the digital representation from the mixture of sound signals 108 by combining the digital representation with an intermediate output of an intermediate layer of the neural network that processes the mixture of sound signals 108. The sound processing system 112 is illustrated in further detail in Figures 2A and 2B.

[0028] 2A illustrates a block diagram of an acoustic processing system 112 for extracting a target acoustic signal 218 according to some embodiments of the present disclosure. The acoustic processing system 112 includes a memory 202, a processor 204, a database 206, a query interface 208, and an output interface 216. The memory 202 corresponds to at least one of RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or any other storage medium that can be used to store desired information and that can be accessed by the acoustic processing system 112. The memory 204 includes non-transitory computer storage media in the form of volatile and / or non-volatile memory. The memory 204 can be removable, non-removable, or a combination thereof. Exemplary memory devices include solid-state memory, hard drives, optical disk drives, etc. The memory 202 stores instructions executed by the processor 204. The memory 202 includes a neural network 210 and a transformation module 212. The memory 202 is associated with a database 206 of the sound processing system 112. The sound processing system 112 collects a mixture of sound signals 108 from a plurality of sound sources 102. The database 206 is configured to store the collected mixture of sound signals 108. The mixture of sound signals 108 corresponds to a mixture of sound signals having different characteristics, including but not limited to a sound source furthest from the one or more microphones 106, a sound source closest to the one or more microphones 106, a female speaker, a French speaker, etc. Additionally, the database 206 stores characteristics of each of the plurality of sound sources 102. In an embodiment, the database 206 is queried to extract a target sound signal 218 using a query interface 208 of the sound processing system 112. Additionally, the database 206 stores a predetermined set of one or more identifiers and a predetermined set of digital representations associated with the target sound signals 218.

[0029] The acoustic processing system 112 is configured, using a query interface 208, to collect a query that identifies a target acoustic signal 218 to be extracted from the mixture of acoustic signals 108.

[0030] Further, the audio processing system 112 is configured to extract, with the aid of an extraction model 214, from the collected queries, each identifier present within a predefined set of one or more identifiers. In one example, an identifier corresponds to any characteristic of a sound source, such as a "loudest" speaker, a "female" speaker, etc. The identifiers are extracted from the collected queries using the extraction model 214. The collected queries are utilized by the extraction model 214 for the analysis of the collected queries. The extraction model 214 is configured to identify each identifier from the collected queries based on the analysis of the collected queries. Each identifier present within a predefined set of one or more identifiers. Furthermore, the predefined set of one or more identifiers defines mutually inclusive and mutually exclusive characteristics of the mixture of audio signals 108. In one example, the predefined set of one or more identifiers is stored in the database 206. The predefined set of one or more identifiers is generated from a set of historical data associated with the mixture of audio signals 108. Additionally, a predetermined set of one or more identifiers may be generated from a set of historical data via one or more third party audio sources associated with the sound processing system 112 .

[0031] The collected queries may include multiple combinations of one or more identifiers. In one example, the collected queries may be “female” and “French” speaker, where “female” speaker and “French” speaker are two identifiers. The multiple combinations of the one or more identifiers are selected using one or more logical operators. The one or more logical operators enable the sound processing system 112 to select the multiple combinations of the one or more identifiers. The extraction model 214 is configured to determine one or more logical operators that connect each of the one or more identifiers 104 extracted from the predetermined set of one or more identifiers. In addition, the extraction model 214 is configured to generate one or more queries using the collected queries, the one or more identifiers 104, and the one or more determined logical operators. The one or more queries are further processed to generate conditioning inputs 222. In one example, the extraction model 214 generates queries such as “Which is the farthest acoustic signal?” and (&) “Which is the source of the English speech?”. Furthermore, the extraction model 214 utilizes all the queries and generates conditioning inputs 222. Here, the conditioning input 222 is, for example, "Which is the most distant source of the English speech?" The conditioning input 222 is an input that includes multiple combinations of one or more identifiers of the target acoustic signal 218. The conditioning input 222 corresponds to a processed query that includes multiple combinations of one or more identifiers.

[0032] Further, the conditioning input 222 is utilized by the transformation module 212. The transformation module 212 is configured to transform the extracted one or more identifiers 220 into a predetermined digital representation for querying the mixture of acoustic signals 108. In one example, the transformation module 212 selects the digital representation of the extracted one or more identifiers 220 from a set of predetermined digital representations of a plurality of combinations of the one or more identifiers of the target acoustic signal 218 (further described in FIG. 2B and FIG. 3A). Furthermore, the one or more identifiers 104 may be utilized to train the neural network 210 to extract the target acoustic signal 218 from the mixture of acoustic signals 108 by generating various training combinations of the one or more identifiers during training. Furthermore, the one or more identifiers 104 are utilized by the extraction model 214 to generate one or more queries. The one or more queries are associated with mutually inclusive and mutually exclusive features of the target acoustic signal 218 used during training of the neural network 210. The extraction model 214 is configured to execute the trained neural network 210 to extract a target acoustic signal from the mixture of acoustic signals 108 by combining a digital representation of the one or more identifiers 104 with an intermediate output of a hidden layer of the neural network 210. Further, the extracted target acoustic signal 218 is output from an output interface 216.

[0033] FIG. 2B shows a functional block diagram 200B of an audio processing system 112 for extracting a target audio signal 218 according to some embodiments of the present disclosure. The audio processing system 112 collects an audio signal mixture 108. The audio signal mixture 108 is collected from multiple audio sources 102 with the aid of one or more microphones 106 (as described in FIG. 1). The audio processing system 112 is configured to collect a query that identifies a target audio signal 218 to be extracted from the audio signal mixture 108 using a query interface 208. The query interface 208 is configured to accept one or more identifiers 104 associated with the target audio signal 218 that indicate mutually inclusive and mutually exclusive characteristics of the target audio signal 218 by collecting the query. The query is collected by the query interface 208. In an embodiment, the query is collected using a voice command with the aid of natural language processing techniques. The collected query is further analyzed (conditioning input 222) to identify one or more identifiers 104 for generating a processed query. For example, one identifier of the one or more identifiers 104 corresponds to the loudest speaker and another identifier of the one or more identifiers 104 corresponds to a female speaker, where the mutually inclusive features of the target acoustic signal 218 correspond to "loudest" and "female" and the processed query corresponds to "female loudest speaker." The target acoustic signal 218 associated with the one or more identifiers 104 exhibiting the mutually inclusive and mutually exclusive features is the source having the loudest female voice from all of the multiple sound sources 102 (described in more detail in FIG. 2C).

[0034] Further, the collected queries are utilized by the extraction model 214 to extract each identifier present within a predefined set of one or more identifiers that define mutually inclusive and mutually exclusive features of the mixture of acoustic signals 108. The extraction model 214 extracts each identifier to generate conditioning inputs 222 (as described above in FIG. 2A).

[0035] Further, the conditioning input 222 is utilized by the transformation module 212. The transformation module 212 transforms the conditioning input 222 into a digital representation 224 (the transformation module is further described in FIG. 3A). The digital representation 224 is further sent to the neural network 210 for training thereof to extract a target acoustic signal 218 (the digital representation is further described in FIG. 3C) and, at test time, to generate an output associated with the extracted target acoustic signal 218 from the mixture of acoustic signals 108. The neural network 210 is trained with machine learning with the help of one or more machine learning algorithms to extract various acoustic signals identified in a predetermined set of digital representations. The predetermined set of digital representations includes representations of various acoustic signals that may be extracted from a set of historical data or one or more third party sources. The target acoustic signal 218 is extracted from these various acoustic signals present in the predetermined set of digital representations. In one embodiment, the one or more machine learning algorithms used to train the neural network 210 may include, but are not limited to, a voice activity detection (VAD) algorithm and a deep speech algorithm. In general, a deep speech algorithm is used to automatically transcribe spoken speech. A deep speech algorithm takes digital speech as input and returns a "most likely" text transcript of the digital speech. Additionally, VAD is a technology that detects the presence or absence of human speech.

[0036]

number

[0037] In one embodiment, section 208A has a drop-down list that allows for the selection of an appropriate identifier, such as "French speaking speaker" 208aa, "English speaking speaker" 208bb, "Male speaker" 208cc, "Female speaker" 208dd, "Lowest speaker" 208ee, and "Softest speaker" 208ff. Section 208B has a drop-down list that allows for the selection of one or more logical operators. In one example, NOT operator 208c1 is selected and "Male speaker" 208cc is selected from section 208A. In addition, AND operator 208b1 is selected from a drop-down list in section 208B. Furthermore, "Lowest speaker" 208ee is selected from section 208A. The conditioning input 222 that is generated using the inputs selected from sections 208A, 208B, and NOT operator 208c1 corresponds to "should be the loudest speaker 208ee and not the male speaker 208cc." Additionally, the query interface 208 includes an "add identifier" section 208d for selecting multiple identifiers from the one or more identifiers 220. The "add identifier" section 208d may or may not be used.

[0038] Additionally, the query interface 208 has a voice interface 226 that allows a user to provide a voice command associated with the target acoustic signal 218. The voice command is analyzed using natural language processing techniques and one or more identifiers 104 (which may be multiple combinations 304 of one or more identifiers) are extracted from the voice command to generate conditioning inputs 222. The query interface 208 is utilized during training of the neural network 210 to accurately extract the target acoustic signal 218.

[0039] 2D illustrates an example of a query interface 208 of an audio processing system 112, according to some embodiments of the present disclosure. The query interface 208 has a section 228. The section 228 allows a user to type a logical expression 230 using one or more identifiers 220 and one or more logical operators. The one or more logical operators correspond to the logical operators described in FIG. 2C, such as the AND operator 208b1, the OR operator 208b2, and the NOT operator 208C1. In one example, the user has typed the logical expression 230. The logical expression 230 is represented as follows:

[0040] (French speaking speaker AND (NOT loudest speaker)) OR (male speaker)

[0041] The logic formula 230 indicates that the user is selecting a speaker who speaks French, who need not be the loudest speaker, and who may or may not be a male speaker.

[0042] Furthermore, the logical expressions 230 are not limited to those shown above.

[0043] FIG. 3A illustrates a block diagram of a method for generating a digital representation 224 according to some embodiments of the present disclosure. In the illustrated example, the database 206 includes a set of predefined digital representations 302. The set of predefined digital representations 302 may be extracted from one or more third-party databases. The set of predefined digital representations 302 may include a plurality of combinations 304 of one or more identifiers of the target acoustic signal 304. The plurality of combinations 304 of one or more identifiers correspond to at least two or more characteristics of a particular sound source. In one example, the plurality of combinations 304 of one or more identifiers include a “loudest” “female” speaker, a “softest” “male” “French speaking” speaker, etc. (further described in FIG. 3B). The set of predefined digital representations 302 is utilized by the conversion module 206 to convert the conditioning input 222 into the digital representation 224.

[0044] The conversion module 206 generates digital representations 224 of the conditioning inputs 222 of the extracted one or more identifiers 104 from the set of predetermined digital representations 302. For example, if the extracted one or more identifiers 104 correspond to the "loudest" "male" speakers, the conversion module 206 considers the "loudest" "male" speakers as identifiers and converts these identifiers into conditioning inputs for extracting the target acoustic signal 218 to generate digital representations 224 of the conditioning inputs.

[0045] 3B illustrates an example block diagram 300B of a plurality of combinations 304 of one or more identifiers according to some embodiments of the present disclosure. The plurality of combinations 304 of one or more identifiers may include, but is not limited to, a French-speaking male speaker 304a, a furthest English-speaking speaker 304b, a loudest Spanish-speaking speaker 304c, and a nearest female speaker 304d. The plurality of combinations 304 of one or more identifiers is not limited to the above examples.

[0046] 3C illustrates a block diagram 300C of a digital representation 224 according to some embodiments of the disclosure. The digital representation 224 corresponds to a transformed representation of the conditioning input 222. The digital representation 224 is represented by at least one of a one-hot conditional vector 306, a multi-hot conditional vector 308, a text description 310, or the like.

[0047] In one example, the digital representation 224 includes a one-hot conditional vector 306. If the conditioning input 222 is "extract the speaker furthest from the microphone," the one-hot conditional vector 306 would contain a "1" in the position corresponding to the furthest sound source in the vector of features of the acoustic signal, and zeros for all other terms, such as closest speaker, male / female, loud / soft, etc. In another example, the digital representation 224 includes a multi-hot conditional vector 308. If the conditioning input 222 is "extract the loudest female speaker," the multi-hot conditional vector 308 would contain ones in the positions corresponding to the loudest speaker and female speaker, and all other terms, such as male speaker, softer speaker, etc., would be set to zero.

[0048] In one example, the conditioning input 222 is converted at run time into a digital representation in the form of a one-hot vector or a multi-hot vector by one or more of the selections in a drop-down menu of possible options, by rule-based parsing of the text input, or by first converting the speech to text and then using rule-based parsing. In addition, logical operators such as "and" and "or" may be combined to generate multi-hot vectors between conditions to generate a multi-hot vector representation. In some embodiments, an additional one-hot dimension is added to indicate an "and" / "or" query for the generation of a digital representation for the conditioning input.

[0049] In yet another example, the digital representation 224 includes a text description 310, which is particularly important when the target acoustic signal 218 is not speech but a general sound source such as industrial equipment or a vehicle. In this situation, descriptions such as male / female and English / French cannot be used. In this case, the text description 310 is converted into an embedding vector, and then the embedding vector is input to the neural network 210 instead of the one-hot conditional vector 306. To generate the embedding vector from the text description 310, a model such as, for example, a word2vec model or a Bidirectional Representation For Transformer (BERT) model may be used. In general, word2vec is a technique for natural language processing. The word2vec model generally uses a neural network to learn word associations from a large corpus of text. In addition, the BERT model is designed to help a computer or machine understand the meaning of ambiguous language in a text by setting the context with the surrounding text. Regardless of the type of representation of the digital representation 224, the neural network 210 is trained and guided by the digital representation 224 to extract the target acoustic signal 218. The training of the neural network 210 is further described in Figures 4 and 5A.

[0050] 4 is a block diagram illustrating an architecture 400 of a neural network 210 according to some embodiments of the present disclosure. The neural network 210 may be a neural network trained to extract a target acoustic signal 218 and in some examples even extract localization information of the target acoustic signal 218. Furthermore, the training is based on the premise that the training data includes an unordered heterogeneous set of training data components. For example, in the case of the neural network 210, a digital representation 224 is generated to train the neural network 210 to extract a target acoustic signal 410. The target acoustic signal 410 is the same as the target acoustic signal 218 of FIG. 2.

[0051] The neural network 210 is trained to extract a target acoustic signal 410 from the mixture of acoustic signals 108 by combining the conditioning inputs 222 with intermediate outputs of an intermediate layer of the neural network 210. The neural network 210 includes one or more interrelated blocks, such as a feature encoder 404, a conditioning network 402, a separation network 406, and a feature decoder 408. In one example, the conditioning network 402 comprises a feature invariant linear modulation (FiLM) layer (described in FIG. 6).

[0052] The conditioning network 402 takes as input the conditioning input 222 converted to a digital representation 224. The conditioning network processes the digital representation 224, which identifies the type of sound source to be extracted from the mixture of acoustic signals 108, into a form useful to the separation network 406. The feature encoder 404 receives the mixture of acoustic signals 108. The feature encoder 404 corresponds to a trained one-dimensional convolutional feature encoder (ConvID) (described below in FIG. 6). Furthermore, the feature encoder 404 is configured to convert the mixture of acoustic signals 108 into a matrix of features for further processing by the separation network 406. The separation network 406 corresponds to a convolutional block layer (described in more detail in FIG. 6). The separation network 406 utilizes the conditioning input 222 and the matrix of features to separate the target acoustic signal 410 from the mixture of acoustic signals 108. The separation network 406 is configured to generate a latent representation of the target acoustic signal 410. The separation network 406 combines the conditioned inputs 222 with a matrix of features to generate a latent representation of a target acoustic signal 410 that is separated from the mixture of acoustic signals 108. The feature decoder 408 is typically the inverse process of the feature encoder 404, and converts the latent representation of the target acoustic signal generated by the separation network 406 into an audio waveform in the form of a target speech signal 410.

[0053] Neural network 210 undergoes a training phase which is further illustrated in FIG. 5A.

[0054] 5A illustrates an example block diagram 500A of training the neural network 210, according to some embodiments of the present disclosure. To that end, during training, a plurality of combinations 304 of one or more identifiers are provided to the neural network 210 for training 504 of the neural network 210. The plurality of combinations 304 of one or more identifiers are converted into a set of predetermined digital representations 302. In one embodiment, the neural network 210 is trained using the set of predetermined digital representations 302 of the plurality of combinations of one or more identifiers. Additionally, the neural network 210 is trained with training data 502.

[0055] In one example, the training data 502 includes a first training data set 502a and a second training data set 502b. The first training data set 502a includes acoustic data recorded in reverberant conditions and includes spatial data of the sound source relative to one or more microphones 106, but does not include data associated with the language of the sound source. The second training data set 502b has data for multiple languages, but was recorded in non-reverberant conditions. Thus, the second training data set includes language-related data for the sound source, but does not include spatial data of the sound source. The neural network 210 is trained using both the first training data set 502a and the second training data set 502b. Furthermore, the trained neural network 210 is configured to separate sound sources based on language in reverberant conditions by using conditioning inputs as described above, even if the combination was not present in the training data 502 during training 504. To enable this, the trained neural network 210 generates test mixtures 506 with all available combinations of feature conditions of the sound sources for a run 508 of the neural network 210. In one example, during run 508, if the requirements are language specific but the recorded sound is reverberant, the trained neural network 210 extracts the target sound source based on the requirements (language) even if no language-labeled reverberant data was present in the training data 502 during training 504. This is particularly useful when bridge conditions exist between two different training data sets.

[0056] FIG. 5B illustrates an example block diagram 500B of training the neural network 210 with a bridge condition 502 according to some embodiments of the present disclosure. A plurality of mutually inclusive feature combinations 304 are provided to the neural network 210 for training 504 of the neural network 210. The plurality of mutually inclusive feature combinations 304 are in the form of digital representations 224. Additionally, the neural network 210 is trained with training data 502. In one example, the training data 502 includes a first training data set 502a and a second training data set 502b. The first training data set 502a includes gender data of the sound source but does not include energy data. The second training data set 502b includes energy data and gender data. The neural network 210 is trained with the first training data set 502a and the second training data set 502b.

[0057] In one example, the bridge condition 502c is the loudest speaker in the mixture of audio signals. Such energy conditioning is convenient because it is easy to control the loudness of each sound source when generating the mixture of audio signals, and therefore training samples can be easily introduced into the training data set 502 in most cases. That is, a simple gain can be applied to the isolated sound source examples when generating the mixture of audio signals during training 504 of the neural network 210, thus making the ability to condition energy available to any data set. The terms loudness and energy are used interchangeably to represent some concept of the loudness of an audio signal.

[0058] The trained neural network 210 generates a test mixture 506 with all possible combinations of sound source feature conditions for the execution 508 of the neural network 210. In an example, if only the first training data set 502a is accessed to extract the gender-specific target sound source, the neural network 210 will be able to accurately extract the gender-specific target sound source thanks to the bridge condition 502c used in the training data. Even if the gender conditioning is not available in the first training data set 502a during training 504, the bridge condition allows for the gender conditioning in the first training data set 502a. In addition, during the execution 508, all possible conditions are available to extract the target sound source. The execution 508 of the neural network 210 is further described in FIG. 6.

[0059] FIG. 6 shows a block diagram 600 for the execution 508 of the neural network 210 to extract the target acoustic signal 218 according to some embodiments of the present disclosure. The neural network 210 inputs and outputs a time domain signal and includes the following components: (1) a trained one-dimensional convolutional feature encoder (ConvID) 606 (hereinafter, feature encoder 606) configured to obtain an intermediate representation, (2) a feature invariant linear modulation (FiLM) layer 602, (3) B intermediate blocks 604 for processing the intermediate representation, and (4) a trained one-dimensional transposed convolutional decoder 608 for converting back to the time domain signal. The FiLM layer 602 corresponds to the B-FiLM layer. The trained one-dimensional convolutional feature encoder (ConvID) 606 corresponds to the feature encoder 404 of FIG. 4. The B intermediate blocks 604 correspond to the convolutional block layer 604. The convolutional block layer 604 corresponds to the separation network 406 of FIG. 4. The convolutional block layer 604 is a stack of U-net convolutional blocks. Each U-net block contains several convolutional blocks that learn high-level latent representations and several transposed convolutional blocks that translate from the high-level latent representations back to a representation comparable to the U-net input. In one example, the combination of the FiLM layer 602 (B-FiLM layer) and B intermediate blocks 604 is repeated B times.

[0060] The mixture of acoustic signals 108 is sent to a feature encoder 606. The feature encoder 506 converts the mixture of acoustic signals 108 into a matrix of features for further processing by the FiLM layer 602 and the convolutional block layer 604. The FiLM layer 602 takes as input the matrix of features of the mixture of acoustic signals 108. In addition, the FiLM layer 602 takes as input the digital representation 224 (e.g., the one-hot conditional vector 306 shown in FIG. 3C). The FiLM layer 602 processes the input (the matrix of features and the one-hot conditional vector 306) and sends the processed input to the convolutional block layer 604. The convolutional layer 604 combines the matrix of features with the processed conditioning input to generate a latent representation of the target acoustic signal 218. The latent representation is sent to the trained one-dimensional transposed convolutional decoder 608 to separate the target acoustic signal 218 from other sound sources 610. The FiLM layer 602 and the convolutional block layer 604 are trained and executed to extract the target acoustic signal 218 and estimate localization information of the extracted target acoustic signal 218. The localization information of the target acoustic signal 218 indicates the location of the origin of the extracted target acoustic signal 218.

[0061]

number

[0062]

number

[0063]

number

[0064] 7 shows a flow diagram 700 illustrating the training of a neural network 210 to function as a heterogeneity separation model 712. The flow diagram 700 includes a database 206, an extraction model 214, a conditioning input 222 converted into a form of a digital representation 224, a negative example selector 702, a positive example selector 704, a voice mixer 706a, a voice mixer 706b, the neural network 210 functioning as the heterogeneity separation model 712, and a loss function 714. The database 206 is a speech database that includes a collection of disaggregated acoustic signals, such as various speech signals for human voice applications, and associated metadata for each disaggregated acoustic signal, such as speaker-to-microphone distance, signal level, language, etc.

[0065] The extraction model 214 is configured to generate one or more random queries associated with mutually inclusive features of the target acoustic signal 610. The one-hot conditional vector 306 or the multi-hot conditional vector 308 of the received one or more identifiers 220 are randomly selected based on the generated one or more random queries. The multi-hot conditional vector 308 can be a multi-hot "and" conditioning vector or a multi-hot "or" conditioning vector. In the case of multi-hot "and" conditioning, all selected identifiers must be true for the acoustic signal to be the associated target acoustic signal. In the case of multi-hot "or" conditioning vector, at least one of the selected identifiers must be true. For the conditioning of the text description 310, all acoustic signals in the database 206 are required to have one or more natural language descriptions of the corresponding acoustic signal.

[0066] In one example, audio signals are randomly selected from the database 206 as positive examples, and the corresponding textual descriptions are used as conditioning inputs 222. The conditioning inputs 222 converted to digital representations 224 are sent to the heterogeneous separation model 712 for further processing. The conditioning inputs 222 converted to digital representations 224 are sent to the negative example selector 702 and the positive example selector 704. The negative example selector 702 returns zero, one, or more acoustic signals from the database 206 that are not related to the given conditioning inputs used to train the heterogeneous separation model 712 on one or more random queries. In one embodiment, the negative example selector 702 may return zero acoustic signals or unrelated acoustic signals so that the heterogeneous separation model 712 can be robust during inference.

[0067] The positive example selector 704 returns zero, one, or multiple acoustic signals from the database 206 that are relevant to a given conditioning input. In some cases, it is important to have the positive example selector return audio signals with zero relevance so that the heterogeneous target acoustic extraction model during inference can be robust to this case.

[0068] Zero, one, or more acoustic signals from the positive example selector 704 are passed to an audio mixer 706a to obtain ground truth target acoustic signals for training. The acoustic signals returned from both the positive example selector 704 and the negative example selector 702 are also passed to an audio mixer 706b to generate an audio mixture signal 708 during training that is input to a heterogeneous separation model 712. The heterogeneous separation model 214 processes the digital representation 224 and the audio mixture signal 708 to extract a separated target acoustic signal 716.

[0069] The ground truth target speech signal is compared with the separated target acoustic signal 716 with the help of a loss function 714. In other words, the loss function 714 compares the ground truth target speech signal 710 with the separated target acoustic signal 716 returned by the heterogeneous separation model 712 with the loss function 714. In an example, the relevant loss function comparing the two acoustic signals (e.g., SNR, scale invariant source-to-distortion ratio, mean squared error, etc.) may be calculated in the time domain, frequency domain, or a weighted combination of time and frequency domain losses.

[0070] In some examples, there may be several sound sources, such as multiple people talking, such as in a business meeting or party, and a machine listening device (e.g., a robot or a hearing aid-like device) that can focus on the speech of a particular person may be needed. However, the machine listening device requires input from the user to identify which person to focus on, which is often context-dependent. For example, if two people are having a conversation, one male and one female, the user may provide the machine listening device with input to focus on the male speaker. In some examples, the machine listening device includes an audio processing system 112 that uses a neural network 210 to perform the task of identifying the target acoustic signal using a heterogeneity separation model 712. If both speakers are male, a heterogeneity separation model 712 is utilized to describe the target person's speech, such as how far they are from the microphone or the volume of the speech of both speakers compared to other competing speakers. The heterogeneous separation model 712 allows the control device to select the signal features for a given mixture of speakers that are best suited to isolate the speaker of interest (a particular sound source) given the particular circumstances.

[0071] The heterogeneous separation model 712 is trained to perform separation based on multiple criteria as described above. Typically, the source separation model is trained using a mixture / target pair, where two or more decoupled source signals (e.g., speech waveforms) are combined to generate a mixture, and the decoupled signal is used as the target. This combination, also called a mixing process, takes each decoupled source signal, optionally applies some basic signal processing applications (e.g., applies gain, equalization, etc.) to the decoupled sources, and then combines them to obtain a target speech mixture signal. The processed decoupled sources then serve as training targets for a given speech mixture signal. However, the heterogeneous separation model 712 uses a triplet that includes (1) the speech mixture signal, (2) a digital representation represented, for example, by a one-hot conditional vector, and (3) a target signal that corresponds to the description represented by the one-hot conditional vector.

[0072] Another example of the Heterogeneous Separation Model 712 system may be combined with a system that uses multiple criteria to identify the signal features of all speakers present in a mixed signal without dissociation. For example, the gender of the speaker and the language being spoken may be detected even when the speech is overlapping. The identified values ​​of these criteria are used to conditionally extract the dissociated signals of the speakers present in the speech mixture. Furthermore, the various criteria present in the speech mixture are combined using a process similar to a "logical and" (i.e., a one-hot vector becomes a multi-hot with a "1" in the position of all relevant criteria) to separate the signals using all the criteria. Also, each of the criteria may be used independently to evaluate which conditioning criterion results in the best target signal separation performance for a given mixture.

[0073] 8 shows a flowchart 800 illustrating a method for identifying a target acoustic signal based on the above-described embodiment according to some embodiments of the present disclosure. The method 800 is executed by the acoustic processing system 112. The flowchart begins at step 802. After step 802, in step 804, the method includes collecting a mixture of acoustic signals 108 from a plurality of sound sources 102 with the aid of one or more microphones 106. The plurality of sound sources 102 correspond to at least one of a speaker, a person or individual, industrial equipment, and a vehicle. The mixture of acoustic signals 108 is collected from the plurality of sound sources 102 with the aid of one or more microphones 106 together with one or more identifiers 104.

[0074] In step 806, the method includes collecting, with the aid of a query interface 208 (as described in FIG. 2 ), a query that identifies a target acoustic signal 218 to be extracted from the mixture of acoustic signals 108. The query indicates mutually inclusive and mutually exclusive characteristics of the target acoustic signal 218. The query is associated with one or more identifiers 104 of the multiple audio sources 102. The one or more identifiers 104 include at least one of the loudest source, the quietest source, the furthest source, the nearest source, a female speaker, a male speaker, and a language-specific source.

[0075] In step 808, the method includes extracting from the query each identifier present in a predetermined set of one or more identifiers defining mutually inclusive and mutually exclusive features of the mixture of acoustic signals 108 with the aid of an extraction model 214 (as illustrated in FIG. 2B). After step 808, in step 810, the method includes determining one or more logical operators connecting the extracted one or more identifiers 220 using a query interface 208 (as illustrated in FIG. 2C).

[0076] In step 812, the method includes converting the extracted one or more identifiers 220 into a digital representation 224 with the aid of a conversion module 206 (as described in Figures 3A and 3C). The conversion module 206 is configured to generate the digital representation 224 of the extracted one or more identifiers 220 from a set 302 of predetermined digital representations of a plurality of combinations 304 of mutually inclusive features of the target acoustic signal 218. The digital representation 224 is represented by a one-hot conditional vector 306, or a multi-hot conditional vector 308, and a text description 310 (as described in Figure 3C).

[0077] In step 814, the method includes executing the trained neural network 210 to extract the target acoustic signal 218 from the mixture of acoustic signals 108 with the aid of an extraction model 214. Additionally, the extraction model 214 is configured to generate one or more queries associated with mutually inclusive and mutually exclusive features of the target acoustic signals during training of the neural network 210. The neural network 210 is trained with a set 302 of predetermined digital representations of a plurality of combinations 304 of mutually inclusive features to extract the target acoustic signal 218. Additionally, the neural network 210 is trained to generate localization information of the target acoustic signal 218 indicative of a location of an origin of a source of the plurality of sound sources 102 of the target acoustic signal 218.

[0078] In step 816, the method includes outputting the extracted target acoustic signal together with the localization information with the aid of the output interface 216. The method ends in step 814.

[0079] 9 illustrates a block diagram 900 of an audio processing system 112 for performing processing of a mixture of audio signals 108 according to some embodiments of the present disclosure. In some exemplary embodiments, the block diagram 900 includes one or more microphones 106 that collect data including the mixture of audio signals 108 of multiple audio sources 102 from an environment 902.

[0080] The audio processing system 112 includes a hardware processor 908. The hardware processor 908 is in communication with a computer storage memory, such as memory 910. The memory 910 includes stored data, including algorithms, instructions, and other data implemented by the hardware processor 908. It is contemplated that the hardware processor 908 may include more than one hardware processor depending on the requirements of a particular application. The more than one hardware processor may be internal or external. The audio processing system 112 may be incorporated into other components, including output interfaces and transceivers, among other devices.

[0081] In some alternative embodiments, the hardware processor 908 is connected to a network 904 that communicates with the mixture of audio signals 108. The network 904 includes, by way of non-limiting example and not limitation, one or more local area networks (LANs) and / or wide area networks (WANs). The network 904 also includes enterprise-wide computer networks, intranets, and the Internet. The audio processing system 112 includes one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources comprises one or more devices that cooperate in the distributed environment of the network 904.

[0082] In some other alternative embodiments, the hardware processor 908 is connected to a network-enabled server 914 that is connected to the client device 916. The network-enabled server 914 corresponds to a dedicated computer connected to a network that executes software intended to process client requests received from the client device 916 and provide appropriate responses on the client device 916. The hardware processor 908 is connected to an external memory device 918 that stores all necessary data used for the target acoustic signal extraction, and to a transmitter 920. The transmitter 920 facilitates the transmission of data between the network-enabled server 914 and the client device 916. Furthermore, an output 922 associated with the target acoustic signal and localization information of the target acoustic signal is generated.

[0083] The mixture of acoustic signals 108 is further processed by a neural network 210. The neural network 210 is trained with mutually inclusive feature combinations 906 of each acoustic signal. The multiple mutually inclusive feature combinations 906 are provided to the neural network 210 for training thereof (as described in FIG. 7 ). The multiple mutually inclusive feature combinations 906 are in the form of a digital representation 224.

[0084] Many variations and other embodiments of the disclosures described herein will occur to those skilled in the art to which these disclosures pertain having the benefit of the teachings presented in the above description and the associated drawings. It is to be understood that the disclosure should not be limited to the specific embodiments disclosed, and that variations and other embodiments are intended to be included within the scope of the appended claims. Furthermore, while the above description and the associated drawings describe exemplary embodiments in the context of certain illustrative combinations of elements and / or functions, it should be recognized that various combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, combinations of elements and / or functions other than those expressly described above are also contemplated, for example, as may be set forth in some of the appended claims. Although certain terms are used herein, they are used in an inclusive and descriptive sense only, and not for purposes of limitation.

Claims

1. 1. An acoustic processing system for extracting a target acoustic signal, comprising: At least one processor; and a memory having instructions stored thereon, the instructions, when executed by the at least one processor, causing the audio processing system to: collecting a mixture of acoustic signals together with the target acoustic signal; collecting a query that identifies the target acoustic signal to be extracted from the mixture of acoustic signals, the query including one or more identifiers; and extracting from the query each identifier of the one or more identifiers, each identifier being in a predetermined set of one or more identifiers, each identifier defining at least one of mutually inclusive and mutually exclusive characteristics of the mixture of acoustic signals; and determining one or more logical operators connecting the extracted one or more identifiers; converting the extracted one or more identifiers and the one or more logical operators into a predetermined digital representation for querying the mixture of acoustic signals; and executing a neural network trained to extract the target acoustic signals identified by the digital representation from the mixture of acoustic signals by combining the digital representation with an intermediate output of an intermediate layer of the neural network that processes the mixture of acoustic signals, the neural network being trained by machine learning to extract distinct acoustic signals identified in a predetermined set of digital representations; and an acoustic processing system configured to output the extracted target acoustic signal (as shown in FIGS. 1, 2A and 2B).

2. 2. The sound processing system of claim 1, wherein the sound signals in the mixture of sound signals are collected from a plurality of sound sources with the aid of one or more microphones, each sound source of the plurality of sound sources corresponding to at least one of a speaker, a person or individual, industrial equipment, a vehicle, or a natural sound (FIG. 1).

3. 2. The sound processing system of claim 1, wherein the predetermined set of one or more identifiers is associated with a plurality of sound sources, and each of the one or more identifiers in the predetermined set of one or more identifiers includes at least one of a loudest sound source identifier, a quietest sound source identifier, a furthest sound source identifier, a nearest sound source identifier, a female speaker identifier, a male speaker identifier, and a language-specific sound source identifier (FIG. 1).

4. 2. The acoustic processing system of claim 1, wherein the one or more identifiers are combined using the one or more logical operators to extract the target acoustic signal having mutually inclusive and mutually exclusive characteristics, the one or more logical operators including at least one of a NOT operator, an AND operator, and an OR operator, and a NOT operator is used with any single identifier of the one or more identifiers.

5. 2. The acoustic processing system of claim 1, wherein the neural network is trained using a predetermined set of the digital representations of multiple combinations of identifiers within the predetermined set of the one or more identifiers (FIGS. 5A, 5B).

6. The sound processing system of claim 1 , wherein the neural network is trained with a positive example selector and a negative example selector to extract the target sound signal (shown in FIG. 7 ).

7. The audio processing system of claim 1 , wherein the digital representation is represented by at least one of a one-hot conditional vector, a multi-hot conditional vector, and a textual description ( FIG. 3C ).

8. 2. The audio processing system of claim 1, wherein the intermediate layer of the neural network includes one or more associated blocks, each of which includes at least one of a feature encoder, a conditioning network, a separation network, and a feature decoder, the conditioning network comprising a feature-invariant linear modulation (FiLM) layer that takes the mixture of audio signals and the digital representation as inputs and modulates the inputs into conditioned inputs, the FiLM layer processing the conditioned inputs and sending the processed conditioned inputs to the separation network (FIG. 6).

9. 9. The sound processing system of claim 8, wherein the separation network includes a convolutional block layer that utilizes the conditioning input to separate the target sound signal from the mixture of sound signals, the separation network being configured to generate a latent representation of the target sound signal (FIGS. 4, 6).

10. The sound processing system of claim 8 , wherein the feature decoder converts the latent representation of the target sound signal produced by the separation network into an audio waveform ( FIG. 6 ).

11. 1. A computer-implemented method for extracting a target acoustic signal, comprising: collecting a mixture of acoustic signals from a plurality of sound sources; and selecting a query that identifies the target acoustic signal to be extracted from the mixture of acoustic signals, the query including one or more identifiers, the method further comprising: extracting from the query each identifier of the one or more identifiers, each identifier being in a predetermined set of one or more identifiers, each identifier defining at least one of mutually inclusive and mutually exclusive characteristics of the mixture of acoustic signals, the method further comprising: determining one or more logical operators connecting the one or more extracted identifiers; converting the extracted one or more identifiers and the one or more logical operators into a predetermined digital representation for querying the mixture of acoustic signals; and performing a neural network trained to extract the target acoustic signals identified by the digital representations from the mixture of acoustic signals by combining the digital representations with an intermediate output of an intermediate layer of the neural network that processes the mixture of acoustic signals, the neural network being trained by machine learning to extract the target acoustic signals identified in a predetermined set of digital representations, the method further comprising: A computer-implemented method comprising outputting the extracted target acoustic signal.

12. 12. The computer-implemented method of claim 11, wherein the mixture of acoustic signals is collected from multiple sound sources with the aid of one or more microphones, the multiple sound sources corresponding to at least one of a speaker, a person or individual, industrial equipment, and a vehicle.

13. 12. The computer-implemented method of claim 11, wherein the predetermined set of one or more identifiers is associated with a plurality of sound sources, and each of the one or more identifiers in the predetermined set of one or more identifiers includes at least one of: a loudest sound source identifier, a quietest sound source identifier, a furthest sound source identifier, a nearest sound source identifier, a female speaker identifier, a male speaker identifier, and a language-specific sound source identifier.

14. 12. The computer-implemented method of claim 11, wherein the one or more identifiers are combined using the one or more logical operators to extract the target acoustic signals having mutually inclusive and mutually exclusive characteristics.

15. 15. The computer-implemented method of claim 14, wherein the neural network is trained with the set of predetermined digital representations of multiple combinations of identifiers within the predetermined set of one or more identifiers.

16. 12. The computer-implemented method of claim 11, further comprising generating one or more queries associated with the mutually inclusive and mutually exclusive features of the target acoustic signal during training of the neural network.

17. 12. The computer-implemented method of claim 11, wherein the intermediate layer of the neural network includes one or more associated blocks, each of the one or more associated blocks including at least one of a feature encoder, a conditioning network, a separation network, and a feature decoder, the conditioning network comprising a feature invariant linear modulation (FiLM) layer that takes the mixture of acoustic signals as an input and modulates the input into a conditioning input, the FiLM layer processes the conditioning input and sends the processed conditioning input to the separation network.

Images