Method and system for sound event localization and detection

Abstract

An embodiment of the present disclosure discloses a system and method for localization of a target sound event. The system collects a first digital representation of an acoustic mixture of sounds of a plurality of sound events by using an acoustic sensor. The system receives a second digital representation of the sounds corresponding to the target sound event. The first digital representation and the second digital representation are further processed by a neural network to generate localization information indicative of a location of a source of the target sound event relative to a location of the acoustic sensor.

Description

[Technical Field]

[0001] The present disclosure relates generally to sound event localization and detection (SELD), and more particularly to a sound event localization and detection system for estimating the direction of arrival of a sound event. [Background technology]

[0002] Existing systems for SELD typically operate by estimating source locations for all classes at each time point. However, SELD remains challenging because sound sources often move, become silent, be obscured by room reflections, or be mixed with interfering sounds. Furthermore, many sound events are easily confused, further complicating the task of SELD.

[0003] Sound localization information, especially the direction of arrival (DOA) estimation, and further classification of the type of event occurring at the source of the sound, are considered to be an important type of front-end processing used in monitoring and robotics applications. Some existing SELD systems follow a pipeline consisting of two main stages. The first stage extracts the spectral and spatial features of the received sound from a microphone array. The second stage uses a deep neural network (DNN) to learn a mapping from multi-channel input features to two output targets: (1) a vector of class probabilities indicating the presence of each class at each time point, and (2) a DOA vector containing the location information of each class. In this case, training the DNN is challenging because the contributions from the two output branches must be balanced.

[0004] Some existing SELD systems, including those based on activity coupled Cartesian direction of arrival (ACCDOA) representations, typically assume a small, fixed set of sound event classes to detect and localize. In practice, if there are 12 classes, the DNN will output 12 ACCDOA representations at each time point. This approach can become impractical for large numbers of classes, and in some situations, it may not be necessary to localize all classes at all times.

[0005] Therefore, training a class-specific system to localize only sound events from a single class allows for focusing on a particular class. However, training class-specific models can be challenging because there may not be enough data available for each class of interest to properly train all models.

[0006] Therefore, there is a need for a SELD system that overcomes the above-mentioned drawbacks.

[0007] Therefore, there is a need for a technical solution to overcome the above-mentioned challenges, and more specifically, for such a SELD system that outperforms conventional sound event detection and localization in the presence of directional interference. Summary of the Invention

[0008] This disclosure provides a class-conditional SELD system for sound event localization. As mentioned above, existing systems for SELD typically operate by estimating source locations for all classes at each time point. However, SELD remains challenging because sound sources often move, become silent, be obscured by room reflections, or be mixed with interfering sounds. In addition, many sound events are easily confused, further complicating the task of SELD.

[0009] Some embodiments are based on the recognition that current SELD systems have only one ACCDOA vector output at each time instant, and the class represented by this output is determined based on inputs describing the type of sound event that needs to be located. Some existing SELD systems, including those based on ACCDOA representations, typically assume a small, fixed set of sound event classes to detect and localize. For example, 12 classes would result in 12 ACCDOA representations being output at each time instant.

[0010] Some embodiments recognize that outputting a single ACCDOA representation per class may become impractical when the number of classes is large, and that in some situations it may not be necessary to localize all classes all the time. Therefore, training a class-specific system to localize only sound events from a single class allows for focusing on a specific class. However, training class-specific models can be difficult because each class of interest requires very large training data to properly train all models.

[0011] As such, various embodiments provided herein provide methods and systems for localizing sound events from a single target class and identifying target sources based on the localized sound events.

[0012] Some embodiments are based on the recognition that a SELD system can be trained as a class-specific system for localizing sound events from a single class and identifying a target sound event based on the localized sound events. The SELD system is further utilized to estimate the DOA of the target sound event and the distance between the source of the target sound event and the sound source. To this end, the SELD system collects a first digital representation of a sound mixture detected by an acoustic sensor. The sound mixture is associated with multiple sound events. The SELD system further receives a second digital representation of the sound corresponding to the target sound event. The second digital representation is utilized to identify the target sound event from among the multiple sound events. The first and second digital representations are processed by a neural network. The neural network is trained to generate localization information indicating the location of the source of the target sound event relative to the location of the acoustic sensor. In this manner, the neural network identifies the target sound event in the presence of other interfering sounds in the sound mixture.

[0013] Some embodiments are based on the recognition that the SELD system of the present disclosure is configured to determine a DOA and a distance between a source of a target sound event and an acoustic sensor. The DOA and distance are determined by processing the first digital representation and the second digital representation with a DNN. The DNN is trained to consider sound events from a plurality of sound events other than the identified target sound event as directional interference. The SELD system further outputs the DOA and distance of the identified target sound event.

[0014] As such, the first digital representation corresponds to a representation of spatial and spectral characteristics of the acoustic mixture. The second digital representation includes one or more example waveform representations indicative of one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event. Additionally, the second digital representation corresponds to a one-hot vector indicative of the sound event type of the target sound event from a set of predetermined sound event types. Furthermore, the second digital representation includes one or more example waveform representations indicative of one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event.

[0015] The SELD system is further configured to assign a one-hot vector indicating a target class among a plurality of classes to the identified target sound event. Additionally, the SELD system utilizes a class-conditional SELD network to determine the DOA and distance of the identified target sound event. The class-conditional SELD network includes at least one FiLM block directed to one or more convolution blocks. The at least one FiLM block and the one or more convolution blocks are trained to identify the target sound event and estimate the DOA and distance from the source of the identified target sound event.

[0016] Accordingly, one embodiment discloses a computer-implemented method for localizing a target sound event. The method includes collecting a first digital representation of an acoustic mixture of sounds from multiple sound events detected by an acoustic sensor. The method further includes receiving a second digital representation of the sound corresponding to the target sound event. The first and second digital representations are processed by a neural network, and the neural network is trained to generate localization information indicating the location of a source of the target sound event relative to the location of the acoustic sensor. The neural network identifies the target sound event interfering with other sounds in the acoustic mixture. The localization information of the source of the target sound event is then output. The localization information includes the DOA of the target sound event from the source of the target sound event toward the acoustic sensor and the distance between the source of the target sound event and the acoustic sensor.

[0017] Various embodiments disclosed herein provide a SELD system that can more accurately, efficiently, and quickly determine localization information associated with a target sound event, even when there is insufficient training data available.

[0018] Further features and advantages will become more readily apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0019] [Figure 1A] 1 is a block diagram representation of an environment for estimating the direction of arrival and direction of arrival (DOA) of one or more target sound events, according to some embodiments of the present disclosure. [Figure 1B] FIG. 1 is a block diagram of a sound event localization and detection (SELD) system for detecting localization information of a target sound event, according to some embodiments of the present disclosure. [Figure 1C] FIG. 1 is a block diagram of a method for collecting a first digital representation according to some embodiments of the present disclosure. [Figure 1D] FIG. 10 is a block diagram of a method for receiving a second digital representation according to some embodiments of the present disclosure. [Figure 1E] FIG. 1 is a block diagram of the processing performed by a neural network on a first digital representation and a second digital representation, according to some embodiments of the present disclosure. [Figure 1F] FIG. 10 is a block diagram of another process performed by a neural network on a first digital representation and a second digital representation, according to some embodiments of the present disclosure. [Figure 2] 1 is a schematic block diagram of a sound event localization and detection (SELD) system for estimating localization information of one or more target sound events, according to some embodiments of the present disclosure. [Figure 3] 3 is a block diagram illustrating an exemplary representation of the operation of the SELD system of FIG. 2 according to some embodiments of the present disclosure. [Figure 4A] FIG. 1 is a block diagram illustrating the architecture of a class-conditional SELD network according to some embodiments of the present disclosure. [Figure 4B] FIG. 10 is another block diagram illustrating the architecture of a class-conditional SELD network for locating multiple events according to some embodiments of the present disclosure. [Figure 5A] FIG. 2 is a block diagram of a feature-invariant linear modulation (FiLM) block and a convolution block according to some embodiments of the present disclosure. [Figure 5B] FIG. 10 is a block diagram of a FiLM block for processing an exemplary waveform, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a block diagram of a SELD system for performing audio signal processing according to some embodiments of the present disclosure. [Figure 7] FIG. 1 is a flowchart illustrating a method for estimating localization information of one or more target sound events, according to some embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates a use case for estimating the DOA of a sound event, according to one exemplary embodiment of the present disclosure. [Figure 9] FIG. 10 illustrates another use case for identifying a target sound event and estimating the DOA of the target sound event, according to some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0020] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown in block diagram form solely to avoid obscuring the disclosure. It is contemplated that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0021] As used in this specification and claims, the terms "for example," "for example," and "such as," as well as the verbs "comprise," "have," and "include," and each of their other verb forms, when used in conjunction with a list of one or more components or other items, should be construed as open-ended, meaning that the list should not be considered as excluding other additional components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the style and terminology used herein are for purposes of description and should not be regarded as limiting. Any headings used herein are for convenience only and have no legal or limiting effect.

[0022] Specific details are provided in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements of the disclosed subject matter may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Furthermore, like reference numbers and names in the various drawings indicate like elements.

[0023] System Overview

[0024] 1A illustrates a representation of an environment 100a for estimating localization information including direction of arrival and distance of arrival of one or more target sound events, according to an embodiment of the present disclosure. The environment 100a includes a plurality of sound events 102, including, but not limited to, an airplane sound 102a, a baby crying 102b, a bird chirping 102c, a mouse sound 102d, a hammering sound 102e, and a dog barking 102f. The environment 100a corresponds to an enclosed environment having an acoustic sensor 108. The acoustic sensor 108 is connected to a sound event localization and detection (SELD) system 112 via a network 110.

[0025] In an illustrative example scenario, multiple sound events 102 travel toward the SELD system 112 via various paths, accompanied by directional interference 104. Generally, the directional interference 104 occurs when a sound source that does not need to be localized emits a sound that collides with the multiple sound events 102. The acoustic sensor 108 captures an acoustic mixture containing data associated with the multiple sound events. When the acoustic mixture reaches the acoustic sensor 108, the acoustic sensor 108 transmits the acoustic mixture to the SELD system 112 via the network 110. In one embodiment of the present disclosure, the network 110 is the Internet. In another embodiment of the present disclosure, the network 110 is a wireless mobile network. The network 110 includes a set of channels. Each channel in the set of channels supports a finite bandwidth. The finite bandwidth of each channel in the set of channels is based on the capacity of the network 110. Furthermore, the acoustic sensor 108 includes multiple microphones arranged in a predetermined pattern. The predetermined pattern is configured to capture each of the multiple sound events 102. A predetermined microphone placement pattern enables the system to estimate localization information associated with a sound event 102 using the relative time difference between the microphones. The localization information may be provided in the form of the sound's DOA or the distance of the sound source from the acoustic sensor 108. In one example, the SELD system 112 identifies multiple classes to identify the multiple sound events 102 and distinguish between different types of sound events. In one embodiment, the SELD system 112 identifies a target class 106. The target class 106 includes a baby crying 102b and a hammering 102e. Furthermore, the SELD system 112 determines localization information for only the target class 106 (only the baby crying 102b and the hammering 102e of the multiple sound events 102).

[0026] The SELD system 112 collects first digital representations of sounds associated with the plurality of sound events 102. The first digital representations associated with the plurality of sound events 102 are collected by the acoustic sensor 108. Additionally, the first set of data is in the form of a first digital representation. The first digital representation is shown in FIG. 1B.

[0027] FIG. 1B shows a block diagram 100b of the SELD system 112 shown in FIG. 1A. The SELD system 112 is configured to collect a first digital representation 114 of an acoustic mixture of sounds detected by the acoustic sensor 108. FIG. 1C shows the first digital representation 114. As shown in FIG. 1C, the first digital representation 114 is a digital representation of an acoustic mixture of sounds 122 obtained as a combination of sounds from multiple sound events 102. As such, the first digital representation 114 includes spatial and spectral features calculated from a waveform representation of the acoustic mixture of sounds 122.

[0028] In one example, the first digital representation includes at least one of a short-time Fourier transform S1 of the input acoustic mixture 122 of the multiple sound events 102 and a directional sound field intensity characteristic S2 of the acoustic mixture of the multiple sound events 102. Referring again to Figure 1B, the SELD system 112 further collects a second digital representation 116 of the sound corresponding to the target sound event.

[0029] FIG. 1D illustrates a second digital representation 116. The second digital representation 116 is a digital representation of a target sound event 124 that the SELD system 112 is “interested” in localizing from the plurality of sound event types defined by the target class of sound events 106 depicted in FIG. 1A. Thus, the second digital representation 116 includes one or more example waveforms, or class index c, of the target sound event 124. However, this target sound event 124 may not be exactly the same as the sound event that the SELD system 112 is attempting to localize. That is, for example, the SELD system 112 may be attempting to localize a crying baby 102b, but the second digital representation 116 is derived from data in the target class 106 that includes class c of crying baby sounds, but this crying baby is different from the crying baby 102b. Thus, an advantage of the SELD system 112 is that it can operate even without sufficient training data or related sound event data and still efficiently localize any desired sound event.

[0030] In one example, the second digital representation 116 may be a one-hot vector 0 indicating the sound event type of the target sound event from a set of predetermined sound types. c =[о1,…,о C ] and the one-hot vector о c is a C-dimensional one-hot vector, and for the class c that the SELD system 112 is interested in localizing, c =1, and о i = 0, ∀i ≠ c. For example, the target class 106 in FIG. 1A represents a set of predetermined sound types. If a baby cry 102b is the target sound event 124, then the class from the target class 106 corresponding to the baby cry is represented by the one-hot vector o c The sound event type is indicated by one-hot vector о cis a type of encoding process that converts categorical data into binary data so that it can be efficiently used by machine learning components. c are converted into embedding vectors (learned in the training phase, one for each class c) that are further processed by a machine learning component. In the system of FIG. 1B, the machine learning component is a neural network 118.

[0031] In one example, the second digital representation 116 includes a sound snippet representing a target sound event 124 to be localized. The sound snippet includes a recording of an event of the same type as the target sound event 124. For example, the sound snippet represents a "query by example" case for the neural network 118. The sound snippet is processed to form the second digital representation 116, such as an "embedding vector," which is then processed by the neural network 118 in a manner similar to the embedding vector obtained in the one-hot case.

[0032] In another example, the second digital representation 116 includes a textual description or encoding of the target sound event 124. In yet another example, the second digital representation includes an imitation of a sound corresponding to the target sound event 124. Regardless of the representation type of the second digital representation 116, the neural network 118 is guided by the second digital representation 116 to identify and localize the target sound event.

[0033] 1B , the SELD system 112 includes a neural network 118 that receives the first digital representation 114 and the second digital representation 116 and processes them to generate localization information 120 of a target sound event 124 (e.g., a baby crying 102) that forms an acoustic mixture 122 of multiple sound events 102. The localization information 120 of the target sound event 124 indicates the location of the source of the target sound event that forms the acoustic mixture relative to the location of the acoustic sensor 108 that detects the acoustic mixture.

[0034] The neural network 118 is trained to combine the encodings of the first digital representation 114 and the second digital representation 116 and to guide the processing of the first digital representation 114 based on the second digital representation 116. This is shown in FIG. 1E. To do so, the neural network 118 generates an embedding vector 128 (a one-hot vector o c ) are co-processed with the first digital representation 114 of the sound mixture 122 to generate localization information 120 for the target sound event 124. Another exemplary operation of the neural network 118 is shown in FIG. 1F.

[0035] As shown in FIG. 1F , the neural network 118 is configured to process the first digital representation 114 and the second digital representation 116 differently. The first digital representation 114 is processed as a primary input, and the second digital representation 116 is used to “bias” how the first digital representation 114 is processed. The neural network 118 processes the first digital representation 114 using an attention mechanism 126 that places the first digital representation 114 in the context of the embedding of the second digital representation 116, as represented by an embedding vector 128. In one example, the attention mechanism 126 may be a multi-head attention mechanism, and correspondingly, the neural network 118 may be a multi-head neural network 118. As such, the multi-head neural network 118 is trained with permutation-invariant training to output different location information for different target sound events to different heads of the multi-head neural network 118. This will be further described in conjunction with FIG. 2 .

[0036] The neural network 118 is configured to identify the target sound event 124 interfering with other sounds in the sound mixture 122 by combining the encoding of the second digital representation 116 of the target sound event 124 with at least some intermediate outputs of at least some layers of the neural network 118. To that end, the neural network 118 includes multiple processing layers, such as layer 1, layer 2, ..., and layer n.

[0037] In one example, the second digital representation 116 is input to one or more feature-invariant linear modulation (FiLM) blocks, which are combined with the output of a hidden layer of the neural network 118. This combination is then presented to the next layer of the neural network 118. The use of FiLM blocks in the neural network 118 is further described in FIG. 4.

[0038] Thus, using the various components described above, the SELD system 112 is configured to provide localization information 120 associated with a target sound event 124 .

[0039] The SELD system 112 will be further described with reference to Figure 2. Figure 2 shows a schematic block diagram 200 of a SELD system 112 for estimating localization information of one or more target sound events, according to an embodiment of the present disclosure. The block diagram 200 includes an acoustic sensor 108, a target data source 202, and the SELD system 112.

[0040] The SELD system 112 includes a memory 204, a neural network 206, and a processor 208. The memory 204 includes at least one of RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other storage medium usable to store desired information and accessible by the SELD system 112. The memory 204 includes non-transitory computer storage media in the form of volatile and / or non-volatile memory. The memory 204 may be removable, non-removable, or a combination thereof. Exemplary memory devices include solid-state memory, hard drives, optical disk drives, etc. The memory 204 stores instructions that are executed by the processor 208. Execution of these instructions by the processor 208 causes the SELD system 112 to perform a series of actions, as described below.

[0041] The SELD system 112 receives a set of data from a target data source 202. The target data source 202 corresponds to a sound source among multiple sound sources that is emitting sound and needs to be localized. This second set of data is associated with a target sound event 124. In one example, there are three sound sources in a house: a television, a cell phone, and a person singing. Because the person singing needs to be localized, singing is identified as the target sound event 124. This set of data is utilized to identify one or more target sound events among the multiple sound events 102. This set of data is used to derive a second digital representation 116 from the acoustic characteristics of the target data source 202.

[0042] The SELD system 112 determines localization information 212 for one or more target sound events identified in the second digital representation 116. The localization information 212 is determined by processing the first digital representation 114 and the second digital representation 116 with a neural network 206. The neural network 206 corresponds to the neural network 118 described above. The neural network 206 is trained to consider sound events among the plurality of sound events 102 other than the identified target sound event as directional interference 104. The directional interference 104 is robustly ignored by combining the second digital representation 116 of the desired target sound event with at least one intermediate output received after processing the first digital representation 114 with the neural network 206. In general, robust ignoring of the directional interference 104 improves the efficiency with which the SELD system 112 determines the localization information 212 and reduces the likelihood of error. In one example, the neural network 206 is a deep neural network (DNN). Generally, a DNN is an artificial neural network (ANN) with multiple layers between the input and output layers.

[0043] In one embodiment, the neural network 206 is a multi-head neural network trained with permutation-invariant training to output different localization information for different target sound events to different heads of the multi-head neural network. A multi-head neural network is necessary when multiple instances of a target sound event type are simultaneously emitting sound from different locations. The multi-head neural network includes multiple output heads, each corresponding to a different instance of the target sound event. For example, if two babies are crying simultaneously, each output head will contain location information for one baby. Thus, output localization information is generated from different output heads of the multi-head neural network for different target sound events. Furthermore, permutation-invariant training assumes that the training data includes a set of unordered training data components. The neural network 206 can identify the desired target class by processing the encoding vector or embedding vector for each target class identified by the second digital representation 116 of the target sound event 124. The output of the permutation-invariant trained multi-head neural network 118 includes multiple output heads for outputting localization information from the SELD system 112 .

[0044] Additionally, the SELD system 112 outputs localization information 212 of the identified target sounds, as further described in the exemplary implementation shown in FIG.

[0045] 3 illustrates an example representation 300 of the operation of the SELD system 112, according to an embodiment of the present disclosure. The acoustic sensor 108 transmits a first digital representation 302 of an acoustic mixture of multiple sound events 102 to the SELD system 112. The first digital representation 302 undergoes a short-time Fourier transform (STFT) 304. The STFT 304 is a Fourier-related transform used to determine the sinusoidal frequency and phase components of localized portions of the first digital representation 302 as it varies over time. Additionally, one or more intensity vectors 306 are calculated to measure the spatial characteristics of the multiple sound signals represented by the first digital representation 302. Generally, the intensity vectors determine the relative position of one point / object / sound source relative to another. As such, the SELD system 112 utilizes one or more intensity vectors 306 to represent spatial information in the sound signals, which, when combined with the STFT 304, classify the multiple sound signals into different classes of multiple sound events 102. Furthermore, the SELD system 112 performs training of the neural network 206 based on the multiple sound signals included in the acoustic mixture of sounds represented by the first digital representation and the output received after the STFT 304. During training, the SELD system 112 performs concatenation 308 of the output received after the STFT 304 and after assigning one or more intensity vectors 306. The concatenated data obtained by the concatenation 308 operation is sent to the neural network 206. The neural network 206 is trained to estimate a target sound event from among the multiple sound events 102, along with a target acoustic source and a target class of the target sound event from the multiple classes, using the second digital representation, as described above. The SELD system 112 also functions as a class-conditional SELD network for determining localization information 212 of the identified target sound event. Class-conditional SELD networks are further explained in FIG.

[0046] 4A illustrates the architecture of a class-conditional SELD network 400a according to some embodiments of the present disclosure. The class-conditional SELD network 400a includes at least one FiLM block 402 directed to one or more convolution blocks 404. The at least one FiLM block 402 and the one or more convolution blocks 404 are trained to identify target sound events and estimate localization information 212 for the identified target sound events. In one example, the class-conditional SELD network 400a includes FiLM block 1402a, FiLM block 2402b, and FiLM block 3402c. In addition, the class-conditional SELD network 400a includes Conv block 1404a, Conv block 2404b, and Conv block 3404c.

[0047] The FiLM blocks 402a, 402b, and 402c output a set of parameters (γ,β) = f(e) used to scale and shift the learned feature maps M of the baseline SELD network, where FiLM(M|γi,βi) = γiM+βi, where i refers to the channel index and γ and β correspond to trainable parameters that result in different linear / affine transformations for every channel. In general, the FiLM layer applies a feature affine transformation to its input. In addition, a FiLM operation 406 is applied to the feature maps of each convolution block 404a, 404b, and 404c in the baseline SELD network. The FiLM operation 406 is denoted as (1 + γ) * x + β, where x is the input.

[0048] The FiLM block 1402a is directed to the Conv block 1404a, and then the FiLM operation 406 is applied to the feature map of the Conv block 1404a. Additionally, the FiLM block 2402b is directed to the Conv block 2404b, and then the FiLM operation 406 is applied to the feature map of the Conv block 2404b. The FiLM block 3402c is directed to the Conv block 3404c, and then the FiLM operation 406 is applied to the feature map of the Conv block 3404c. Furthermore, the feature maps of each of the Conv blocks 404a, 404b, and 404c are sent as outputs to a bidirectional gated recurrent unit (BiGRU) 408. Generally, the BiGRU is a sequence processing model consisting of two GRUs (one that takes input in the forward direction and one that takes input in the backward direction). The BiGRU 408 is a bidirectional recurrent neural network with only input gates and forget gates. Additionally, the outputs of both the forward neural network and the backward neural network are connected within the same or a single output layer. The output generated by the BiGRU 408 is then sent to the linear and dropout layer 410. Generally, the linear layer can learn the average correlation between the output and the input. For example, if x and y are positively correlated, w will be positive; if x and y are negatively correlated, w will be negative. If x and y are completely independent, w will be near 0. Generally, dropout is used to prevent the model from overfitting. The output generated by the linear and dropout layer 410 is then sent to the linear layer 412. The linear layer 412 has a tanh activation function. The tanh activation function maps values ​​between -1 and 1. The linear layer 412 generates three outputs: estimated x, y, and z coordinates of one or more target sound events. During the training phase, localization information, including the DOA (azimuth and elevation angles) of one or more target sound events, is present.The DOA is converted to Cartesian (x, y, z) coordinates using typical formulas for converting between spherical and Cartesian coordinates, with the radius of the DOA set to 1 while the target source is active and 0 while the target source is inactive. Then, during the training phase, the x, y, z output from the linear layer 412 is compared with the ground truth x, y, z data to calculate the mean squared error (MSE) 414. Additionally, the x, y, z coordinates output by the linear layer 412 are converted to estimated directions of arrival 416 of the identified target sound events by converting the x, y, and z position coordinates of the identified target sound events into azimuth and elevation angles.

[0049]

number

[0050] In one example, the architecture shown in FIG. 4B is used to detect multiple target events.

[0051] 4B illustrates the architecture of class-conditional SELD network 400b, according to some embodiments of the present disclosure. Class-conditional SELD network 400b is similar to class-conditional SELD network 400a, except for the addition of blocks 417 and 418 to enable multiple target event detection.

[0052] Therefore, the single-output-class-conditional SELD network 400a is converted to output multiple estimated source locations, as shown in FIG. 4B. Network 400b in FIG. 4B illustrates the maximum number of possible simultaneous output events (N), and the SELD network 400b outputs a separate location representation for each of the N sources 417. During training, because all network outputs correspond to target events, the order in which the network chooses to output the target events may not match the order of events in the ground truth signal, requiring permutation invariant training (PIT) 418. PIT calculates the mean squared error (MSE) for all possible orderings between the ground truth and output locations. For example, for N=2, there are two possible orderings: 1. (ground_truth_1->output_1, ground_truth_2->output_2) and 2. (ground_truth_1->output_2, ground_truth_2->output_1). PIT 418 then uses the ordering with the lowest MSE as the error signal for backpropagation during neural network training.

[0053] Neural network architectures 400a and 400b represent class-conditional SELD networks that include at least one FiLM block 402 directed to one or more convolution blocks 404. The at least one FiLM block 402 and the one or more convolution blocks 404 are trained to identify target sound events and estimate localization information for the target sound events.

[0054] The FiLM block 402 is described in more detail in Figures 5A and 5B.

[0055] 5A shows a block diagram 500a of the FiLM block 402 and the convolution block 404 according to an embodiment of the present disclosure. The second digital representation S is used as input to at least one FiLM block 402. The FiLM block 402 includes an embedding layer 402aa, a linear layer 402bb, a dropout layer 402cc, a tile layer 402dd, and an output layer 402ee. The one-hot vector o c is passed through the trained embedding layer 402aa to obtain an embedding e∈R that encodes the target class information. N Further, the target class information passes through a linear layer 402bb and a dropout layer 402cc. The linear layer 402bb is used to learn a mapping of the embedding dimension N to the dimension of the feature map to which FiLM conditioning is applied. Additionally, the dropout layer 402cc randomly sets input units to 0 at a frequency of rate at each step of the training phase, which prevents overfitting. Input units that are not set to 0 are further scaled up by 1 / (1-rate) so that the total sum of all input units remains unchanged. Generally, the dropout layer 402cc is only applied when training is set to "true" to prevent values ​​from being dropped during inference.

[0056] Additionally, the FiLM block 402 utilizes a tiling layer 402dd. Because the conditioning inputs, which represent the general acoustic characteristics of the target class, are the same at all time points, the tiling layer 402dd represents copying the conditioning information across time steps. The trained model is then used to infer outcomes regarding the DOAs of one or more target sound events 124. Additionally, the FiLM block 402 outputs a set of parameters 402ee(γ,β)=f(e) used to scale and shift the learned feature map M of the baseline network, where FiLM(Mi|γi,βi)=γiMi+βi, where i refers to the channel index.

[0057] The FiLM block 402 is fed to one or more convolutional blocks 404. Additionally, each of the one or more convolutional blocks 404 includes a two-dimensional convolutional layer 404aa, a batch normalized feature 404bb, a rectified linear activation function (ReLU) 404cc, a max pooling layer 404dd, and a dropout layer 404 ee The input of the 2D convolutional layer 404aa includes a 64-dimensional mel-spectrogram and a three-channel first-order Ambisonic (FOA) intensity vector. Generally, the 2D first-order Ambisonic format consists of one signal corresponding to an omnidirectional pickup pattern (called W) and two signals corresponding to a figure-of-eight pickup pattern aligned with the Cartesian axes (X and Y). In one embodiment, a filter or kernel in the 2D convolutional layer slides over the input data, performing element-by-element calculations. The filter or kernel then sums the results into a single output. The kernel performs the same operation at each position along which the kernel slides, transforming the 2D feature matrix into a different 2D feature matrix. Additionally, the convolutional block 404 utilizes a batch normalization feature 404bb technique to train the deep neural network 206, which standardizes the input to the layer for each mini-batch.

[0058] The convolution block 404 includes a ReLU 404cc, which is a piecewise linear function that outputs the input directly if the input is positive and zero otherwise. In general, a ReLU does not activate all neurons simultaneously.

[0059] Furthermore, the convolution block 404 utilizes a max pooling layer 404dd, which is a two-dimensional layer. The max pooling layer 404dd downsamples the input along with its spatial dimension by taking the maximum value of an input window (of a size defined by the pool size) for each channel of the input. Furthermore, the convolution block 404 includes a dropout layer 404ee to prevent overfitting.

[0060] The neural network 118 processes the first digital representation 114 and the second digital representation 116 using a combination of FiLM blocks 402 and Conv blocks. To do so, the second digital representation 116 is presented to one or more FiLM blocks 402. The output of the FiLM blocks 402 is combined with the output of a hidden layer of the neural network 118, and this combination is presented to the next layer of the neural network 118. This is shown in Figures 4A and 4B.

[0061] FIG. 5B illustrates FiLM block 403, a slight modification of FiLM block 402, where the second digital representation S includes one or more example waveform representations that indicate one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event 124. In this case, the target sound event 124 may not have been seen during training, but example(s) of the sound type to be localized are available at inference time. In this case, one or more example waveforms S are first processed by feature extraction block 403aa and converted into an embedding vector. Examples of the feature extraction process could be the average frequency spectrum of all example waveforms, or a separate network that inputs the example waveforms and outputs a single embedding vector. The remaining components of the FiLM block (403bb, 403cc, 403dd, 403ee) remain the same when the second digital representation is in a different format, such as a one-hot vector.

[0062] In this way, the SELD 112 system uses different combinations of the FiLM block 402 or 403 and the Conv block 404 to localize the target sound event 124, even with less training data. The SELD system 112 can then be used in many audio processing applications.

[0063] 6 is a block diagram 600 of a SELD system 112 for performing audio signal processing according to an embodiment of the present disclosure. In some exemplary embodiments, the block diagram 600 includes an acoustic sensor 108 or sensors that collect data from an environment 602, including multiple sound signals 606 of multiple sound events 102. In addition, the acoustic sensor 108 collects data associated with one or more target sound events from target data source(s) 210.

[0064] The SELD system 112 includes a hardware processor 608. The hardware processor 608 communicates with computer storage memory, such as memory 610. The memory 610 contains stored data, including algorithms, instructions, and other data, that are executed by the hardware processor 608. The hardware processor 608 may include two or more hardware processors, depending on the requirements of a particular application. The two or more hardware processors may be either internal or external. The SELD system 112 may be integrated with other components, including output interfaces and transceivers, among other devices.

[0065] In some alternative embodiments, the hardware processor 608 is connected to a network 604 that communicates with the target data source 210. The network 604 may include, by way of non-limiting example, one or more local area networks (LANs) and / or wide area networks (WANs). The network 604 may also include enterprise-wide computer networks, intranets, and the Internet. The SELD system 112 includes one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources may comprise a single device or multiple devices that cooperate in a distributed environment across the network 604.

[0066] In some other alternative embodiments, the hardware processor 608 is connected to a network-enabled server 614 that is connected to the client device 616. The network-enabled server 614 corresponds to a dedicated computer connected to a network that runs software intended to process client requests received from the client device 616 and provide an appropriate response on the client device 616. The hardware processor 608 is connected to an external memory device 618 that stores all necessary data used for DOA estimation, and to a transmitter 620. The transmitter 620 facilitates data transmission between the network-enabled server 614 and the client device 616. Additionally, an output 622 for one or more target sound events is generated.

[0067] The target data source(s) 202 include data resources for training the deep neural network 206. For example, in one embodiment, the training data includes acoustic signals of multiple speakers speaking simultaneously. The training data also includes acoustic signals of a single speaker speaking alone, acoustic signals of single or multiple speakers speaking in a noisy environment, and acoustic signals of a noisy environment.

[0068] The SELD system 112 also includes a third-party device 624, which may include any type of computing device, such as an automatic speech recognition (ASR) system. For example, the third-party device may include, but is not limited to, a computer device or a mobile device. A mobile device may include, but is not limited to, a personal data assistant (PDA), a smartphone, a smartwatch, smart glasses (or other wearable smart device), an augmented reality headset, a virtual reality headset, a laptop, a tablet, a remote control, an entertainment system, a vehicle computer system, an embedded system controller, an appliance, a home computer system, a security system, a consumer electronic device, or other similar electronic device. In addition, the mobile device may include, but is not limited to, a microphone or line input for receiving audio information, or a communication component (e.g., Wi-Fi capability) for receiving such information from another source, such as the Internet. In one exemplary embodiment, the mobile device may receive input data, such as a sound signal. For example, the input data may include sound events from one speaker into a microphone of the mobile device while multiple speakers are speaking in a room. The input data is processed by an ASR in the mobile device using the SELD system 112 to determine the class of the sound event. The SELD system 112 enhances the input data by reducing noise in the speaker's environment, separating the speaker from other speakers, or enhancing the audio signal, allowing the ASR to output the DOA of the sound event.

[0069] Additionally, the SELD system 112 stores input data in storage 612. Storage 612 stores data, computer instructions (e.g., software program instructions, routines, or services), and / or information including data related to the deep neural network 206. For example, storage 612 stores data from the target data source(s) 210, one or more deep neural network models, information for generating and training the deep neural network models, and computer-usable information output by the one or more deep neural network models.

[0070] In addition to estimating the DOA, the distance to the target sound event can also be included in the localization data output by the network. If training data for distance estimation is available, this training data can be used as an additional output of the neural network. Alternatively, DOA can be estimated from multiple subarrays and triangulation can be used to estimate distance from multiple DOAs at known subarray locations.

[0071] FIG. 7 shows a flowchart depicting a method 700 for localizing one or more target sound events according to various embodiments of the present disclosure. The method is performed by the SELD system 112. The flowchart begins at step 702. Following step 702, in step 704, the method includes collecting, in the SELD system 112, a first digital representation of an acoustic mixture of sounds detected by an acoustic sensor 108, such as an acoustic mixture 122 of sounds from multiple sources 102 detected by the acoustic sensor 108. The acoustic sensor 108 includes multiple microphones arranged in a predetermined pattern. The first digital representation is the first digital representation 114 shown in FIG. 1C , which corresponds to a representation of spatial and spectral characteristics of the acoustic mixture of the multiple sound events 102. In one example, the first digital representation 114 includes at least one of a short-time Fourier transform of the input acoustic mixture 122 of the multiple sound events 102 and a directional sound field intensity characteristic of the acoustic mixture 122 of the multiple sound events 102.

[0072] At step 706, the method 700 includes receiving, from the target data source, a second digital representation associated with one or more target sound events. For example, FIG. 1D illustrates a second digital representation 116 representing spatial, spectral, and class information associated with a target sound event among the plurality of sound events 102. The method 700 further includes creating a plurality of classes for classifying the plurality of sound events 102. The method 700 includes utilizing the second digital representation to identify one or more target sound events among the plurality of sound events 102 from the plurality of classes, along with a target acoustic source and a target class for the one or more target sound events. In one embodiment, the second digital representation 116 includes a one-hot vector for identifying a sound event type of the target sound event from among a set of predetermined sound event types, such as the set of target classes 106 shown in FIG. 1A.

[0073] In step 708, the method 700 includes processing the first digital representation 114 and the second digital representation 116 with a neural network 118 to determine localization information 120. The localization information 120 includes a direction of arrival of the target sound event from a source of the target sound event toward an acoustic sensor and a distance between the source of the target sound event and the acoustic sensor. The neural network 118 is trained to consider sound events of the plurality of sound events 102 other than the identified target sound event 124 as directional interference 104. The directional interference 104 is identified by combining the second digital representation 116 of the identified target sound event 124 with at least one intermediate output received after processing the first digital representation 114 with the neural network 118.

[0074] In step 710, the method 700 includes outputting the localization information, which may be used for further processing in an application, such as an audio processing application. The method ends in step 712.

[0075] 8 illustrates a use case 800 for estimating localization information including DOA of a sound event according to some other exemplary embodiments of the present disclosure. Use case 800 corresponds to a factory floor scene including one or more speakers, such as speaker 802A and speaker 802B. The factory floor has multiple sound events and noises due to the operation of various industrial machines. The factory floor includes an audio device 804 for facilitating communication between a factory floor control operator (not shown) and one or more speakers 802A and 802B within the factory floor. Audio device 804 includes SELD system 112.

[0076] In an illustrative example scenario, audio device 804 is transmitting a voice command addressed to a person 802A managing a factory floor. The voice command states, "Please report the status of machine 1." Speaker 802A utters, "Machine 1 running." However, the speech signal of speaker 802A's utterance is mixed with noise from the machine, noise from the background, and other speech from speaker 802B in the background. Such noise and reverberant signals are mitigated by SELD system 112. SELD system 112 identifies a target sound event and outputs a DOA for the target sound event. The output DOA is further used to identify the sound of speaker 802A and ignore sounds from other sources. Responses from speaker 802A are then sent back to audio device 804 in a focused and targeted manner to perform better voice control operations in the factory floor scene depicted in FIG. 8 .

[0077] 9 illustrates another use case 900 for identifying a target sound event and estimating the direction of arrival of the target sound event, according to some exemplary embodiments of the present disclosure. Use case 900 corresponds to an in-vehicle infotainment system 904. In-vehicle infotainment system 904 is implemented in a vehicle, such as a manually operated vehicle, an automated vehicle, or a semi-automated vehicle. The vehicle is occupied by multiple people, such as person 902a, person 902b, and person 902c. In-vehicle infotainment system 904 includes SELD system 112. For example, in-vehicle infotainment system 904 is remotely connected to SELD system 112 via a network. In some alternative exemplary embodiments, SELD system 112 is incorporated within in-vehicle infotainment system 904.

[0078] The in-vehicle infotainment system 904 also includes one or more microphones for receiving a mixture of acoustic signals, including speech signals from persons 902a, 902b, and 902c, and external noise signals, such as the honking of other vehicles. In this case, person 902a is speaking to person 902c, and another person 902b is speaking / singing louder than persons 902a and 902c. b Speech from persons 902a and 902c interferes with the speech commands of persons 902a and 902c. For example, person 902a's speech command is "My gas tank is almost empty," and person 902c's speech command is "Please find the nearest gas pump." In such a case, SELD system 112 processes the speech of each of persons 902a, 902b, and 902c and identifies the target sound event as belonging to the speech command of persons 902a and 902c. SELD system 112 considers all speech except for the speech of persons 902a and 902c as directional interference and outputs the DOA of the target sound event accordingly. The output DOA is further used to identify the sounds of persons 902a and 902c and ignore sounds from other sources.

[0079] Many modifications and other embodiments of these inventions will come to mind to one skilled in the art to which the inventions described herein pertain having the benefit of the teachings presented in the foregoing description and the associated drawings. It is to be understood that the invention is not limited to the particular embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, while the foregoing description and the associated drawings describe exemplary embodiments in the context of particular example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided in alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or functions other than those explicitly described above are also contemplated as being set forth in some of the appended claims. Although specific terms are used herein, they are used for purposes of general description only and not of limitation.

Claims

1. 1. A sound event localization and detection (SELD) system for localizing one or more target sound events, the SELD system comprising: at least one processor; and a memory having stored thereon instructions, the instructions, when executed by the at least one processor, causing the SELD system to: collecting a first digital representation of an acoustic mixture of sounds of a plurality of sound events detected by an acoustic sensor; receiving a second digital representation of a class of sounds corresponding to a target sound event within the acoustic mixture; processing the first digital representation and the second digital representation with a neural network, the neural network being trained to generate target sound event localization information indicative of a location of a source of the target sound event within the sound mixture relative to a location of the acoustic sensor detecting the sound mixture; outputting the localization information of the source of the target sound event.

2. 2. The SELD system of claim 1, wherein the target sound event is identified by the neural network by combining the second digital representation of the target sound event with at least some intermediate outputs of at least some layers of the neural network that process the first digital representation of the sound mixture.

3. 2. The SELD system of claim 1, wherein the localization information includes one or a combination of a direction of arrival of the target sound event from a source of the target sound event toward the acoustic sensor and a distance between the source of the target sound event and the acoustic sensor.

4. 2. The SELD system of claim 1, wherein the second digital representation identifies multiple target sound events, the neural network is a multi-head neural network, and the multi-head neural network is trained with permutation-invariant training to output different location information for different target sound events to different heads of the multi-head neural network.

5. 2. The SELD system of claim 1, wherein the neural network identifies the target sound event interfering with other sounds in the sound mixture by combining the encoding of the second digital representation of the target sound event with at least some intermediate outputs of at least some layers of the neural network that process the first digital representation of the sound mixture.

6. 2. The SELD system of claim 1, wherein the neural network processes the first digital representation using attention that places the first digital representation in the context of an embedding of the second digital representation.

7. The processor: inputting the second digital representation into one or more feature-invariant linear modulation (FiLM) blocks; combining the output of the one or more FiLM blocks with the output of a hidden layer of the neural network; The SELD system of claim 1 configured to present the combination to a next layer of the neural network.

8. 8. The SELD system of claim 7, wherein the neural network includes a class-conditional SELD network for determining the localization information, the class-conditional SELD network including at least one FiLM block directed to one or more convolution blocks, the at least one FiLM block and the one or more convolution blocks being trained to identify the target sound event and estimate the localization information.

9. The SELD system of claim 8 , wherein each of the FiLM blocks includes an embedding layer, a linear layer, a dropout layer, a tile layer, and an output layer.

10. 10. The SELD system of claim 8, wherein each of the one or more convolutional blocks includes a two-dimensional convolutional layer, a batch normalization layer, a rectified linear activation function (ReLU), a max pooling layer, and a dropout layer.

11. The SELD system of claim 8 , wherein the neural network is a convolutional recurrent neural network.

12. The SELD system of claim 1 , wherein the second digital representation comprises a one-hot vector indicating a sound event type of the target sound event from a set of predetermined sound event types.

13. The SELD system of claim 1 , wherein the second digital representation includes one or more example waveform representations indicative of one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event.

14. The neural network converts the one-hot vector into an embedding vector and processes the embedding vector together with a first digital representation of the sound mixture to obtain the definition. The SELD system of claim 12, wherein the SELD system generates position information.

15. 15. The SELD system of claim 14, wherein the neural network uses one or more feature-invariant linear modulation (FiLM) blocks to jointly process the embedding vector with the first digital representation of the sound mixture.

16. The SELD system of claim 1 , wherein the first digital representation comprises a waveform representation of spatial and spectral characteristics of the acoustic mixture.

17. The SELD system of claim 1 , wherein the acoustic sensor comprises a plurality of microphones arranged in a predetermined pattern operatively connected to the SELD system via a wired or wireless communication channel.

18. 1. A method for localization of a target sound event, said method comprising: collecting a first digital representation of an acoustic mixture of sounds of a plurality of sound events detected by an acoustic sensor; receiving a second digital representation of a class of sounds corresponding to the target sound event; processing the first digital representation and the second digital representation with a neural network, wherein the neural network is trained to generate localization information indicative of a location of a source of the target sound event relative to a location of the acoustic sensor, the neural network identifying the target sound event interfering with other sounds in the acoustic mixture, the method further comprising: outputting the localization information of the source of the target sound event.

19. 20. The method of claim 18, wherein the target sound event is identified by the neural network by combining the second digital representation of the target sound event with at least some intermediate outputs of at least some layers of the neural network that process the first digital representation of the sound mixture.

20. 20. The method of claim 18, wherein the second digital representation comprises one or more example waveform representations indicative of one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event.

Images