Method and system for sound event localization and detection - Patents.com

Abstract

An embodiment of the present disclosure discloses a system and method for localization of a target sound event. The system collects a first digital representation of an acoustic mixture of sounds of a plurality of sound events by using an acoustic sensor. The system receives a second digital representation of the sounds corresponding to the target sound event. The first digital representation and the second digital representation are further processed by a neural network to generate localization information indicative of a location of a source of the target sound event relative to a location of the acoustic sensor.

Description

[Technical field]

[0001] The present disclosure relates generally to sound event localization and detection (SELD), and more specifically to a sound event localization and detection system for estimating the direction of arrival of a sound event. [Background technology]

[0002] Existing systems for SELD typically operate by estimating source locations for all classes at each time instant. However, SELD remains challenging because sound sources often move, go silent, have their location obscured by room reflections, or are mixed with interfering sounds. Moreover, many sound events are easily confused, further complicating the task of SELD.

[0003] Sound localization information, especially estimating the direction of arrival (DOA) of a sound and further classifying the type of event occurring at the source of the sound, is considered to be an important type of front-end processing used in monitoring and robotics applications. Some existing SELD systems follow a pipeline consisting of two main stages. The first stage extracts the spectral and spatial features of the received sound from a microphone array. The second stage uses a deep neural network (DNN) to learn the mapping from the multi-channel input features to two output targets: (1) a vector of class probabilities indicating the presence of each class at each time point, and (2) a DOA vector containing the location information of each class. In this case, training the DNN is challenging because the contributions from the two output branches must be balanced.

[0004] Some existing SELD systems, including those based on activity coupled Cartesian direction of arrival (ACCDOA) representations, typically assume a small, fixed set of sound event classes to detect and localize. In practice, if there are 12 classes, the DNN will output 12 ACCDOA representations at each time point. This approach can become impractical for large numbers of classes, and in some situations it may not be necessary to localize all classes all the time.

[0005] Training a class-specific system to localize only sound events from a single class would therefore allow us to focus on a particular class. However, training class-specific models can be difficult because there may not be enough data available for each class of interest to adequately train all models.

[0006] Therefore, there is a need for a SELD system that overcomes the above-mentioned shortcomings.

[0007] Therefore, there is a need for a technical solution to overcome the above-mentioned challenges, and more specifically, for such a SELD system that outperforms conventional sound event detection and localization in the presence of directional interference. Summary of the Invention

[0008] This disclosure provides a class-conditional SELD system for sound event localization. As mentioned above, existing systems for SELD typically operate by estimating source positions for all classes at each time point. However, SELD remains challenging because sound sources often move, go silent, are obscured by room reflections, and are mixed with interfering sounds. In addition, many sound events are easily confused, further complicating the task of SELD.

[0009] Some embodiments are based on the recognition that in current SELD systems, there is only one ACCDOA vector output at each time instant, and the class represented by this output is determined based on inputs describing the type of sound event that needs to be located. Some existing SELD systems, including those based on ACCDOA representations, typically assume a small, fixed set of sound event classes to detect and localize. For example, 12 classes would output 12 ACCDOA representations at each time instant.

[0010] Some embodiments are based on the recognition that outputting one ACCDOA representation per class may become impractical when the number of classes is large, and in some situations it may not be necessary to localize all classes all the time. Therefore, training a class-specific system to localize only sound events from a single class allows focusing on a specific class. However, training class-specific models can be difficult because very large training data is required for each class of interest in order to properly train all models.

[0011] Thus, various embodiments provided herein provide methods and systems for localizing sound events from a single target class and identifying the target source based on the localized sound events.

[0012] Some embodiments are based on the recognition that the SELD system can be trained as a class-specific system for localizing sound events from a single class and identifying a target sound event based on the localized sound events. The SELD system is further utilized for estimating the DOA of the target sound event and the distance between the source of the target sound event and the sound source. To that end, the SELD system collects a first digital representation of the sound mixture detected by the acoustic sensor. The sound mixture is associated with a plurality of sound events. The SELD system further receives a second digital representation of the sound corresponding to the target sound event. The second digital representation is utilized to identify the target sound event among the plurality of sound events. The first digital representation and the second digital representation are processed by a neural network. The neural network is trained to generate localization information indicative of the location of the source of the target sound event relative to the location of the acoustic sensor. In this manner, the neural network identifies the target sound event with interference from other sounds in the sound mixture.

[0013] Some embodiments are based on the recognition that the SELD system of the present disclosure is configured to determine a DOA and a distance between a source of a target sound event and an acoustic sensor. The DOA and the distance are determined by processing the first digital representation and the second digital representation with a DNN. The DNN is trained to consider a sound event of a plurality of sound events other than the identified target sound event as a directional interference. The SELD system further outputs the DOA and the distance of the identified target sound event.

[0014] As such, the first digital representation corresponds to a representation of spatial and spectral features of the acoustic mixture. The second digital representation includes one or more example waveform representations indicative of one or a combination of spatial, spectral, or class information of the target sound event. Additionally, the second digital representation corresponds to a one-hot vector indicative of a sound event type of the target sound event from a set of predefined sound event types. Furthermore, the second digital representation includes one or more example waveform representations indicative of one or a combination of spatial, spectral, or class information of the target sound event.

[0015] The SELD system is further configured to assign a one-hot vector indicative of a target class among a plurality of classes to the identified target sound event. In addition, the SELD system utilizes a class-conditional SELD network to determine the DOA and distance of the identified target sound event. The class-conditional SELD network includes at least one FiLM block directed to one or more convolution blocks. The at least one FiLM block and the one or more convolution blocks are trained to identify the target sound event and estimate the DOA and distance from a source of the identified target sound event.

[0016] Accordingly, one embodiment discloses a computer-implemented method for localizing a target sound event. The method includes collecting a first digital representation of an acoustic mixture of sounds of a plurality of sound events detected by an acoustic sensor. The method further includes receiving a second digital representation of the sound corresponding to the target sound event. The first digital representation and the second digital representation are processed with a neural network, and the neural network is trained to generate localization information indicative of a location of a source of the target sound event relative to a location of the acoustic sensor. The neural network identifies the target sound event with interference from other sounds in the acoustic mixture. The localization information of the source of the target sound event is then output. The localization information includes a DOA of the target sound event from a source of the target sound event toward the acoustic sensor and a distance between the source of the target sound event and the acoustic sensor of the sound.

[0017] Various embodiments disclosed herein provide a SELD system that can more accurately, efficiently, and quickly determine localization information associated with a target sound event, even when insufficient training data is available.

[0018] Further features and advantages will become more readily apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1A] 1 is a block diagram representation of an environment for estimation of direction of arrival and direction of arrival (DOA) of one or more target sound events according to some embodiments of the present disclosure. [Figure 1B] FIG. 1 is a block diagram of a sound event localization and detection (SELD) system for detection of localization information of a target sound event according to some embodiments of the present disclosure. [Figure 1C] 1 is a block diagram of a method for collecting a first digital representation according to some embodiments of the present disclosure. [Figure 1D] 1 is a block diagram of a method of receiving a second digital representation according to some embodiments of the present disclosure. [Figure 1E] FIG. 2 is a block diagram of the processing performed by a neural network on a first digital representation and a second digital representation according to some embodiments of the present disclosure. [Figure 1F] FIG. 11 is a block diagram of another process that a neural network performs on a first digital representation and a second digital representation, according to some embodiments of the present disclosure. [Diagram 2] FIG. 1 is a schematic block diagram of a sound event localization and detection (SELD) system for estimation of localization information of one or more target sound events, according to some embodiments of the present disclosure. [Diagram 3] 3 is a block diagram illustrating an example representation of the operation of the SELD system of FIG. 2 according to some embodiments of the present disclosure. [Figure 4A] FIG. 2 is a block diagram illustrating an architecture of a class-conditional SELD network according to some embodiments of the present disclosure. [Figure 4B] FIG. 13 is another block diagram illustrating an architecture of a class-conditional SELD network for locating multiple events according to some embodiments of the present disclosure. [Figure 5A] FIG. 2 is a block diagram of a feature-invariant linear modulation (FiLM) block and a convolution block according to some embodiments of the present disclosure. [Figure 5B] FIG. 13 is a block diagram of a FiLM block for processing an exemplary waveform according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a block diagram of a SELD system for performing audio signal processing according to some embodiments of the present disclosure. [Figure 7] FIG. 2 is a flowchart illustrating a method for estimating localization information of one or more target sound events according to some embodiments of the present disclosure. [Figure 8] FIG. 2 illustrates a use case for estimating the DOA of a sound event according to one exemplary embodiment of the present disclosure. [Figure 9] FIG. 2 illustrates another use case for identifying a target sound event and estimating the DOA of the target sound event, according to some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, the apparatus and methods are shown in block diagram form solely to avoid obscuring the present disclosure. It is contemplated that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0021] As used in this specification and claims, the terms "for example," "for example," and "such as," as well as the verbs "comprise," "have," "include," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should be construed as open-ended, meaning that the list should not be considered as excluding other additional components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the style and terminology used herein are for purposes of illustration and should not be considered limiting. Any headings used herein are for convenience only and should not be considered as having any legal or limiting effect.

[0022] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments can be practiced without these specific details. For example, systems, processes, and other elements of the disclosed subject matter may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Additionally, like reference numbers and names in the various drawings refer to like elements.

[0023] System Overview

[0024] 1A illustrates a representation of an environment 100a for estimating localization information including direction of arrival and distance of arrival of one or more target sound events according to an embodiment of the present disclosure. The environment 100a includes a plurality of sound events 102, including but not limited to an airplane sound 102a, a baby crying 102b, a bird chirping 102c, a mouse sound 102d, a hammer sound 102e, and a dog barking 102f. The environment 100a corresponds to a closed environment having an acoustic sensor 108. The acoustic sensor 108 is connected to a sound event localization and detection (SELD) system 112 via a network 110.

[0025] In an illustrative example scenario, the multiple sound events 102 travel toward the SELD system 112 in various paths with directional interference 104. In general, the directional interference 104 occurs when a sound source that does not need to be localized emits a sound that collides with the multiple sound events 102. The acoustic sensor 108 captures an acoustic mixture that includes data associated with the multiple sound events. When the acoustic mixture reaches the acoustic sensor 108, the acoustic sensor 108 forwards the acoustic mixture to the SELD system 112 via the network 110. In one embodiment of the present disclosure, the network 110 is the Internet. In another embodiment of the present disclosure, the network 110 is a wireless mobile network. The network 110 includes a set of channels. Each channel of the set of channels supports a finite bandwidth. The finite bandwidth of each channel of the set of channels is based on the capacity of the network 110. Furthermore, the acoustic sensor 108 includes multiple microphones arranged in a predetermined pattern. The predetermined pattern is set such that each of the multiple sound events 102 is captured. A predefined microphone placement pattern allows the system to estimate localization information associated with a sound event 102 using the relative time difference between the microphones. The localization information may be provided in the form of a sound's DOA or a distance of the sound source from the acoustic sensor 108. In one example, the SELD system 112 identifies multiple classes to identify the multiple sound events 102 and distinguish between different types of sound events. In one embodiment, the SELD system 112 identifies a target class 106. The target class 106 includes a baby cry 102b and a hammer sound 102e. Furthermore, the SELD system 112 determines localization information for only the target class 106 (only the baby cry 102b and the hammer sound 102e of the multiple sound events 102).

[0026] The SELD system 112 collects a first digital representation of sounds associated with the plurality of sound events 102. The first digital representations associated with the plurality of sound events 102 are collected by the acoustic sensor 108. Additionally, the first set of data is in the form of a first digital representation. The first digital representation is shown in FIG. 1B.

[0027] FIG 1B illustrates a block diagram 100b of the SELD system 112 shown in FIG 1A. The SELD system 112 is configured to collect a first digital representation 114 of an acoustic mixture of sounds detected by the acoustic sensor 108. FIG 1C illustrates the first digital representation 114. As shown in FIG 1C, the first digital representation 114 is a digital representation of an acoustic mixture of sounds 122 obtained as a combination of sounds from multiple sound events 102. As such, the first digital representation 114 includes spatial and spectral features calculated from a waveform representation of the acoustic mixture of sounds 122.

[0028] In one example, the first digital representation includes at least one of a short-time Fourier transform S1 of the input acoustic mixture 122 of the multiple sound events 102 and a directional sound field intensity characteristic S2 of the acoustic mixture of the multiple sound events 102. Referring again to Figure 1B, the SELD system 112 further collects a second digital representation 116 of the sound corresponding to the target sound event.

[0029] 1D illustrates a second digital representation 116. The second digital representation 116 is a digital representation of a target sound event 124 that the SELD system 112 is "interested" in localizing from a plurality of sound event types defined by the target class 106 of sound events depicted in FIG. 1A. Thus, the second digital representation 116 includes one or more example waveforms, or class index c, of the target sound event 124. However, this target sound event 124 may not be exactly the same as the sound event that the SELD system 112 is attempting to localize. That is, for example, the SELD system 112 may be attempting to localize a baby crying 102b, but the second digital representation 116 is derived from data of the target class 106 that includes class c of baby crying, but this crying baby is different from the crying baby 102b. Thus, an advantage of the SELD system 112 is that it can operate without sufficient training data or related sound event data and still efficiently localize any desired sound event.

[0030] In one example, the second digital representation 116 may be a one-hot vector о indicating a sound event type of the target sound event from a set of predefined sound types. c =[о1,…,о C ] and the one-hot vector о c is a C-dimensional one-hot vector, and for a class c that the SELD system 112 is interested in localizing, c =1, and о i 1A represents a set of predefined sound types. If a baby crying 102b is the target sound event 124, the class from the target class 106 corresponding to the baby crying is represented by a one-hot vector o c The sound event type is indicated by one-hot vector о cis a type of encoding process that converts categorical data into binary data so that it can be efficiently used by machine learning components. c are converted into embedding vectors (learned in the training phase, one for each class c) that are further processed by a machine learning component. In the system of FIG. 1B, the machine learning component is a neural network 118.

[0031] In one example, the second digital representation 116 includes a sound snippet that represents a target sound event 124 to be localized. The sound snippet includes a recording of an event of the same type as the target sound event 124. For example, the sound snippet represents a "query by example" case for the neural network 118. The sound snippet is processed to form the second digital representation 116, such as an "embedding vector," which is then processed by the neural network 118 in a manner similar to the embedding vector obtained in the one-hot case.

[0032] In another example, the second digital representation 116 includes a textual description or encoding of the target sound event 124. In yet another example, the second digital representation includes an imitation of a sound corresponding to the target sound event 124. Regardless of the type of representation of the second digital representation 116, the neural network 118 is guided by the second digital representation 116 to identify and localize the target sound event.

[0033] 1B, the SELD system 112 includes a neural network 118 that receives the first digital representation 114 and the second digital representation 116 and processes them to generate localization information 120 of a target sound event 124 (e.g., a baby crying 102) that forms an acoustic mixture 122 of the multiple sound events 102. The localization information 120 of the target sound event 124 indicates the location of a source of the target sound event that forms the acoustic mixture relative to the location of the acoustic sensor 108 that detects the acoustic mixture.

[0034] The neural network 118 is trained to combine the encodings of the first digital representation 114 and the second digital representation 116 to guide the processing of the first digital representation 114 based on the second digital representation 116. This is shown in FIG. 1E. To that end, the neural network 118 is trained to derive the embedding vector 128 (one-hot vector o c ) together with the first digital representation 114 of the sound mixture 122 to generate localization information 120 for a target sound event 124. Another exemplary operation of the neural network 118 is illustrated in FIG.

[0035] As shown in FIG. 1F, the neural network 118 is configured to process the first digital representation 114 and the second digital representation 116 differently. The first digital representation 114 is processed as a primary input, and the second digital representation 116 is used to "bias" the way the first digital representation 114 is processed. The neural network 118 processes the first digital representation 114 with an attention mechanism 126 that places the first digital representation 114 in the context of the embedding of the second digital representation 116, represented by the embedding vector 128. In one example, the attention mechanism 126 may be a multi-head attention mechanism, and correspondingly, the neural network 118 may be a multi-head neural network 118. As such, the multi-head neural network 118 is trained with permutation invariant training to output different position information of different target sound events to different heads of the multi-head neural network 118. This will be further described in conjunction with FIG. 2.

[0036] The neural network 118 is configured to identify the target sound event 124 interfering with other sounds in the acoustic mixture 122 by combining the encoding of the second digital representation 116 of the target sound event 124 with at least some intermediate outputs of at least some layers of the neural network 118. To that end, the neural network 118 includes multiple processing layers, such as layer 1, layer 2, ..., and layer n.

[0037] In one example, the second digital representation 116 is input to one or more feature-invariant linear modulation (FiLM) blocks, which are combined with the output of a hidden layer of the neural network 118. This combination is then presented to the next layer of the neural network 118. The use of FiLM blocks in the neural network 118 is further described in FIG.

[0038] Thus, using the various components described above, the SELD system 112 is configured to provide localization information 120 associated with a target sound event 124 .

[0039] The SELD system 112 is further described with reference to Figure 2. Figure 2 shows a schematic block diagram 200 of a SELD system 112 for estimating localization information of one or more target sound events, according to an embodiment of the present disclosure. The block diagram 200 includes an acoustic sensor 108, a target data source 202, and the SELD system 112.

[0040] The SELD system 112 includes a memory 204, a neural network 206, and a processor 208. The memory 204 includes at least one of RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other storage medium that can be used to store desired information and that can be accessed by the SELD system 112. The memory 204 includes non-transitory computer storage media in the form of volatile and / or non-volatile memory. The memory 204 may be removable, non-removable, or a combination thereof. Exemplary memory devices include solid state memory, hard drives, optical disk drives, and the like. The memory 204 stores instructions that are executed by the processor 208. Execution of the instructions by the processor 208 causes the SELD system 112 to perform a series of actions, which are described below.

[0041] The SELD system 112 receives a set of data from a target data source 202. The target data source 202 corresponds to a sound source among a plurality of sound sources that is emitting sound and needs to be localized. This second set of data is associated with a target sound event 124. In one example, there are three sound sources in a house. The three sound sources are a television, a cell phone, and a person singing. Since the person singing needs to be localized, singing is identified as the target sound event 124. This set of data is utilized to identify one or more target sound events among the plurality of sound events 102. This set of data is used to derive a second digital representation 116 from the acoustic characteristics of the target data source 202.

[0042] The SELD system 112 determines localization information 212 of one or more target sound events identified in the second digital representation 116. The localization information 212 is determined by processing the first digital representation 114 and the second digital representation 116 with a neural network 206. The neural network 206 corresponds to the neural network 118 described above. The neural network 206 is trained to consider sound events of the plurality of sound events 102 other than the identified target sound event as directional interference 104. The directional interference 104 is robustly ignored by combining the second digital representation 116 of the desired target sound event with at least one intermediate output received after processing the first digital representation 114 with the neural network 206. In general, the robust ignoring of the directional interference 104 improves the efficiency of the SELD system 112 in determining the localization information 212 and reduces the chance of error. In one example, the neural network 206 is a deep neural network (DNN). In general, a DNN is an artificial neural network (ANN) that has multiple layers between an input layer and an output layer.

[0043] In one embodiment, the neural network 206 is a multi-head neural network trained with permutation-invariant training to output different localization information for different target sound events to different heads of the multi-head neural network. A multi-head neural network is required when multiple instances of a target sound event type are simultaneously sounding from different locations. The multi-head neural network includes multiple output heads, each output head corresponding to a different instance of the target sound event. For example, if two babies are crying at the same time, each output head will include location information for one baby. Thus, output localization information is generated from different output heads of the multi-head neural network for different target sound events. Furthermore, permutation-invariant training assumes that the training data includes a set of training data components that are not ordered. The neural network 206 can identify the desired target class by processing the encoding vector or embedding vector of each target class identified by the second digital representation 116 of the target class of the target sound event 124. The output of the permutation invariant trained multi-head neural network 118 includes multiple output heads for outputting localization information from the SELD system 112 .

[0044] Additionally, the SELD system 112 outputs localization information 212 of the identified target sounds, as further described in the exemplary implementation shown in FIG.

[0045] 3 illustrates an example representation 300 of the operation of the SELD system 112, according to an embodiment of the present disclosure. The acoustic sensor 108 transmits a first digital representation 302 of an acoustic mixture of multiple sound events 102 to the SELD system 112. The first digital representation 302 undergoes a Short-time Fourier Transform (STFT) 304. The STFT 304 is a Fourier-related transform used to determine sinusoidal frequency and phase components of localized portions of the first digital representation 302 as it changes over time. Additionally, one or more intensity vectors 306 are calculated to measure spatial characteristics of the multiple sound signals represented by the first digital representation 302. In general, the intensity vectors determine the relative position of one point / object / source with respect to another point / object / source. Therefore, the SELD system 112 utilizes one or more intensity vectors 306 to represent spatial information in the sound signals, which, when combined with the STFT 304, classifies the multiple sound signals into different classes of multiple sound events 102. Furthermore, the SELD system 112 performs training of the neural network 206 based on the multiple sound signals contained in the acoustic mixture of sounds represented by the first digital representation and the output received after the STFT 304. During training, the SELD system 112 performs a concatenation 308 of the output received after the STFT 304 and after assigning the one or more intensity vectors 306. The concatenated data obtained by the operation of concatenation 308 is sent to the neural network 206. The neural network 206 is trained to estimate a target sound event among the multiple sound events 102, as well as a target acoustic source and a target class of the target sound event from the multiple classes, utilizing the second digital representation, as described above. The SELD system 112 also functions as a class-conditioned SELD network for determining the localization information 212 of the identified target sound event. Class-Conditional SELD networks are further explained in FIG.

[0046] 4A illustrates an architecture of a class-conditional SELD network 400a according to some embodiments of the present disclosure. The class-conditional SELD network 400a includes at least one FiLM block 402 directed to one or more convolution blocks 404. The at least one FiLM block 402 and the one or more convolution blocks 404 are trained to identify a target sound event and estimate localization information 212 of the identified target sound event. In one example, the class-conditional SELD network 400a includes a FiLM block 1402a, a FiLM block 2402b, and a FiLM block 3402c. In addition, the class-conditional SELD network 400a includes a Conv block 1404a, a Conv block 2404b, and a Conv block 3404c.

[0047] The FiLM blocks 402a, 402b, and 402c output a set of parameters (γ,β)=f(e) used to scale and shift the learned feature map M of the baseline SELD network, where FiLM(Mi|γi,βi)=γiMi+βi, where i refers to the channel index, and γ and β correspond to trainable parameters that result in a different linear / affine transformation for every channel. In general, the FiLM layer applies a feature affine transformation to its input. In addition, a FiLM operation 406 is applied to the feature maps of each convolution block 404a, 404b, and 404c in the baseline SELD network. The FiLM operation 406 is denoted as (1+γ)*x+β, where x is the input.

[0048] The FiLM block 1402a is directed to the Conv block 1404a, and then the FiLM operation 406 is applied to the feature map of the Conv block 1404a. In addition, the FiLM block 2402b is directed to the Conv block 2404b, and then the FiLM operation 406 is applied to the feature map of the Conv block 2404b. The FiLM block 3402c is directed to the Conv block 3404c, and then the FiLM operation 406 is applied to the feature map of the Conv block 3404c. Furthermore, the feature maps of each of the Conv blocks 404a, 404b, and 404c are sent as output to a bidirectional gated recurrent unit (BiGRU) 408. In general, the BiGRU is a sequence processing model consisting of two GRUs (a GRU that takes input in the forward direction and a GRU that takes input in the backward direction). The BiGRU 408 is a bidirectional recurrent neural network with only input gates and forget gates. In addition, the outputs of both the forward neural network and the backward neural network are connected in the same or a single output layer. Furthermore, the output generated by BiGRU 408 is sent to the linear and dropout layer 410. In general, the linear layer can learn the average correlation rate between the output and the input. For example, if x and y are positively correlated, w will be positive, and if x and y are negatively correlated, w will be negative. If x and y are completely independent, w will be near 0. In general, dropout is a technique used to prevent the model from overfitting. The output generated by the linear and dropout layer 410 is further sent to the linear layer 412. The linear layer 412 has a tanh activation function. The tanh activation function maps to values ​​between -1 and 1. The linear layer 412 generates three outputs, namely, the estimated x, y, and z coordinates of one or more target sound events. During the training phase, localization information including the DOA (azimuth and elevation) of one or more target sound events is present.The DOA is converted to Cartesian (x,y,z) coordinates using typical formulas for converting between spherical and Cartesian coordinates, and the radius of the DOA is set to 1 while the target source is active and 0 while the target source is inactive. Then, during the training phase, the x,y,z output from the linear layer 412 is compared to the ground truth x,y,z data to calculate the mean squared error (MSE) 414. In addition, the x,y,z coordinates output by the linear layer 412 are converted to an estimated direction of arrival 416 of the identified target sound event by converting the x,y and z position coordinates of the identified target sound event to azimuth and elevation angles.

[0049]

number

[0050] In one example, the architecture shown in FIG. 4B is used to detect multiple target events.

[0051] 4B illustrates the architecture of a class-conditional SELD network 400b according to some embodiments of the present disclosure. Class-conditional SELD network 400b is similar to class-conditional SELD network 400a, except for the addition of blocks 417 and 418 to enable multiple target event detection.

[0052] Therefore, the single output class conditional SELD network 400a is converted to output multiple estimated source locations as shown in FIG. 4B. The network 400b in FIG. 4B illustrates the maximum number of possible simultaneous output events (N), and the SELD network 400b outputs a separate location representation for each of the N sources 417. During training, since all network outputs correspond to target events, the order in which the network chooses to output the target events may not match the order of events in the ground truth signal, so permutation invariant training (PIT) 418 is required. PIT calculates the mean squared error (MSE) for all possible orderings between the ground truth and the output locations. For example, for N=2, there are two possible orderings: 1. (ground_truth_1->output_1, ground_truth_2->output_2) and 2. (ground_truth_1->output_2, ground_truth_2->output_1). PIT 418 then uses the ordering with the lowest MSE as the error signal for backpropagation during neural network training.

[0053] Neural network architectures 400a and 400b represent class-conditioned SELD networks that include at least one FiLM block 402 directed to one or more convolutional blocks 404. The at least one FiLM block 402 and the one or more convolutional blocks 404 are trained to identify a target sound event and estimate localization information of the target sound event.

[0054] The FiLM block 402 is described in further detail in Figures 5A and 5B.

[0055] 5A illustrates a block diagram 500a of the FiLM block 402 and the convolution block 404 according to an embodiment of the present disclosure. The second digital representation S is used as an input to at least one FiLM block 402. The FiLM block 402 includes an embedding layer 402aa, a linear layer 402bb, a dropout layer 402cc, a tile layer 402dd, and an output layer 402ee. The one-hot vector o c is passed through a trained embedding layer 402aa to obtain an embedding e ∈ R N Further, the target class information is passed through a linear layer 402bb and a dropout layer 402cc. The linear layer 402bb is utilized to learn a mapping of the embedding dimension N to the dimension of the feature map on which the FiLM conditioning is applied. In addition, the dropout layer 402cc randomly sets input units to 0 at each step of the training phase with a frequency of rate, which prevents overfitting. Input units that are not set to 0 are further scaled up by 1 / (1-rate) so that the sum of the overall input units remains unchanged. In general, the dropout layer 402cc is only applied when training is set to "true" to ensure that no values ​​are dropped during inference.

[0056] Furthermore, the FiLM block 402 utilizes a tile layer 402dd. The tile layer 402dd represents copying the conditioning information across time steps, since the conditioning inputs representing the general acoustic properties of the target class are the same at all time points. The trained model is then used to infer outcomes regarding the DOA of one or more target sound events 124. Furthermore, the FiLM block 402 outputs a set of parameters 402ee(γ,β)=f(e) that are used to scale and shift the learned feature map M of the baseline network, where FiLM(Mi|γi,βi)=γiMi+βi, where i refers to the channel index.

[0057] The FiLM block 402 is directed to one or more convolution blocks 404. In addition, each of the one or more convolution blocks 404 includes a 2D convolution layer 404aa, a batch normalized feature 404bb, a rectified linear activation function (ReLU) 404cc, a max pooling layer 404dd, and a dropout layer 404dd. The input of the 2D convolution layer 404aa includes a 64-dimensional mel spectrogram and a 3-channel first-order Ambisonic (FOA) intensity vector. In general, the 2D first-order Ambisonic format consists of one signal corresponding to an omnidirectional pickup pattern (called W) and two signals corresponding to a figure-of-eight pickup pattern aligned with the Cartesian axes (X and Y). In one embodiment, a filter or kernel in the 2D convolution layer slides over the input data to perform element-wise calculations. The filter or kernel then sums the results into one output. The kernel performs the same operation for each position the kernel slides to transform the 2D feature matrix into a different 2D feature matrix. Additionally, the convolution block 404 utilizes a batch normalization feature 404bb technique to train the deep neural network 206 that standardizes the input to the layer for each mini-batch.

[0058] The convolution block 404 includes a ReLU 404cc, which is a piecewise linear function that outputs the input directly if the input is positive and zero otherwise. In general, a ReLU does not activate all neurons simultaneously.

[0059] Furthermore, the convolution block 404 utilizes a max pooling layer 404dd. The max pooling layer 404dd is a two-dimensional layer. The max pooling layer 404dd downsamples the input along with its spatial dimension by taking the maximum value of an input window (of a size defined by the pool size) for each channel of the input. Furthermore, the convolution block 404 includes a dropout layer 404ee to prevent overfitting.

[0060] The neural network 118 processes the first digital representation 114 and the second digital representation 116 using a combination of FiLM blocks 402 and Conv blocks. To do so, the second digital representation 116 is presented to one or more FiLM blocks 402. The output of the FiLM blocks 402 is combined with the output of an intermediate layer of the neural network 118, and this combination is presented to the next layer of the neural network 118. This is shown in Figures 4A and 4B.

[0061] FIG. 5B describes FiLM block 403, a slight modification of FiLM block 402, when the second digital representation S includes one or more example waveform representations that indicate one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event 124. In this case, the target sound event 124 may not have been seen during training, but an example(s) of the type of sound to be localized is available at inference time. In this case, one or more example waveforms S are first processed by feature extraction block 403aa and converted into an embedding vector. An example of the feature extraction process could be the average frequency spectrum of all example waveforms, or a separate network that inputs the example waveforms and outputs a single embedding vector. The remaining components of the FiLM block (403bb, 403cc, 403dd, 403ee) remain the same as when the second digital representation is in a different format, such as in the form of a one-hot vector.

[0062] In this way, the SELD 112 system uses different combinations of the FiLM blocks 402 or 403 and the Conv block 404 to localize the target sound event 124, even with less training data. The SELD system 112 can then be used in many audio processing applications.

[0063] 6 is a block diagram 600 of a SELD system 112 for performing audio signal processing, according to an embodiment of the disclosure. In some exemplary embodiments, the block diagram 600 includes an acoustic sensor 108 or sensors that collect data from an environment 602 including multiple sound signals 606 of multiple sound events 102. In addition, the acoustic sensor 108 collects data associated with one or more target sound events from a target data source(s) 210.

[0064] The SELD system 112 includes a hardware processor 608. The hardware processor 608 communicates with a computer storage memory, such as memory 610. The memory 610 contains stored data, including algorithms, instructions, and other data, executed by the hardware processor 608. The hardware processor 608 may include two or more hardware processors, depending on the requirements of a particular application. The two or more hardware processors may be either internal or external. The SELD system 112 is integrated with other components, including output interfaces and transceivers, among other devices.

[0065] In some alternative embodiments, the hardware processor 608 is connected to a network 604 that communicates with the target data source 210. The network 604 may include, by way of non-limiting example and without limitation, one or more local area networks (LANs) and / or wide area networks (WANs). The network 604 may also include enterprise-wide computer networks, intranets, and the Internet. The SELD system 112 includes one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources may include a single device or multiple devices that cooperate in a distributed environment of the network 604.

[0066] In some other alternative embodiments, the hardware processor 608 is connected to a network-enabled server 614 that is connected to the client device 616. The network-enabled server 614 corresponds to a dedicated computer connected to a network that executes software intended to process client requests received from the client device 616 and provide an appropriate response on the client device 616. The hardware processor 608 is connected to an external memory device 618 that stores all the necessary data used for the DOA estimation, and to a transmitter 620. The transmitter 620 facilitates data transmission between the network-enabled server 614 and the client device 616. Additionally, an output 622 for one or more target sound events is generated.

[0067] The target data source(s) 202 include data resources for training the deep neural network 206. For example, in one embodiment, the training data includes acoustic signals of multiple speakers speaking simultaneously. The training data also includes acoustic signals of a single speaker speaking alone, acoustic signals of single or multiple speakers speaking in a noisy environment, and acoustic signals of a noisy environment.

[0068] The SELD system 112 also includes a third party device 624, which may include any type of computing device, such as an automatic speech recognition (ASR) system. For example, the third party device may include, but is not limited to, a computer device or a mobile device. A mobile device may include, but is not limited to, a personal data assistant (PDA), a smartphone, a smart watch, smart glasses (or other wearable smart device), an augmented reality headset, a virtual reality headset, a laptop, a tablet, a remote control, an entertainment system, a vehicle computer system, an embedded system controller, an appliance, a home computer system, a security system, a consumer electronic device, or other similar electronic device. In addition, the mobile device may include, but is not limited to, a microphone or line input terminal for receiving voice information, or a communication component (e.g., Wi-Fi capability) for receiving such information from another source, such as the Internet. In one exemplary embodiment, the mobile device may receive input data, such as a sound signal. For example, the input data may include sound events of one speaker into a microphone of the mobile device while multiple speakers in a room are speaking. The input data may be processed by an ASR in the mobile device using the SELD system 112 to determine a class of sound events. The SELD system 112 enhances the input data by reducing noise in the speaker's environment, separating the speaker from other speakers, or enhancing the speech signal to enable the ASR to output the DOA of the sound event.

[0069] In addition, the SELD system 112 stores input data in storage 612. Storage 612 stores data, computer instructions (e.g., software program instructions, routines, or services), and / or information including data related to the deep neural network 206. For example, storage 612 stores data from the target data source(s) 210, one or more deep neural network models, information for generating and training the deep neural network models, and computer-usable information output by the one or more deep neural network models.

[0070] In addition to estimating the DOA, it is also possible to include the distance to the target sound event in the localization data output by the network. If training data for estimating distance is available, this training data can be used as an additional output of the neural network. Alternatively, the DOA can be estimated from multiple subarrays and triangulation can be used to estimate the distance from multiple DOAs at known subarray locations.

[0071] FIG. 7 shows a flowchart illustrating a method 700 for localization of one or more target sound events according to various embodiments of the disclosure. The method is performed by the SELD system 112. The flowchart begins at step 702. Following step 702, at step 704, the method includes collecting, at the SELD system 112, a first digital representation of an acoustic mixture of sounds detected by an acoustic sensor, such as an acoustic mixture 122 of sounds from multiple sources 102 detected by an acoustic sensor 108. The acoustic sensor 108 includes multiple microphones arranged in a predetermined pattern. The first digital representation is the first digital representation 114 shown in FIG. 1C, which corresponds to a representation of spatial and spectral features of the acoustic mixture of the multiple sound events 102. In one example, the first digital representation 114 includes at least one of a short-time Fourier transform of the input acoustic mixture 122 of the multiple sound events 102 and a directional sound field intensity feature of the acoustic mixture 122 of the multiple sound events 102.

[0072] At step 706, the method 700 includes receiving a second digital representation associated with one or more target sound events from a target data source. For example, FIG. 1D illustrates a second digital representation 116 representing spatial, spectral and class information associated with a target sound event of the plurality of sound events 102. The method 700 further includes creating a plurality of classes for classifying the plurality of sound events 102. The method 700 includes utilizing the second digital representation to identify one or more target sound events of the plurality of sound events 102 from the plurality of classes, along with a target acoustic source and a target class of the one or more target sound events. In one embodiment, the second digital representation 116 includes a one-hot vector for identifying a sound event type of the target sound event from among a set of predefined sound event types, such as the set of target classes 106 illustrated in FIG. 1A.

[0073] In step 708, the method 700 includes processing the first digital representation 114 and the second digital representation 116 by a neural network 118 to determine localization information 120. The localization information 120 includes a direction of arrival of the target sound event from a source of the target sound event toward the acoustic sensor and a distance between the source of the target sound event and the acoustic sensor. The neural network 118 is trained to consider sound events of the plurality of sound events 102 other than the identified target sound event 124 as directional interference 104. The directional interference 104 is identified by combining the second digital representation 116 of the identified target sound event 124 with at least one intermediate output received after processing the first digital representation 114 by the neural network 118.

[0074] In step 710, the method 700 includes outputting the localization information, which may be used for further processing in an application, such as an audio processing application. The method ends in step 712.

[0075] 8 illustrates a use case 800 for estimating localization information including DOA of a sound event according to some other exemplary embodiments of the present disclosure. Use case 800 corresponds to a factory floor scene including one or more speakers, such as speaker 802A and speaker 802B. The factory floor has multiple sound events and noises due to the operation of various industrial machines. The factory floor includes an audio device 804 to facilitate communication between a factory floor control operator (not shown) and one or more speakers 802A and 802B within the factory floor. Audio device 804 includes SELD system 112.

[0076] In an illustrative example scenario, audio device 804 is transmitting a voice command addressed to a person 802A managing a factory floor. The voice command states, "Report the status of machine 1." Speaker 802A utters, "Machine 1 running." However, the speech signal of speaker 802A's utterance is mixed with noise from the machine, noise from the background, and other utterances from speaker 802B in the background. Such noise and reverberant signals are mitigated by SELD system 112. SELD system 112 identifies a target sound event and outputs a DOA for the target sound event. The output DOA is further used to identify the sound of speaker 802A and ignore sounds of other sources. The response from speaker 802A is then sent back to audio device 804 in a focused and targeted manner to perform better voice control operations in the factory floor scene depicted in FIG. 8.

[0077] FIG. 9 illustrates another use case 900 for identifying a target sound event and estimating the direction of arrival of the target sound event according to some exemplary embodiments of the present disclosure. The use case 900 corresponds to an in-vehicle infotainment system 904. The in-vehicle infotainment system 904 is implemented in a vehicle, such as a manually operated vehicle, an automated vehicle, or a semi-automated vehicle. The vehicle is occupied by a plurality of people, such as person 902a, person 902b, and person 902c. The in-vehicle infotainment system 904 includes a SELD system 112. For example, the in-vehicle infotainment system 904 is remotely connected to the SELD system 112 via a network. In some alternative exemplary embodiments, the SELD system 112 is integrated within the in-vehicle infotainment system 904.

[0078] The in-vehicle infotainment system 904 also includes one or more microphones to receive a mixture of acoustic signals. The mixture of acoustic signals includes speech signals from the persons 902a, 902b, and 902c, and external noise signals, such as honking of other vehicles. In this case, the person 902a is speaking to the person 902c, and the other person 902b is uttering / singing louder than the persons 902a and 902c. The speech from the person 902c is interfering with the speech commands of the persons 902a and 902c. For example, the speech command of the person 902a is "My gas tank is almost empty" and the speech command of the person 902c is "Find the nearest gas pump". In such a case, the SELD system 112 processes the speech of each of the persons 902a, 902b, and 902c and identifies that the target sound event belongs to the speech command of the persons 902a and 902c. The SELD system 112 considers all speech except that of persons 902a and 902c as directional interference and outputs the DOAs of the target sound events accordingly, which are further used to distinguish the sounds of persons 902a and 902c and ignore sounds of other sources.

[0079] One skilled in the art to which the inventions described herein pertain, having the benefit of the teachings presented in the above description and the associated drawings, will come up with many modifications and other embodiments of these inventions. It is to be understood that the invention is not limited to the particular embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, while the above description and the associated drawings describe example embodiments in the context of certain example combinations of elements and / or features, it should be understood that different combinations of elements and / or features may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or features other than those expressly set forth above are also intended to be set forth in some of the appended claims. Although specific terms have been used herein, they are used for purposes of general description only and not of limitation.

Claims

1. 1. A sound event localization and detection (SELD) system for localizing one or more target sound events, the SELD system comprising at least one processor and a memory having instructions stored thereon, the instructions, when executed by the at least one processor, providing the SELD system with: collecting a first digital representation of an acoustic mixture of sounds of a plurality of sound events detected by an acoustic sensor; receiving a second digital representation of a class of sounds corresponding to a target sound event within the acoustic mixture; and and processing the first digital representation and the second digital representation with a neural network, the neural network being trained to generate target sound event localization information indicative of a location of a source of the target sound event within the acoustic mixture relative to a location of the acoustic sensor detecting the acoustic mixture. The SELD system further comprises: outputting the localization information of the source of the target sound event.

2. 2. The SELD system of claim 1, wherein the target sound event is identified by the neural network by combining the second digital representation of the target sound event with at least some intermediate outputs of at least some layers of the neural network that process the first digital representation of the acoustic mixture.

3. 2. The SELD system of claim 1, wherein the localization information includes one or a combination of a direction of arrival of the target sound event from a source of the target sound event toward the acoustic sensor and a distance between the source of the target sound event and the acoustic sensor.

4. 2. The SELD system of claim 1, wherein the second digital representation identifies multiple target sound events, the neural network is a multi-head neural network, and the multi-head neural network is trained with permutation invariant training to output different location information for different target sound events to different heads of the multi-head neural network.

5. 2. The SELD system of claim 1, wherein the neural network identifies the target sound event that is interfering with other sounds in the sound mixture by combining the encoding of the second digital representation of the target sound event with at least some intermediate outputs of at least some layers of the neural network that process the first digital representation of the sound mixture.

6. 2. The SELD system of claim 1, wherein the neural network processes the first digital representation using attention that places the first digital representation in the context of an embedding of the second digital representation.

7. The processor, inputting said second digital representation into one or more feature invariant linear modulation (FiLM) blocks; combining the output of the one or more FiLM blocks with an output of a hidden layer of the neural network; The SELD system of claim 1 configured to present the combination to a next layer of the neural network.

8. 8. The SELD system of claim 7, wherein the neural network includes a class-conditional SELD network for determining the localization information, the class-conditional SELD network including at least one FiLM block directed to one or more convolutional blocks, the at least one FiLM block and the one or more convolutional blocks being trained to identify the target sound event and estimate the localization information.

9. The SELD system of claim 8 , wherein each of the FiLM blocks includes an embedding layer, a linear layer, a dropout layer, a tile layer, and an output layer.

10. 10. The SELD system of claim 8, wherein each of the one or more convolutional blocks includes a two-dimensional convolutional layer, a batch normalization layer, a rectified linear activation function (ReLU), a max pooling layer, and a dropout layer.

11. The SELD system of claim 8 , wherein the neural network is a convolutional recurrent neural network.

12. The SELD system of claim 1 , wherein the second digital representation comprises a one-hot vector indicating a sound event type of the target sound event from a set of predetermined sound event types.

13. The SELD system of claim 1 , wherein the second digital representation includes one or more example waveform representations indicative of one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event.

14. The SELD system of claim 13 , wherein the neural network converts the one-hot vector into an embedding vector and co-processes the embedding vector with the first digital representation of the sound mixture to generate the localization information.

15. 15. The SELD system of claim 14, wherein the neural network uses one or more feature invariant linear modulation (FiLM) blocks to jointly process the embedding vector with the first digital representation of the sound mixture.

16. The SELD system of claim 1 , wherein the first digital representation comprises a waveform representation of spatial and spectral characteristics of the acoustic mixture.

17. The SELD system of claim 1 , wherein the acoustic sensor includes a plurality of microphones arranged in a predetermined pattern operatively connected to the SELD via a wired or wireless communication channel.

18. 1. A method for localization of a target sound event, the method comprising: collecting a first digital representation of an acoustic mixture of sounds of a plurality of sound events detected by an acoustic sensor; receiving a second digital representation of a sound corresponding to the target sound event; and processing the first digital representation and the second digital representation with a neural network, the neural network being trained to generate localization information indicative of a location of a source of the target sound event relative to a location of the acoustic sensor, the neural network identifying the target sound event interfering with other sounds in the acoustic mixture, the method further comprising: outputting the localization information of the source of the target sound event.

19. 20. The method of claim 18, wherein the target sound event is identified by the neural network by combining the second digital representation of the target sound event with at least some intermediate outputs of at least some layers of the neural network that process the first digital representation of the sound mixture.

20. 20. The method of claim 18, wherein the second digital representation includes one or more example waveform representations indicative of one or a combination of spatial characteristics, spectral characteristics, or class information of the target sound event.

Images