Method and system for reverberation modeling of speech signals

Abstract

A system and method for reverberation reduction is disclosed. A first deep neural network (DNN) generates a first estimate of a target direct path signal from a mixture of acoustic signals comprising the target direct path signal and a reverberation of the target direct path signal. A filter that models a first estimated room impulse response (RIR) is estimated. The filter, when applied to the first estimate of the target direct path signal, generates a result of a residual between the mixture of closest acoustic signals and the first estimate of the target direct path signal according to a distance function. The estimated filter is used to model the RIR.

Description

Technical Field

[0001] The present disclosure relates generally to audio signal processing, and more particularly to methods and systems for reverberation modeling of speech signals. Background Art

[0002] Typically, in a closed room, reverberation of an audio signal (e.g., speech) occurs in modern hands-free voice communications, such as teleconferencing and interactions with smart devices (such as microphones in smart speakers). In such a closed room, the voice signal propagates in the air and can be reflected by walls, floors, ceilings, and any other objects in the room before being captured by the microphone. Reverberation is the multipath propagation of a voice signal from a source or speaker to a receiving end (such as a microphone). This voice reverberation occurs when sound is reflected from a surface in an environment. Some sounds may be absorbed by the surface, due to which the voice signal is attenuated multiple times. The reflection and absorption of sound by the surface can generate multiple attenuated copies and delayed copies of the voice signal. Multiple attenuated copies and delayed copies reduce the quality of the voice, which may hinder the performance of an automatic speech recognition (automatics speech recognition, ASR) system or any speech / audio processing system. For example, ASR may generate inaccurate output due to audio input with reduced voice quality.

[0003] Speech reverberation can be mitigated by removing the effects of reverberation from the sound. This removal of the effects of reverberation is known as dereverberation. Dedereverberation can include identifying and distinguishing the direct path signal from the attenuated replica and the delayed replica. The direct path signal corresponds to the signal that the sound propagates when the source and the microphone are in line of sight. However, identifying and distinguishing the direct path signal from the replica can be difficult, especially when the reverberation is high and noise from non-steady-state sources is present. For example, an environment (such as an enclosed room with a non-steady-state source such as an air conditioning system) may have high room reverberation. Reducing reverberation due to noise from an air conditioning system or any multi-source environmental noise can be challenging. Multi-source environmental noise can also correspond to a scenario where multiple people are speaking in the environment.

[0004] For this reason, reverberation modeling of such closed environments is advantageous for use in a variety of applications such as speech dereverberation, room impulse response modeling, acoustic modeling, and the like.

[0005] Therefore, there is a need to overcome the above problems. More specifically, there is a need to develop a method and system for reverberation modeling of speech signals while overcoming reverberation conditions and non-stationary noise in a reverberant environment. Summary of the invention

[0006] It is an object of some embodiments to develop methods and systems for reverberation modeling of speech signals. It is another object of some embodiments to perform dereverberation of speech signals using deep learning techniques. Dedereverberation of speech signals can be extended to tasks such as reverberation reduction, speech enhancement, speaker separation, etc.

[0007] Some embodiments are based on the understanding that clean speech exhibits a spectrum-time pattern. This spectrum-time pattern is a unique pattern exhibited in the time-frequency domain and can provide informative cues for reducing reverberation. While some patterns originate from the structure of the speech signal itself, some patterns can also correspond to a linear filter structure of reverberation (i.e., reflections of sound waves), which is a characteristic of the physical space in which the recording is made, including all objects, structures, or entities present in the space, as well as the location of the source speech signal and the receiver (such as a microphone recording the signal). The signal generated from the source signal at the microphone location and its reflections on the walls and surfaces of objects or people in the space can be described using this linear filter structure, representing the effect of reverberation on the input signal as a linear convolution of the input signal and a room impulse response (RIR). The input signal is the original source signal, also known as the dry source signal. The room impulse response is a representation of the effect of the space and everything inside it on the input signal. An estimate of the RIR between the source location and the receiver location can be recorded in a physical space such as a room, for example by playing an impulse sound as a short duration time domain signal (e.g., an empty pistol or balloon explosion) at the source location in the room and recording the subsequent signal at the receiver location. The impulse excites the room and produces a reverberant impulse signal that can be used to estimate the RIR. The reverberation of a dry source sound signal that will be played at the same source location and recorded at the same receiver location can then be modeled by convolving the dry source signal with the estimated RIR. To this end, some embodiments also aim to estimate a basic filter for approximating or modeling the RIR. In some example embodiments, the RIR is estimated based on a linear regression problem solved per frequency in the time-frequency domain. The filter estimate that models the RIR can be used to identify delayed and attenuated copies of the input signal for de-reverberation of the speech signal.

[0008] In addition, such linear filters can be used as a regularization means to improve the dereverberation process. For example, the linear filter as a regularization means prevents the model used for the dereverberation process from overfitting to the training data. Some embodiments are based on the following recognition: the linear filter structure can be used for single-channel and multi-channel reverberation speaker separation and dereverberation tasks. A combination of linear prediction and deep learning. To this end, deep learning techniques supported by convolutional prediction can be used for dereverberation in environments with noisy signals, reverberation of audio signals, etc. Convolutional prediction is a linear prediction method for speech dereverberation under reverberant conditions, which relies on source estimates obtained by a deep neural network (DNN) and utilizes a linear filter structure between the source estimate and the reverberated version of the source signal within the observed input signal.

[0009] To obtain source estimates, a DNN is trained in the time-frequency domain or time domain to predict the target speech from the reverberant speech. The target speech corresponds to the target direct path signal between the source and the receiver (such as a microphone). This method can exploit prior knowledge of speech patterns.

[0010] Previous work has also attempted to use some form of linear filter structure in order to perform dereverberation. For example, weighted prediction error (WPE) can be used for dereverberation of speech signals. The WPE method calculates an inverse linear filter based on variance normalized delayed linear prediction. The calculated linear filter is applied to past observations of a reverberation and potential noise mixed input signal to estimate the late reverberation of a target source signal within the mixed input signal from past observations of the reverberation to perform dereverberation. The estimated late reverberation is subtracted from a mixture of acoustic signals received from different sources to estimate the target speech signal in the mixture of acoustic signals. In some embodiments, the filter can also be estimated using the time-varying power spectral density (PSD) of the target speech signal. PSD is the distribution of the power of a signal over the frequency range of the signal. Such a linear filter can be iteratively estimated using WPE in an unsupervised manner. However, the iterative process of WPE for filter estimation may result in suboptimal results and is computationally expensive.

[0011] To overcome the above-mentioned defects of WPE, the iterative process for filter estimation can be replaced as in the DNN-based WPE (DNN-WPE) method. DNN-WPE uses the DNN-estimated amplitude as the PSD of the target speech signal for filter estimation. However, DNN-WPE may not reduce early reflections because it requires a strict non-zero frame delay to avoid trivial solutions, and there may be no mechanism to utilize the DNN-estimated phase for filter estimation. DNN-WPE may also lack robustness to interference due to noisy signals. For example, DNN-WPE can estimate a filter that associates past noise observations with current noise observations, thereby limiting the filter estimation accuracy. In addition, DNN-WPE can directly use the linear prediction result as its output, resulting in partial or minimal reduction of reverberation.

[0012] To this end, another object of some embodiments is to remove early reflections and late reverberation for dereverberation. A convolutional prediction method can be used to remove early reflections and late reverberation. The convolutional prediction method uses both the amplitude and phase estimated by the DNN to perform filter estimation. The convolutional prediction method also provides a closed form solution for the linear filter (just like the previously mentioned DNN-WPE method), which can be suitable for online real-time processing applications and can be jointly trained with other DNN modules such as acoustic models.

[0013] In some embodiments, a first estimate of a target direct path signal is used to determine a filter using a convolution prediction method, wherein the filter makes the application of the filter to the target direct path estimate as close as possible to the residual between a mixture of acoustic signals and the first estimate of the target direct path signal under a certain weighted distance function. The filter models the reverberation of the target direct path signal within the mixture. Alternatively, and functionally equivalently, the filter can make the application of the filter to the target direct path estimate as close as possible to the mixture of acoustic signals under a certain weighted distance function, in which case the filter will model the reverberation, including the contribution of the direct path leading to the target direct path signal. Each filter can be easily obtained from the other filter, and we will consider them to be equivalent. In addition, the filter is applied to the first estimate of the target direct path signal in the time-frequency domain. When the filter is applied to the first estimate of the target direct path signal, a result of identifying a delayed copy and an attenuated copy of the estimated direct path signal is obtained from the mixture of acoustic signals. The delayed copy and the attenuated copy are derivative signals of the target direct path signal reflected in multiple paths due to reverberation in this article. For example, the target direct path signal is reflected in different directions by different objects in an environment (such as a room). The delayed and decayed copies so identified may be removed from the mixture of acoustic signals for dereverberation. Removing the delayed and decayed copies produces a mixture with reduced reverberation.

[0014] The result obtained by the filter when applied to the first estimate of the target direct path signal is a residual between the mixture of the acoustic signal and the first estimate of the target direct path signal that is closest to the above construction according to the distance function. The distance function is a weighted distance between the filtered direct path signal and the residual obtained by subtracting the direct path estimate from the mixture, and the weight at each time-frequency point in the time-frequency domain is determined by one or a combination of the mixture of the received acoustic signal and the estimate of the target direct path signal. In some embodiments, the distance function is based on a least squares distance. In addition, the result of applying the filter to the first estimate of the target direct path signal is removed from the mixture of the acoustic signal to obtain a reverberation-reduced mixture of the target direct path signal.

[0015] In some embodiments, the reverberation-reduced mixture is input to a second DNN. The second DNN outputs a second estimate of the target direct path signal, which may be an improved estimate of the target direct path signal compared to a previous estimate of the target direct path signal. The second DNN may also perform steps similar to those of the first DNN. However, in some embodiments, the second DNN may take as input a different set of signals, such as one or a combination of a mixture of acoustic signals, a reverberation-reduced mixture, and an estimate of the direct path signal. The second estimate of the target direct path signal may be similarly used to obtain an improved filter using a convolutional prediction method, wherein the improved filter makes the application of the improved filter to the improved estimate of the target direct path signal as close as possible to the residual between the mixture of acoustic signals and the improved estimate of the target direct path signal under a certain weighted distance function. Alternatively, and functionally equivalently, the improved filter may make the application of the improved filter to the improved estimate of the target direct path signal as close as possible to the mixture of acoustic signals under a certain weighted distance function, in which case the improved filter will model the reverberation, including the contribution of the direct path that leads to the target direct path signal.

[0016] In some embodiments, the downstream DNN uses one or more filter estimates output by the first estimation or an improved filter estimate output by the second DNN to perform a task that benefits from the RIR estimation. For example, the downstream DNN is used to perform a positioning function.

[0017] Some embodiments are also based on the understanding that each speaker, either a single speaker or multiple speakers, is convolved with a different RIR. It may also be meaningful to retrieve each RIR individually for downstream applications or for performing data augmentation. The WPE approach estimates a single filter to reduce reverberation for all sources. However, when the noise or competing speakers are louder than the target source, it may not be feasible to compute a single filter to dereverberate the mixture. Filters computed in this way are biased towards suppressing reverberation for higher energy sources. To do this, it may be necessary to estimate a dereverberation filter for each source because each source is convolved with a different RIR. The DNN-WPE approach can compute a different filter for each source, but it can only do so by using the estimated PSD for each source as a weight in the distance function that the DNN-WPE uses to estimate the linear prediction filter, which may limit the accuracy and diversity of the different filters.

[0018] In some embodiments, two DNNs (a first DNN and a second DNN) are trained for dereverberation of speech signals based on a convolutional prediction method. Initially, the first DNN of the two DNNs outputs a first estimate of a direct path signal of a target source (such as a person speaking referred to as a speaker hereinafter) from an input (i.e., a mixture of acoustic signals including the speech of a speaker). This direct path signal of the target source is referred to above as a target direct path signal. The first estimate of the target direct path signal is used to determine a filter using a convolutional prediction method, wherein the filter makes the application of the filter to the target direct path estimate as close as possible to the residual obtained by subtracting the target direct path estimate from the mixture under a certain weighted distance function. In addition, the filter is applied to a first estimate of the target direct path signal in the time-frequency domain. When the filter is applied to the first estimate of the target direct path signal, a result of identifying a delayed copy and an attenuated copy of the estimated target direct path signal is obtained from the mixture of acoustic signals. The delayed copy and the attenuated copy are derivative signals of the target direct path signal reflected in multiple paths due to reverberation in this article. For example, the target direct path signal is reflected in different directions by different objects in an environment (such as a room). The delayed and decayed copies so identified are removed from the mixture of acoustic signals for dereverberation. Removing the delayed and decayed copies generates a mixture with reduced reverberation.

[0019] The result obtained when the filter is applied to the first estimate of the target direct path signal is a residual between the mixture of the closest acoustic signal and the first estimate of the target direct path signal according to the distance function constructed above. In some embodiments, the distance function is based on a least squares distance. In addition, the result of applying the filter to the first estimate of the target direct path signal is removed from the mixture of the acoustic signal to obtain a reverberation-reduced mixture of the target direct path signal. In some embodiments, the reverberation-reduced mixture is input to a second DNN of the two DNNs. The second DNN outputs a second estimate of the target direct path signal, which can be an improved estimate of the target direct path signal compared to the first estimate of the target direct path signal.

[0020] In some embodiments, a first DNN may be trained with speaker separation as a goal. To this end, the first DNN generates a plurality of outputs corresponding to first estimates of target direct path signals for speakers of a plurality of speakers. Furthermore, the estimation of the filter and the obtaining of the reverberation-reduced mixture are repeated for each of the plurality of speakers to generate a corresponding reverberation-reduced mixture and a corresponding filter for each of the plurality of speakers. The corresponding reverberation-reduced mixtures for each of the plurality of speakers are then combined, and the reverberation-reduced combined mixtures for each of the plurality of speakers are submitted to a second DNN. The second DNN then generates a second estimate of the target direct path signal for each of the plurality of speakers.

[0021] Additionally or alternatively, the reverberation-reduced mixture (i.e., the delayed replica and the attenuated replica) can be used as additional features to a second DNN to determine a second estimate of the target direct path signal, which improves dereverberation. Additionally or alternatively, features corresponding to the delayed replica and the attenuated replica can also be used for speaker separation tasks. In some example embodiments, the delayed replica and the attenuated replica can be identified based on a linear regression problem. In some embodiments, one or a combination of the mixture of the acoustic signal and the first estimate of the target direct path signal can be provided as input to a second DNN to produce a second estimate of the target direct path signal. In some embodiments, the mixture of the acoustic signal, the first estimate, and the reverberation-reduced mixture are provided as input to a second DNN to determine a second estimate of the target direct path signal.

[0022] Some embodiments are based on the recognition that in the case of multiple speakers in a room, a corresponding filter for each individual speaker is estimated for dereverberation. In the case of multiple speakers, the mixture of acoustic signals includes speech signals from the multiple speakers. In this case, a first DNN generates a corresponding first estimate of a target direct path signal for each of the multiple speakers. In order to generate a reverberation-reduced mixture for each of the multiple speakers, the steps of determining for each speaker, a first estimate of a filter for each speaker, and submitting one or a combination of the first estimate for each speaker and the reverberation-reduced mixture can be combined and submitted to a second DNN for generating a second estimate of the target direct path signal for each of the multiple speakers.

[0023] In some cases, a mixture of acoustic signals may be received from a single channel (such as a single microphone) or from multiple channels (such as a microphone array). Each different channel measures a different version of the mixture of acoustic signals. A DNN may be trained to estimate a target direct path signal at a reference channel or at each channel. The training may be based on a complex spectrum mapping at one or more channels, wherein the DNN is trained to output an estimate of the target direct path signal at one or more channels in the time-frequency domain such that the distance between the estimate and the reference of the target direct path signal at one or more channels in the time-frequency domain is minimized. In the case of a microphone array, a beamforming output may be obtained. The beamforming output may be obtained based on statistics calculated from one or a combination of a first estimate of the target direct path signal at each microphone in the microphone array and a reverberation-reduced mixture of the target direct path signal. The beamforming output may be input to a second DNN to produce a second estimate of the target direct path signal for each speaker in a plurality of speakers. Additionally or alternatively, the beamforming output and the dereverberation results may be used as additional features of a second DNN to perform better separation and dereverberation tasks.

[0024] In some embodiments, a first DNN may be pre-trained to obtain a first estimate of a target direct path signal from a mixture of observed acoustic signals. Pre-training of the first DNN may be performed using a training data set of a mixture of acoustic signals and a corresponding reference target direct path signal in the training data set. In particular, pre-training of the first DNN may be performed by minimizing a loss function. The loss function may include one or a combination of distance functions defined based on real and imaginary (RI) components of the first estimate of the target direct path signal in the complex time-frequency domain and the RI component of the corresponding reference target direct path signal. The distance function may also be defined based on an amplitude obtained from the RI component of the first estimate of the target direct path signal in the complex time-frequency domain and a corresponding amplitude of the reference target direct path signal.

[0025] Additionally or alternatively, a distance function may be defined based on a reconstructed waveform obtained by reconstruction in the time domain from the first estimated RI component of the target direct path signal and a corresponding waveform of the reference target direct path signal.

[0026] In some alternative embodiments, a distance function may be defined based on an RI component of the first estimate in the second complex time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and a corresponding RI component of the reference target direct path signal in the second time-frequency domain.

[0027] In some alternative embodiments, a distance function may be defined based on a magnitude obtained from the RI component of the first estimate in a second complex time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and a corresponding magnitude of the reference target direct path signal in the second time-frequency domain.

[0028] In some example embodiments, the first estimate of the target direct path signal may be replaced with a second estimate of the target direct path signal to obtain an updated first estimate of the target direct signal. The steps of obtaining the first estimate, obtaining the filter, and submitting the mixture of the first estimate and the reverberation reduction may be iterated for the updated first estimate of the target direct signal to obtain an updated second estimate of the target direct signal.

[0029] In some examples, the above steps are repeated for each speaker of the plurality of speakers in the multi-speaker scenario to generate a corresponding filter for each speaker of the plurality of speakers. In addition, a portion of the received mixture of acoustic signals corresponding to a speaker of the plurality of speakers may be extracted by removing the reverberant speech of the other speakers of the plurality of speakers from the mixture of acoustic signals. An estimate of the reverberant speech of another speaker of the plurality of speakers is obtained by adding a first estimate of the target direct path signal of the other speaker to a result of applying a corresponding filter of the other speaker to the first estimate of the target direct path signal of the other speaker. After extraction, a filter for a reverberation-reduced mixture for each speaker of the plurality of speakers may be estimated based on the portion of the received mixture.

[0030] Some embodiments provide evaluation results on speech dereverberation and speaker separation, which show the effectiveness of dereverberation of speech signals based on convolutional prediction methods.

[0031] Therefore, one embodiment of the present disclosure discloses a method for estimating a room impulse response (RIR) performed by a computer. The method is performed by a processor connected to stored instructions that implement the method. The instructions perform the steps of the method when executed by the processor. The method includes receiving a mixture of acoustic signals, the mixture of acoustic signals including a target direct path signal propagating in a room and a reverberation of the target direct path signal in the room. The mixture of acoustic signals is received through a wired or wireless communication channel. The method also includes submitting the received mixture of acoustic signals to a first deep neural network (DNN) to produce an estimate of the direct path signal. The method also includes estimating a filter that models a room impulse response (RIR) representing a relationship between the direct path signal and the reverberation of the direct path signal, wherein the filter, when applied to the estimate of the direct path signal, generates a result that is closest to one of the mixture of acoustic signals and the residual between the mixture of acoustic signals and a first estimate of the target direct path signal according to a distance function. The filter is then transmitted. The filter is used for further applications, such as performing speech processing, generating augmented training data, multiple speaker separation, etc.

[0032] In some embodiments, the first DNN is trained to estimate the amplitude and phase of the direct path signal, wherein applying the filter to the estimate of the direct path signal involves computing both the estimated amplitude and phase. In addition, a distance function between the result of applying the filter to the estimate of the direct path signal and the mixture of the acoustic signal is measured in the amplitude domain and the phase domain.

[0033] In some implementations, the coefficients of the filter are estimated in the time-frequency domain.

[0034] In various embodiments, the direct path signal is a signal transmitted from a source of the original signal to the sensor. To this end, the direct path signal represents the original signal at the source as it would be measured by the sensor without reverberation, and wherein the reverberation of the direct path signal represents one or more transmissions of the original signal from the source to the sensor over one or more paths that are longer than the shortest path, such that the RIR is modeled with respect to the direct path signal.

[0035] In some embodiments, the filter that models the RIR is a linear filter estimated using a linear convolution prediction module. The linear convolution prediction forces the linear structure of the filter to approximate the linear reflection of the direct path signal from the surface in the room.

[0036] Some embodiments provide for applying a filter that models the RIR to an external signal measured outside the room to synthesize a new acoustic mixture, wherein the new acoustic mixture provides the acoustic effect of propagation of the external signal within the room. The new acoustic mixture can be used to update the first DNN.

[0037] Some embodiments provide updating an estimate of the direct path signal based on application of a filter that models the RIR. The training data is further updated based on a mixture of the acoustic signal and the updated estimate of the direct path signal as a pseudo label. The updated training data is used to retrain the first DNN based on the updated training data.

[0038] Various embodiments provide for generating enhanced data based on a filter that models the RIR, and creating an enhanced training data set obtained by applying the filter that models the RIR to data including one or more of: external data measured outside the room, data for another mixture of acoustic signals in the training data obtained as one or more of a first estimate of a target direct path signal and a second estimate of the target direct path signal. In addition, one or more of the first DNN and the second DNN can be trained to output a first estimate or a second estimate of the direct path signal based on the enhanced training data set. In some embodiments, the enhanced training data set can also be used to train a system for automatic speech recognition or for sound event detection.

[0039] In some embodiments, the filter that models the RIR is used to perform audio analysis for at least one or a combination of room acoustic parameter analysis, room geometry reconstruction, speech enhancement, and de-reverberation of speech signals. Room acoustic parameters include at least one of the following: direct-to-reverberation ratio, reverberation time (RT60), early decay time (EDT), center time, clarity C80, and definition D50.

[0040] Therefore, another embodiment of the present disclosure discloses a system for estimating RIR. The system includes an input interface, which is configured to receive a mixture of acoustic signals through a wired or wireless communication channel, and the mixture of acoustic signals includes a direct path signal propagating in a room and a reverberation of the direct path signal in the room. The system also includes a memory storing at least a first DNN. The system also includes a processor, which is configured to submit the received mixture of acoustic signals to the first DNN to produce an estimate of the direct path signal. The processor is also configured to estimate a filter that models the RIR representing the relationship between the direct path signal and the reverberation of the direct path signal, wherein the filter, when applied to the estimation of the direct path signal, generates a result of one of the residuals between the mixture of the acoustic signal and the first estimate of the target direct path signal according to a distance function. The system also includes an output interface, which is configured to output the filter that models the RIR through a communication channel.

[0041] Further features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0042] By way of non-limiting examples of exemplary embodiments of the present disclosure, the present disclosure is further described in the following detailed description with reference to the accompanying drawings, wherein like reference numerals represent similar parts throughout the several views of the drawings. The drawings shown are not necessarily drawn to scale, with emphasis generally being placed upon illustrating the principles of the presently disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] [ Figure 1A ]

[0044] Figure 1A An exemplary representation of reverberation modeling of a speech signal according to an embodiment of the present disclosure is shown.

[0045] [ Figure 1B ]

[0046] Figure 1B An exemplary representation for reverberation modeling of a speech signal according to another embodiment of the present invention is shown.

[0047] [ Figure 2A ]

[0048] Figure 2A A schematic block diagram of a system for performing reverberation modeling on a speech signal according to an embodiment of the present disclosure is shown.

[0049] [ Figure 2B ]

[0050] Figure 2B A schematic block diagram of another system for reverberation modeling of a speech signal according to an embodiment of the present disclosure is shown.

[0051] [ Figure 2C ]

[0052] Figure 2C A schematic block diagram of another system for reverberation modeling of a speech signal according to an embodiment of the present disclosure is shown.

[0053] [ Figure 3A ]

[0054] Figure 3A A schematic block diagram of a process for performing reverberation modeling on a speech signal according to an embodiment of the present disclosure is shown.

[0055] [ Figure 3B ]

[0056] Figure 3BA representation of a room impulse response (RIR) in the time domain is shown according to an embodiment of the present disclosure.

[0057] [ Figure 3C ]

[0058] Figure 3C A representation of the application of a filter to model RIRs in a frequency bin is shown in accordance with an embodiment of the present disclosure.

[0059] [ Figure 4 ]

[0060] Figure 4 is a schematic diagram depicting an architecture for reverberation modeling of speech signals according to an embodiment of the present disclosure.

[0061] [ Figure 5 ]

[0062] Figure 5 is a schematic diagram depicting an architecture for reverberation modeling of speech signals for multiple speakers according to some embodiments of the present disclosure.

[0063] [ Figure 6 ]

[0064] Figure 6 is a schematic diagram depicting an architecture for reverberation modeling of speech signals for multiple speakers according to some other embodiments of the present disclosure.

[0065] [ Figure 7 ]

[0066] Figure 7 is a schematic diagram depicting an architectural representation for reverberation modeling for enhancing speech signals according to some embodiments of the present disclosure.

[0067] [ Fig. 8A ]

[0068] Fig. 8A A schematic diagram depicting a network architecture for reverberation modeling of speech signals according to some other embodiments of the present disclosure is shown.

[0069] [ Figure 8B ]

[0070] Figure 8B A schematic diagram depicting a network architecture for reverberation modeling of speech signals according to some other embodiments of the present disclosure is shown.

[0071] [ Figure 8C ]

[0072] Figure 8C A schematic diagram depicting a network architecture for reverberation modeling of speech signals according to some other embodiments of the present disclosure is shown.

[0073] [ Fig.8D ]

[0074] Fig.8D A schematic diagram depicting a network architecture for reverberation modeling of speech signals according to some other embodiments of the present disclosure is shown.

[0075] [ Fig.9A ]

[0076] Fig.9A A flow chart of a method for estimating a room impulse response (RIR) according to an embodiment of the present disclosure is shown.

[0077] [ Fig. 9B ]

[0078] Fig. 9B A flow chart of a method for reverberation modeling of a speech signal according to an embodiment of the present disclosure is shown.

[0079] [ Fig.10 ]

[0080] Fig.10 A tabular representation corresponding to a simulation test for dereverberation of a speech signal according to an embodiment of the present disclosure is shown.

[0081] [ Fig.11 ]

[0082] Fig.11 A tabular representation depicting evaluation results of dereverberation of a speech signal using a test dataset is shown in accordance with an embodiment of the present disclosure.

[0083] [ Fig.12 ]

[0084] Fig.12 A tabular representation depicting evaluation results of dereverberation of a speech signal using a test dataset is shown according to some other embodiments of the present disclosure.

[0085] [ Fig.13 ]

[0086] Fig.13 is a block diagram of an audio processing system according to an embodiment of the present disclosure.

[0087] [ Fig.14A ]

[0088] Fig.14A A block diagram of a system for reverberation modeling of a speech signal according to some example embodiments of the present disclosure is shown.

[0089] [ Fig. 14B ]

[0090] Fig. 14B A block diagram of a system for reverberation modeling of a speech signal according to some other example embodiments of the present disclosure is shown.

[0091] [ Fig.15 ]

[0092] Fig.15 A use case of reverberation modeling of a speech signal according to some example embodiments of the present disclosure is shown.

[0093] [ Fig.16 ]

[0094] Fig.16 A use case of reverberation modeling of a speech signal according to some other example embodiments of the present disclosure is shown.

[0095] [ Fig.17 ]

[0096] Fig.17 A use case of reverberation modeling of a speech signal according to yet other example embodiments of the present disclosure is shown.

[0097] [ Fig.18 ]

[0098] Fig.18 A use case of reverberation modeling of a speech signal according to yet other example embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0099] Although the above-mentioned drawings illustrate the embodiments of the present disclosure, other embodiments may also be envisioned, as described in the discussion. The present disclosure presents illustrative embodiments by way of representation and not limitation. Those skilled in the art may design many other modifications and embodiments that fall within the scope and spirit of the principles of the embodiments of the present disclosure.

[0100] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other cases, the apparatus and methods are shown only in block diagram form to avoid obscuring the present disclosure. Various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the disclosed subject matter set forth in the appended claims.

[0101] As used in this specification and claims, the terms "for example," "for example," and "such as," and the verbs "include," "have," "comprise," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, are each interpreted as open-ended, meaning that the list should not be viewed as excluding other additional components or items. The term "based on" means based, at least in part, on. In addition, it should be understood that the words and terminology employed herein are for descriptive purposes and should not be considered limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0102] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that the embodiments may be practiced without these specific details. For example, the systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other cases, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. In addition, the same reference numerals and numbers in the various drawings represent the same elements.

[0103] Although most of the descriptions are performed using speech as the target sound source, the same approach can be applied to other types of audio signals.

[0104] System Overview

[0105] Figure 1A A representation of an environment 100A for reverberation modeling of speech signals according to an embodiment of the present disclosure is shown. The environment 100A may correspond to a closed environment with speakers 102 (such as speaker 102A and speaker 102B). Figure 1A In the figure, a device 104 including at least a microphone or a microphone array is also shown. In some example embodiments, the device 104 may correspond to an automatic speech recognition (ASR) system, an audio signal processing system, or any speech processing system.

[0106] In the illustrative example scenario, when the speaker 102 outputs speech, the corresponding acoustic speech signal may travel in different paths toward the device 104. The acoustic speech signal may be linearly distorted by reflections from objects such as wall reflections, ceiling reflections, etc. Figure 1A Specifically, the acoustic speech signal of speaker 102 is distorted in the multipath directions before reaching device 104, resulting in reverberation of the acoustic speech signal.

[0107] Therefore, device 104 receives this acoustic speech signal of speaker 102A as a mixture of acoustic signals. The mixture of acoustic signals includes an anechoic speech signal and a reverberant speech signal. The anechoic speech signal is a target direct path signal 106A. The reverberant speech signal 108A, hereinafter collectively referred to as reverberation, includes a non-direct path signal or a multipath signal. In some cases, there may be multiple speakers in environment 100, such as speaker 102A and another speaker 102B. In this case, the mixture of acoustic signals includes a target direct path signal 106B and a reverberant speech signal, hereinafter collectively referred to as a reverberation 108B corresponding to speaker 102B. The mixture of acoustic signals may also include a reverberant noise signal 110A of a non-target source (such as an air conditioning device 110 in environment 100A).

[0108] In some cases, the speech signals of speaker 102A and / or speaker 102B may be blocked before reaching device 104, such as Figure 1B shown.

[0109] Figure 1B FIG. 2 shows an exemplary representation of reverberation modeling of a speech signal according to another embodiment of the present disclosure. Figure 1B As shown in environment 100B, the speech signal of speaker 102A or speaker 102B is blocked by block 114 before reaching device 104. Block 114 may cause the speech signal of the corresponding speaker (such as speaker 102A or speaker 102B) to reverberate in different directions. This reverberation may increase the attenuated copy and delayed copy (in the case of speaker 102A or speaker 102B) of the speech signal of speaker 102A or speaker 102B. Figure 1B 114). When device 104 is blocked by block 114, the speech signal of speaker 102A or speaker 102B may not have a corresponding target direct path signal. Instead, the speech signal may include a shortest path, such as shortest path 106C corresponding to the speech signal of speaker 102A and / or shortest path 106D corresponding to the speech signal of speaker 102B. For the purposes described in this application, under such conditions we consider the shortest path signal to be the target direct path signal, and signals corresponding to paths longer than the shortest path are considered to be reverberation.

[0110] Device 104 may use system 112 that may be integrated or embedded in device 104 to reduce such reverberation, such as reverberation 108A and 108B. Figure 2A and Figure 2B The system 112 is further explained.

[0111] Figure 2A FIG. 2 is a schematic block diagram of a system 200a for dereverberation of a speech signal according to an embodiment of the present disclosure. The system 200a corresponds to Figure 1Aand Figure 1B system 112.

[0112] In some example embodiments, system 200a includes an input interface 202, a memory 204 storing a first deep neural network (DNN1 (e.g., DNN1 206A), a processor 208, and an output interface 210. Input interface 202 and output interface 210 are further coupled to a wired or wireless communication channel for communication between system 200a and components external to system 200a.

[0113] The input interface 202 is configured to receive a mixture of acoustic signals including a target direct path signal (e.g., target direct path signal 106A or target direct path signal 106B) and a reverberation of the target direct path signal (e.g., reverberation 108A and / or reverberation 108B) through a wired or wireless communication channel. In some example embodiments, the input interface 202 may be configured to connect to at least a microphone of the device 104 or a microphone array of the device 104.

[0114] Processor 208 submits the mixture of the acoustic signal including the target direct path signal 106A and the reverberation 108A to DNN1 206A. DNN1 206A generates a first estimate of the target direct path signal 106A. In a multi-speaker scenario including speaker 102A and speaker 102B generating sound signals in environment 100A or environment 100B, the target direct path signal corresponding to each of speakers 102A and 102B is estimated by DNN1 206A. DNN1 206A may generate corresponding estimates of the target direct path signal for each of speakers 102A and 102B individually or simultaneously. For example, DNN1 206A simultaneously determines a first estimate of the target direct path signal 106A for speaker 102A and a first estimate of the target direct path signal 106B for speaker 102B.

[0115] The first estimate of the target direct path signal 106A is used together with the mixture of the received acoustic signals to estimate a filter that models a room impulse response (RIR) of the generated first estimate of the target direct path signal 106A. The RIR is an impulse response of a room (e.g., environment 100A or environment 100B) between a sound source (e.g., speaker 102A and speaker 102B) and a microphone in device 104. The filter that models the RIR when applied to the first estimate of the target direct path signal generates a result of at least one of the mixture of the closest acoustic signals and the residual between the mixture of the acoustic signals and the first estimate of the target direct path signal according to a distance function. The estimated filter that models the RIR is transmitted over a communication channel. To this end, the filter that models the RIR can be output via output interface 210 and then transmitted over a wired or wireless communication channel connected to system 200a.

[0116] The filter that models the RIR is a linear filter estimated using a linear convolution prediction module. This linearity of the filter is modeled in order to reduce computational complexity in applications related to dereverberation of sound signals. Additionally, linear filters provide a good approximation of the physical relationship between a sound signal and its reflections. However, in reality, the reflection of sound is a non-linear phenomenon. However, as one of ordinary skill in the art would appreciate, the choice of whether the filter is linear or non-linear is a design choice and is well within the scope of the present disclosure.

[0117] Linear filters are chosen in applications that require simplicity and accuracy. Therefore, linear convolution prediction implements a linear structure of the filter to approximate the linear reflection of the direct path signal from the surfaces in the room.

[0118] In some embodiments, a filter is applied to a first estimate of the target direct path signal in the time-frequency domain. This is functionally and computationally superior to applying the filter to the first estimate of the target direct path signal in the time domain because the time-frequency domain allows for accurate modeling of the filter. In addition, estimating the coefficients of the filter in the time-frequency domain provides a low-cost closed-form solution to the estimation problem. This may also be useful because other types of processing are performed in the time-frequency domain, so it avoids having to go back and forth between the time domain and the time-frequency domain.

[0119] When the filter of the estimate modeling the RIR is applied to the first estimate of the target direct path signal 106A, a corresponding result is obtained. According to the distance function, the result is closest to one of the mixture of the acoustic signal and the residual between the mixture of the acoustic signal and the first estimate of the target direct path signal. The distance function provides the distance between the residual between the mixture of the acoustic signal and the first estimate of the target direct path signal and the result of applying the filter to the first estimate of the target direct path signal. In some embodiments, the distance function may correspond to a weighted distance with a weight at each time-frequency point in the time-frequency domain. The weight may be determined by one or a combination of the received mixture of the acoustic signal and the estimate of the target direct path signal. In an example embodiment, the distance function may be based on a minimum square distance.

[0120] In some embodiments, the first DNN DNN1 206A is trained to estimate both the amplitude and phase of the target direct path signal. To this end, applying a filter estimate (which is also interchangeably and equivalently referred to as an estimated filter that models the RIR) to the first estimate of the target direct path signal involves calculating both the estimated amplitude and phase, and measuring the distance function between the result of applying the filter to the estimate of the direct path signal and the residual between the mixture of the acoustic signal and the direct path signal in both the amplitude domain and the phase domain. The calculation of both the amplitude and the phase allows information related to the phase of the estimated direct path signal to be used in the estimate of the filter, which results in improving the accuracy of the estimate of the filter.

[0121] In some embodiments, the first DNN DNN1 206A is pre-trained to obtain a first estimate of a target direct path signal from a mixture of observed acoustic signals. Pre-training of DNN1 206A is performed using a training dataset of a mixture of acoustic signals and a corresponding reference direct path signal in the training dataset by minimizing a loss function.

[0122] When the result of applying the filter estimate to the first estimate of the target direct path signal 106A is removed from the mixture of acoustic signals, a reduced reverberation mixture of the target direct path signal 106A is obtained so as to achieve the goal of reverberation modeling of the speech signal as the mixture of acoustic signals.

[0123] Figure 2B FIG. 2 is a schematic block diagram of a system 200b for reverberation modeling of a speech signal according to an embodiment of the present disclosure. The system 200b corresponds to Figure 1A and Figure 1B System 112. System 200b includes all components of system 200a and an additional second DNN2 206B.

[0124] To this end, the system 200b includes an input interface 202, a memory 204 storing a first DNN1 (e.g., DNN1 206A), a second DNN2 (e.g., DNN2 206B), a processor 208, and an output interface 210.

[0125] When the result of applying the filter estimate to the first estimate of the target direct path signal 106A is removed from the mixture of acoustic signals, a reverberation-reduced mixture of the target direct path signal 106A is obtained. The reverberation-reduced mixture of the target direct path signal 106A is given as an input to DNN2 206B. DNN2 206B generates a second estimate of the target direct path signal 106A. The second estimate of the target direct path signal is used together with the mixture of received acoustic signals to estimate an improved filter (or an improved filter estimate) that models the RIR of the target direct path signal with higher accuracy than the filter obtained using the first estimate. The improved filter estimate is output via output interface 210.

[0126] Similarly, for speaker 102B, a first estimate of target direct path signal 106B is used with the mixture of received acoustic signals to estimate a filter that models the RIR of target direct path signal 106B. The filter estimate is applied to the first estimate of target direct path signal 106B to obtain a corresponding result. The result is removed from the mixture of acoustic signals to obtain a reverberation-reduced mixture of target direct path signal 106B. The reverberation-reduced mixture of target direct path signal 106B is input to DNN2 206B, which generates a second estimate of target direct path signal 106B. The second estimate of target direct path signal 106B for speaker 102B is used with the mixture of received acoustic signals to estimate an improved filter that models the RIR of target direct path signal 106B for speaker 102B with greater accuracy than the filter obtained using the first estimate. The improved filter that models the RIR is output via output interface 210.

[0127] In some embodiments, the downstream DNN is used to apply one or more of the filter estimates output by the first DNN or DNN1 206A or the improved filter estimates output by the second DNN or DNN2 206B to perform different types of speech processing.

[0128] Figure 2CA downstream DNN DNN3 206C is shown for performing some tasks that are not necessarily related to speech transcription and / or separation. For example, in some embodiments, the downstream DNN DNN3 206C is used for tasks such as speaker identification, emotion recognition, event detection, localization, etc. To this end, for all of these tasks, having information about the reverberation of the target direct path signal (modeled as the RIR of the target direct path signal) is beneficial for training or inference purposes. In various embodiments, the downstream DNN utilizes the RIP of the filter determined by the first DNN or, optionally, by the second DNN.

[0129] For example, in the application, the RIR is modeled as an estimate of the filter in a location with a different setup compared to the user's current location. For example, for a movie, an RIR is determined for a church scene, and now this RIR needs to be applied to a recording obtained in a small conference room. Thus, the RIR of the different setup can be combined with the RIR of the current location / setup to obtain a novel sound effect. In this way, the reverberation of the church setting is synthesized as if it were applied in the same room setting. In this way, any number and type of sounds can be synthesized, providing different applications of the filter.

[0130] Furthermore, as previously described, a second estimate of the target direct path signal (such as the second estimate of the target direct path signal 106A or the second estimate of the target direct path signal 106B) is obtained as a dereverberated speech signal of the corresponding speaker (such as the speaker 102A or the speaker 102B). Figure 3A , Figure 3B and Figure 3C Any of the systems 200a, 200b, or 200c is described in further detail to model reverberation of a speech signal.

[0131] Figure 3A A schematic block diagram of a process 300 for reverberation modeling of a speech signal according to an embodiment of the present disclosure is shown. The process 300 is performed by the system 200. In an example embodiment, a mixture of acoustic signals 302 (Y) is received via the input interface 202 of the system 200. The mixture of acoustic signals includes a target direct path signal (such as the target direct path signal 106A of the speaker 102A) and reverberation (such as the reverberation 108A of the target direct path signal 106A, or the target direct path signal 106B of the speaker 102B and the reverberation 108B of the target direct path signal 106B) and reverberation of other sources (such as the noise signal 110A of the device 110). The received mixture of acoustic signals 302 is submitted to the DNN1 206A.

[0132] DNN1 206A determines a first estimate 304 of a target direct path signal, such as target direct path signal 106A or target direct path signal 106B. In addition, a filter estimate 306 (hereinafter interchangeably referred to as filter 306) is determined for modeling a room impulse response (RIR) 308 for the first estimate 304 of the target direct path signal 106A. The RIR model 308 (hereinafter referred to as RIR 308) may correspond to an impulse response of an environment (such as environment 100A or environment 100B) between a source (such as speaker 102A and / or speaker 102B) and a receiver (such as device 104). To this end, the absolute delay and attenuation due to propagation from the source to the microphone is not modeled, but only the relative delay and attenuation using the direct path signal as a reference is modeled. For the actual dry source signal at the source position, the impulse response is not considered, but relative to the direct path signal received within the mix as a reference. For ease of presentation, filter estimate 306 is such that applying filter estimate 306 to the direct path signal includes only the early reflections and late reverberation of the direct path signal, and does not include the direct path signal. The associated full filter estimate is equivalently obtained by modifying filter estimate 306 to further include the direct path signal. The two filter estimates are equivalent, and one filter estimate can be obtained directly from the other filter estimate.

[0133] In some example embodiments, the mixture of acoustic signals 302 may correspond to a monophonic signal recorded in a noisy reverberant environment, such as environment 100A or environment 100B. Such a monophonic signal may be formulated as a physical model in the time domain. The physical model describes the relationship between the mixture of acoustic signals 302 (y), a reverberant target speech signal (x) (which includes both a target direct path signal, such as target direct path signal 106A, and a reverberation, such as reverberation 108A), and a non-target source (v) (e.g., device 110) including a reverberant noise signal (e.g., reverberant noise signal 110A) and a reverberant competing speaker (e.g., speaker 102B).

[0134] In the time domain, for simplicity of notation, considering the target direct path signal 106A as the only target, the formulated physical model of the mixture of the acoustic signal 302 can be expressed as:

[0135] y[n]=x[n]+v[n]=(a*r)[n]+v[n]

[0136] =(a*r d )[n]+(a*r e+l )[n]+v[n]

[0137] =(a*r d )[n]+(a*r e+a*r)[n]+v[n]

[0138] =s[n]+h[n]+v[n], (1)

[0139] Where "n" indexes the discrete time scale, "*" represents the convolution operator, and "x" is the linear convolution between the dry source signal "a" (e.g., the original signal source from speaker 102A) and the RIR "r" of environment 100 (i.e., RIR model 308).

[0140] The term “r d ", "r e ” and “r l ” represent the direct portion, early portion, and late portion of RIR 308 of environment 100, respectively. The term “s” represents a target direct path signal (such as target direct path signal 106A), and the target direct path signal is defined as s=a*r d The term “h” represents the non-direct path signal (eg, reverberation 108A), and the non-direct path signal is defined as the early reflection a*r e and late reverb l The sum of, that is, h = a*r e +a*r l =a*r e+l The part of RIR 308 corresponding to both direct path and early reflections d+e can be defined as the set of pulses up to 50 ms after the direct path peak of r, and the early reflection component of RIR r e can be defined as r e =r d+e -r d . Reference d Let us consider the filter modeling RIR in this application, i.e., the time origin of the filter is implicitly considered to be r d The time of the pulse, and the scaling of the filter elements relative to r d The height of the pulse should be taken into consideration.

[0141] The above physical model of the mixture 302 of acoustic signals can be formulated in a short-time Fourier transform (STFT) for evaluating a time series, i.e., the Fourier transform of the mixture 302 of acoustic signals within a short time window. The physical model of the mixture of acoustic signals 302 in the STFT domain is formulated as:

[0142] Y(t,f)=X(t,f)+V(t,f)

[0143] =S(t,f)+H(t,f)+V(t,f), (2)

[0144] Wherein, Y(t,f), X(t,f), S(t,f), H(t,f), and V(t,f) represent the STFT coefficients of the acoustic signal 302, the reverberant target speech, the target direct path signal, the early reflection plus the late reverberation and the mixture of the non-target source (e.g., the air conditioning device 110) captured by the device 104 at time t and frequency f, respectively. The corresponding spectrogram is represented by Y, X, S, H, and V.

[0145] The target direct path signal 106A, represented as S(t, f) in equation (2), is estimated from the STFT coefficients of the mixture of the acoustic signal 302 (Y(t, f)) using the DNN. The recovered target direct path signal 106A (S(t, f)) can be used as the first estimate 304 of the target direct path signal 106A.

[0146] Direct path signal to the target (ie, the first estimate 304 of the target direct path signal) is filtered to estimate filter 306 to approximate the residual between the mixture of acoustic signal 302 and the first estimate of direct path signal 304 .

[0147] To this end, in some embodiments, the filter 306 is estimated based on the convolution prediction. The convolution prediction corresponds to a forward convolution prediction (FCP) that forward filters the first estimate 304 of the target direct path signal 106A obtained by DNN1 206A. The forward filtering can estimate the filter 306 by solving the following minimization problem:

[0148]

[0149] in, The dereverberation result of the mixed reverberation reduction as the target direct path signal is calculated as Among them, the subtracted term The reverberation is considered to be estimated by forward filtering or filter 306. Represented by DNNDNN b The first estimate of the target direct path signal 106A is obtained by using a filter (such as DNN1 206A) at each frequency. The reverberation is performed to find a delayed and attenuated copy of the target direct path signal 304. Such a copy is a repeated signal of the first estimate 304, which is considered to be a reverberation of the target direct path signal 106A.

[0150] Then, filter 306 is applied to the first estimate of the target direct path signal and the result is subtracted from the mixture of the acoustic signal 302. This results in the removal of early reflections and late reverberation of the target direct path signal 106A, while utilizing the magnitude and phase of the first estimate in the filter estimation by DNN1 206A.

[0151] By placing the first estimate 304 Absorbed The new formula can be given as

[0152]

[0153] where |·| calculates the magnitude of the first estimate 304, and is a scalar used to weight the contribution of each time-frequency bin in the loss function.

[0154] Using equation (4), the first estimate 304 is filtered to approximate the reverberant target speech X. The dereverberation result is obtained as,

[0155]

[0156] in, is an estimate of the reverberant target speech X(t, f), and is the estimated reverberation corresponding to speaker 102A. In addition,

[0157]

[0158] Equation (6) can be interpreted as adding to the first estimate The Y cannot be passed The residual component is explained by linear filtering of the target direct path signal 106A. The filter estimate for the forward filtering is obtained by solving a quadratic minimization problem, and a closed form solution can be used, which is convenient for online processing applications. Using equation (3) or equation (6), the early reflections and late reverberation of the first estimate of the target direct path signal 106A are removed by forward filtering, and the filter 306 is estimated using both amplitude and phase. The filter 306 is applied to the first estimate 304 instead of the mixture of the acoustic signal 302. Note that the filter 306 corresponds to the filter obtained using equation (3) And the full filter 307 corresponds to the filter filter and The difference is That is, they differ in whether the target direct path signal itself is included in the result of filtering. In the case of filter 306, the target direct path signal is not included, and it is only the reverberation.

[0159] Furthermore, solving equation (6) results in good filter estimates even in the presence of interference for the corresponding target speaker (such as speaker 102A or speaker 102B). Equation (6) can be reformulated based on the reverberant target speech (X) of speaker 102A. For example, a mixture of the acoustic signal (Y) 302 including the reverberant target speech and interference is defined as Y=X+V,

[0160]

[0161] in, and X can be uncorrelated with V, such that

[0162]

[0163] In equation (7), the forward filtering is based on the first estimate 304 and the reverberant target speech (X) to estimate the full filter. 304 and the reverberant target speech, a linear filter structure may appear that produces a full filter estimate for target speaker 102A. Thus, even if the mixture of acoustic signal 302 contains noise (e.g., noise signal 110A) or a competing speaker (e.g., speaker 102B), a good full filter estimate for the target speaker (e.g., speaker 102A) may be obtained (and a good filter estimate 306 may be obtained therefrom).

[0164] If an estimate of the reverberant target speech X is available, the estimate of the filter 306 can be improved by calculating the full filter estimate using equation (7). In some embodiments, the estimate of the reverberant speech of each speaker is iteratively removed from the mixture of the acoustic signal 302 to refine the reverberant target signal used in the estimate of the filter 306.

[0165] In the current implementation, equation (4) of the FCP can remove the reverberation associated with the target speaker 102A. Being able to obtain the reverberation of the target speaker 102A may be particularly useful in multi-speaker separation tasks because each target speaker is convolved with a different RIR. To this end, in some implementations, different filters can be calculated to dereverberate each speaker (in Figure 6). The estimated filter (e.g., filter 306) can focus on reducing the reverberation of the target speaker 102A, rather than the reverberation of another speaker (e.g., speaker 102B) and non-target sources (e.g., air conditioning equipment 110) in combination. In order to dereverberate the speech signal even in the presence of non-target sources, the output of DNN1 206A (such as, the first estimate 304 of the target direct path signal 106A) and the reverberation-reduced mixture obtained using filter 306 can be used to dereverberate the speech signal. To this end, the first estimate 304 and the reverberation-reduced mixture obtained using filter 306 can be input to DNN2 206B to output a second estimate 314 of the target direct path signal 106A (or the target direct path signal 106B). The output generated by DNN2 206B (such as the second estimate 314) may be better than the output of DNN1 206A because the input of DNN2 206B (i.e., the first estimate 304 and the reverberation-reduced mixture obtained using the filter 306) is more refined than the input of DNN1 206A. For example, the first estimate 304 and the reverberation-reduced mixture obtained using the filter 306 output by DNN1 206A may have less interference. When these first estimate 304 and the reverberation-reduced mixture obtained using the filter 306 with less interference are processed by DNN2 206B, the corresponding output (i.e., the second estimate 314) may be better than the output of DNN1 206A (i.e., the first estimate 304). Therefore, the second estimate generated by DNN2 206B can be used to perform another iteration of convolution prediction to obtain a second filter and a reverberation-reduced second mixture, and the reverberation-reduced second mixture can be input to DNN2 206B together with the second estimate to generate a refined output.

[0166] In some example embodiments, the corresponding RIRs for each speaker, such as RIR 308, may be estimated by solving a linear regression problem for each frequency in the time-frequency domain or the time domain. Filter 306, which models RIR 308, may be used to identify delayed and attenuated copies of the target direct path signal for speaker 102A and / or speaker 102B. Delayed and attenuated copies that are repeating patterns due to reverberation may be removed from the mixture of received acoustic signals 302. To this end, filter 306 is applied to first estimate 304 to output result 310. Based on a distance function, such as a weighted least squares distance function, result 310 may be closest to the residual between the mixture of acoustic signals 302 and first estimate 304 of the target direct path signal. When result 310 is removed from the mixture of acoustic signals 302, a reverberation-reduced mixture 312 is obtained.

[0167] The reduced reverberation mix 312 is mathematically represented as Among them, the subtracted term is considered to be the reverberation estimated by the filter 306 .

[0168] The removal of the result 310 reduces the delayed and decayed copies from the reverberation-reduced mix 312. The delayed and decayed copies may correspond to the late reverberation and early reflections of the target direct path signal. These early reflections and late reverberation may be identified from the RIR 308 modeled by the filter estimate 306. The RIR 308 with early reflections and late reverberation is Figure 3B Shown in.

[0169] Figure 3B A representation 316 of a room impulse response (RIR) model 316A of an original signal source from a speaker, such as speaker 102A, is shown, showing impulses corresponding to a target direct path signal 320A, impulses corresponding to early reflections 320B, and impulses corresponding to late reverberation 320C. In this application, we consider the target direct path signal to be a reference, rather than the original signal source from the speaker. In other words, applying the RIR to the target direct path signal results in a reverberant signal for the speaker that is the sum of the target direct path signal and the early reflections and late reverberation of the target direct path signal.

[0170] Figure 3C An application 326 of filter 316B to model RIR 316A in frequency bin f is shown in accordance with an embodiment of the present disclosure. RIR model 316A corresponds to RIR model 308, and filter estimate 316B corresponds to a full filter estimate associated with filter estimate 306.

[0171] RIR model 316A has a structure that can be represented as a pulse train in the time domain. For example, RIR model 316A is represented as a graphical illustration having an amplitude 318A and no taps representing a time delay 318B axis. The structure of RIR model 316A may include a pulse train corresponding to target direct path signal 320A (r d ) pulse, corresponding to the target direct path signal 320A (r d ) of discrete early reflections 320B(r e ) is followed by a late reverberation 320C (r ) of the target direct path signal 320A due to reverberation in an environment such as environment 100. l ) The target direct path signal 320A may correspond to the target direct path signal 106A or the target direct path signal 106B.

[0172] In some example embodiments, early reflections 320B and late reverberation 320C are identified from the RIR model 316A. Assuming that the filter is modeled using K coefficients at each frequency f, the coefficients of the filter estimate 307 at frequency f are obtained such that applying 326 the filter to the first estimate 304 of the target direct path signal can best approximate the reverberant mixture 322 at the current time t and the same frequency f by adding the kth coefficient to the time-frequency bin multiplied by the first estimate at the same frequency f and time t-k+1 (for all k=1, ..., K).

[0173] As shown in graphical diagram 316B, it represents approximating a mixture 322 (Y) of acoustic signals by applying 326 a K-tap filter 324 to the first estimate 304 of the target direct path signal. The filter 324 is estimated by optimizing the forward filtering of the first estimate 304 of the target direct path signal 302A. The filter 324 is an example of the filter 307. For example, the number of taps K of the filter 324 can be set to 40, which can be equivalent to a filter length in the time domain of ((40-1)×8+32) ms.

[0174] There may be different scenarios for reverberation modeling, including dereverberation of speech signals by DNN1 206A and DNN2 206B. For example, a mixture of acoustic signals 302 may be received from a single speaker (e.g., speaker 102A) or from multiple speakers (e.g., speakers 102A and 102B) via a single microphone or via a microphone array of device 104. In the case of multiple speakers, first DNN1 206A estimates a different first estimate of the target direct path signal for each of the multiple speakers. Reference Figure 4 , Figure 5 and Figure 6 The dereverberation of speech signals for different scenarios is further explained.

[0175] Figure 4 FIG. 4 is a schematic diagram showing an architecture representation 400 for reverberation modeling of speech signals according to an embodiment of the present disclosure. Figure 4 As shown, the architecture representation 400 includes DNN1 402, DNN2 406, and a convolutional prediction module 404 between DNN1 402 and DNN1 406. DNN1 402 corresponds to DNN1 206A, and DNN2 406 corresponds to DNN2 206B.

[0176] DNN1 402 receives a mixture of acoustic signals, such as the mixture of acoustic signals 302 represented as "Y". The mixture of acoustic signals includes the target direct path signal 106A and the reverberation 108A. In some embodiments, DNN1 402 is pre-trained to generate a first estimate 408 of the target direct path signal 106A of the speaker 102A located at the microphone of the device 104. The first estimate 408 is an example of the first estimate 304. The first estimate 408 of the target direct path signal 106A is used to estimate the reverberation of the target direct path signal 106A based on the convolution prediction. To this end, the convolution prediction module 404 estimates a linear filter (e.g., filter 306) that models the RIR of the first estimate 408 of the target direct path signal 106A based on the convolution prediction between the mixture of the acoustic signal and the first estimate 408. The convolution prediction utilizes the linear filter structure in the reverberation of the first estimate 408.

[0177] The filter that models the RIR is applied to the first estimate 408 to obtain a result (e.g., result 310) that estimates a delayed copy and an attenuated copy (e.g., early reflections 320B and late reverberation 320C) of the first estimate 408 from the mixture of the acoustic signal. The delayed copy and the attenuated copy correspond to the reverberation of the target direct path signal 106A in the mixture of the acoustic signal 302. In some example embodiments, the reverberation of the target direct path signal 106A is estimated by minimizing the distance between the result and the residual between the mixture of the acoustic signal 302 and the first estimate 408. The estimated reverberation of the target direct path signal 106A is removed from the mixture of the acoustic signal 302 to obtain another mixture, i.e., a reverberated reduced mixture 412 of the target direct path signal 106A.

[0178] The reverberation-reduced mixture 412 is input to DNN2 406. DNN2 406 uses the reverberation-reduced mixture 412 to generate a second estimate 410 of the target direct path signal 106A. Second estimate 410 is an example of second estimate 314 .

[0179] Some embodiments are based on the recognition that the second estimate 410 is better than the first estimate 408 because the DNN2 406 processes a refined mixture of the acoustic signal, which is a reverberant-reduced mixture 412. The second estimate 410 can be further improved to perform better than the first estimate 408. To this end, the DNN2 406 can be input with one or a combination of the mixture of the acoustic signal 302 and the first estimate 408 to generate the second estimate 410. In some cases, the mixture of the acoustic signal 302, the first estimate 408, and the reverberant-reduced mixture 412 can be input to the DNN2 406 to generate the second estimate 410. In some other cases, the first estimate 408 and the reverberant-reduced mixture 412 can be input to the DNN2 406 to generate the second estimate 410. In addition, the estimation of the filter, the acquisition of the reverberation-reduced mixture 412, and the submission of the reverberation-reduced mixture 412 can be iterated to gradually refine the second estimate 410 of the target direct path signal 106A and improve the dereverberation of the speech signal of the speaker 102A. The iteration can terminate when a termination condition is met. The termination condition can correspond to a user-defined condition. Therefore, the second estimate 410 can be better than the first estimate 408 because the second estimate 410 is refined with the reverberation-reduced mixture 412. In some embodiments, the DNN2406 can be trained using the mixture of the acoustic signal 302, the reverberation-reduced mixture 412, and the first estimate 408 to output the second estimate 410 that improves the dereverberation of the speech signal.

[0180] In some cases of multiple speakers, the mixture of received acoustic signal 302 may include speech signals from multiple speakers (such as speaker 102A and speaker 102B). In this case, DNN1 402 may generate multiple outputs (such as different first estimates of the target direct path signal), from which different filters that model the corresponding RIRs of the multiple speakers may be obtained, which will be referred to as Figure 5 Further description.

[0181] Figure 5 is a schematic diagram depicting an architectural representation 500 for modeling reverberation of speech signals of multiple speakers (eg, speakers 102A and 102B) according to some embodiments of the present disclosure. Figure 5 As shown, the architecture representation 500 corresponds to a multi-speaker scenario and includes DNN1 502, DNN2 506, and multiple instances of convolutional prediction modules, such as convolutional prediction module 504A and convolutional prediction module 504B between DNN1 502 and DNN2 506. DNN1 502 corresponds to DNN1 206A, and DNN2 506 corresponds to DNN2 206B.

[0182] In a multiple speaker scenario, the mixture 302 of received acoustic signals includes speech signals from multiple speakers, such as speaker 102A and speaker 102B. In this case, DNN1 502 simultaneously estimates first estimates of corresponding target direct path signals, such as target direct path signal 106A and target direct path signal 106B for speakers 102A and 102B. For example, DNN1 502 estimates first estimate 508A (denoted as ) of target direct path signal 106A for speaker 102A. ) and a first estimate 508B of the target direct path signal 106B for speaker 102B (denoted as ). Figure 5 The case of C speakers is described, but for simplicity and as an example, we describe here the case C=2.

[0183] Convolution prediction module 504A uses first estimate 508A to estimate filters (e.g., filter 306) that model the RIR (e.g., RIR 308) of speaker 102A. Convolution prediction module 504B uses first estimate 508B to estimate filters for speaker 102B. Each filter corresponding to speaker 102A and speaker 102B is applied to first estimate 508A and first estimate 508B to obtain a corresponding reverberation-reduced mixture 510A (denoted as ) and the reverberant-reduced mix 510B of speaker 102B (denoted as ).

[0184] The reduced reverberation mixture 510A and the reduced reverberation mixture 510B are concatenated and provided as input to DNN2 506 to produce corresponding second estimates 512A and second estimates 512B for speakers 102A and 102B. In some example embodiments, DNN2 506 may be input with the first estimate 508A and the reduced reverberation mixture 510A and the first estimate 508B and the reduced reverberation mixture 510B and the mixture 302 of the acoustic signal to output second estimates 512A and 512B.

[0185] In some example embodiments, the estimates of the filters for the first estimates 508A and 508B and the reduced reverberation mixture 510A and the reduced reverberation mixture 510B may be iterated to produce corresponding filters and reduced reverberation mixtures for each of the plurality of speakers 102A and 102B by replacing the first estimates 508A and 508B with the second estimates 512A and 512B. The iterations terminate when a user-defined termination condition is met. The termination condition may include a user-defined termination condition, such as, for example, terminating after 3 iterations.

[0186] In some example implementations, the reverberant-reduced mixture 510A and the reverberant-reduced mixture 510B may be combined into a tensor. A tensor is a dimensional data structure representing all reverberant-reduced mixtures of the plurality of speakers 102A and 102B. The tensor is submitted to DNN2 506 to output a corresponding second estimate 512A and a second estimate 512B for each of the plurality of speakers 102A and 102B.

[0187] In some cases, a corresponding second estimate of each of the plurality of speakers 102A and 102B may be estimated one by one, which is then Figure 6 Described in.

[0188] Figure 6 6 is a schematic diagram depicting an architectural representation 600 for reverberation modeling of speech signals of multiple speakers 102A and 102B according to some other embodiments of the present disclosure. Figure 6 As shown, the architecture representation 600 corresponds to a multi-speaker scenario and includes DNN1 602, multiple instances of a second DNN (such as DNN2 606A and DNN2 606B), and multiple instances of convolutional prediction modules (such as convolutional prediction module 604A and convolutional prediction module 604B between DNN1 602 and the multiple instances of the second DNN (DNN2 606A and DNN2 606B). DNN1 602 corresponds to DNN1 206A, and each of DNN2 606A and DNN2 606B corresponds to DNN2 206B.

[0189] DNN1 602 receives the mixture of acoustic signals 302 and estimates a corresponding target direct path signal for each of the plurality of speakers 102A and 102B. For example, DNN1 602 estimates a first estimate 608A of target direct path signal 106A for speaker 102A. DNN1 602 estimates a first estimate 608B of target direct path signal 106B for speaker 102B. First estimate 608A is input to convolutional prediction module 604A, and first estimate 608B is input to convolutional prediction module 604B.

[0190] The convolution prediction module 604A estimates a filter that models the RIR of the first estimate 608A. The filter that models the RIR is applied to the first estimate 608A to obtain a reduced reverberation mixture 610A of the target direct path signal 106A. In a similar manner, the convolution prediction module 604B estimates a filter that models the RIR of the first estimate 608B. The estimated filter that models the RIR output by the convolution prediction module 604B is applied to the first estimate 608B to obtain a reduced reverberation mixture 610B of the target direct path signal 106B.

[0191] In some embodiments, each of the reverberation-reduced mixture 610A and the reverberation-reduced mixture 610B may be obtained by removing from the mixture of the received acoustic signals the sum of the results of applying each of the filters to each of the first estimates of the target direct path signal for each of the plurality of speakers 102A and 102B instead of only the results of applying the filters to the first estimates of the target direct path signal for the corresponding speaker. For all speakers c, the reverberation-reduced mixture may thus be obtained as The same for all c.

[0192] In addition, each of the reverberation-reduced mixes is submitted to an instance of DNN2. Each of the reverberation-reduced mix 610A and the reverberation-reduced mix 610B is submitted to corresponding instances DNN2 606A and DNN2 606B, respectively (which are essentially the same DNN2, but applied to different inputs). DNN2 606A generates a second estimate 612A of the target direct path signal 106A for speaker 102B. DNN2 606B generates a second estimate 612B of the target direct path signal 106B for speaker 102B. Multiple instances of a second DNN, such as DNN2 606A and DNN2 606B that generate each of the second estimates 612A and 612B for corresponding speakers 102A and 102B, can be used to obtain clean speech of a single speaker from multiple speakers.

[0193] To improve second estimate 612A and second estimate 612B, DNN2 606A and DNN2 606B may be input with one or a combination of a mixture of acoustic signals, first estimate 608A and first estimate 608B, and the reverberation-reduced mixture 610A and 610B.

[0194] In some example implementations, the first estimate 608A may be replaced with the second estimate 612A to generate an updated first estimate 608A of the target direct path signal 106A. Likewise, the first estimate 608B may be replaced with the second estimate 612B to generate an updated first estimate 608B of the target direct path signal 106B. Furthermore, the estimation of the filter, the estimation of the reverberant-reduced mixtures 610A and 610B, and the submission of the reverberant-reduced mixtures 610A and 610B by DNN1 602 may be iterated to output an updated second estimate of the target direct path signal for each of the plurality of speakers 102A and 102B.

[0195] In some other example embodiments, a portion of a mixture of acoustic signals corresponding to a speaker (e.g., speaker 102A) may be extracted. The portion is extracted by removing the reverberant speech of another speaker (e.g., speaker 102B) from the mixture of acoustic signals. An estimate of the reverberant speech of another speaker of the plurality of speakers is obtained by adding a first estimate of a target direct path signal of the other speaker to a result of applying a corresponding filter of the other speaker to the first estimate of the target direct path signal of the other speaker. After extracting the portion of the acoustic signal corresponding to speaker 102A, a filter for the first estimate of the extracted portion is estimated. The filter is used to estimate a reverberation-reduced mixture for speaker 102A based on the portion. This processing of the portion may improve the quality of the filter used for the speaker's estimate and the quality of the corresponding second estimate.

[0196] In some example embodiments, a mixture of acoustic signals of a single speaker 102A and / or multiple speakers 102A and 102B may be received from a single microphone or microphone array. To this end, DNNs such as DNN1 602 and DNN2 606A and DNN2 606B may be trained based on spectral mapping corresponding to the single microphone and microphone array. The spectral mapping trains DNN1 602 to predict an estimated real component and imaginary (RI) component (i.e., frequency), for example, a first estimate 608A of a target direct path signal 106A from the RI component of the mixture of acoustic signals 704. The RI component of the mixture of acoustic signals 704 and the RI component of the first estimate 608A may be input to DNN2 606A to predict a second estimate of the target direct path signal 106A. DNN1 602 may be pre-trained using a training data set of a mixture of acoustic signals and a corresponding reference target direct path signal in the training data set.

[0197] In some embodiments, pre-training of DNN1 602 may be performed by minimizing a loss function. The loss function may include one or a combination of distance functions defined based on the RI component of the target direct path signal 106A in the first time-frequency domain and the RI component of the reference target direct path signal in the first time-frequency domain. The reference target direct path signal may be obtained from a training data set of an utterance, and a corresponding reverberant mixture may be obtained by convolving the reference target direct path signal with a recorded RIR or a synthesized RIR and summing with other interference signals. The distance function may be defined based on an amplitude obtained from the RI component of the estimated target direct path signal in the first time-frequency domain and a corresponding amplitude of the reference target direct path signal.

[0198] In an alternative embodiment, the distance function may be defined based on a reconstructed waveform obtained from the RI component of the estimated target direct path signal in the first time-frequency domain by reconstruction in the time domain and the waveform of the reference target direct path signal. The distance function may also be defined based on the RI component in the complex time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and the RI component of the reference target direct path signal in the second time-frequency domain. The distance function may also be defined based on an amplitude obtained from the RI component in the second time-frequency domain obtained by transforming the reconstructed waveform in the second time-frequency domain and a corresponding amplitude of the reference target direct path signal in the second time-frequency domain.

[0199] The loss function for predicting the RI component can be defined as:

[0200]

[0201] in, and is the predicted RI component produced by using the linear activation in the output layer, i.e., the output of DNN1 602 (e.g., the first estimate 608A). b∈{1,2} indicates which DNN produced the estimate (e.g., the first estimate 608A and the second estimate 612A) because two DNNs are used, such as DNN1 602 and DNN2 606A. The real (·) and imaginary (·) parts extract the RI components, and ||.||1 calculates the L1 norm, which is the sum of the magnitudes of the RI components. The loss is added to the calculated magnitudes, which gives:

[0202]

[0203] The improvement result is Wherein, j represents an imaginary unit. The trained DNN is supervised to learn a nonlinear inverse filter based on a large receptive field. To this end, the inverse STFT is applied to equation (9) to obtain the estimated time domain signal.

[0204] For speaker separation, utterance-wise permutation-invariant training (PIT) can be used to calculate the loss based on the predicted RI component. The speaker index of multiple speakers (such as speakers 102A and 102B c∈{1,...,C}) is used to distinguish C speakers S(1),...,S(C). The loss function is defined as:

[0205]

[0206] where P is a set of permutations on {1, ..., C}. The separation result is obtained as Add loss to the sum of target speech estimates It can improve the performance of speaker separation task in noisy reverberation conditions. The loss can be defined as:

[0207]

[0208] In some embodiments, the or To train DNN1 602.

[0209] In some example embodiments, the mixture of acoustic signals 302 may correspond to a multi-channel signal that may be received from a microphone array. Beamforming is performed on such a multi-channel signal, which will be referred to as Figure 7 Further description.

[0210] Figure 7 is a schematic diagram depicting an architectural representation 700 for improving dereverberation of a speech signal according to some embodiments of the present disclosure. The architectural representation 700 is similar to Figure 5 506) outputted by a second DNN, such as DNN2 506, wherein the second estimate may have been computed using a convolutional prediction module comprising only two DNNs. Figure 5 The architecture represents or includes MVDR beamforming Figure 7The beamforming output for the speaker may be obtained by applying a beamforming filter to the reverberation-reduced mixture 510A or the mixture 502. The MVDR beamforming module may be used between two DNNs, such as DNN1 502 and DNN2 506. The output of the MVDR beamforming module, such as beamforming output 514A (and / or beamforming output 514B), may be used as an input to a second DNN, such as DNN2 506. In some example embodiments, the output of the MVDR beamforming module, such as beamforming output 514A, may be combined with one or a combination of the first estimate, such as first estimate 508A, the reverberation-reduced mixture, such as reverberation-reduced mixture 510A, and the mixture 502. In some example embodiments, the beamforming output for all speakers is combined with the reverberation-reduced mixture for all speakers, the first estimate for all speakers, and the mixture, and used as an input to DNN2 506. In some example embodiments, the MVDR beamforming module may output beamforming using MVDR techniques such that signals from multiple channels may be combined to result in a better estimate of the target direct path signal.

[0211] To this end, MVDR beamforming can be applied to the reverberation-reduced mix to further improve the dereverberation and separation tasks.

[0212] In the frequency interval f omitted here, the MVDR beamformer for obtaining beamforming of the multi-channel signal at the reference microphone q can be calculated as:

[0213]

[0214] in,

[0215] is the target covariance matrix of the estimated target direct path signal (eg, first estimate 608A), and is calculated as

[0216] is the non-target covariance matrix, calculated as:

[0217]

[0218] is the estimate of the non-target component, represents the reduced-reverberation mixture for speaker c,

[0219] and(·) * The complex conjugate is calculated, and the beamforming result is calculated as

[0220]

[0221] That is, all components are considered in their unaural (or reduced reverberant) versions.

[0222] Alternatively, you can use To calculate All reverberation is considered as part of the non-target component and the beamforming result is obtained by applying the calculated beamforming filter to the mixture Y:

[0223]

[0224] Furthermore, DNNs (e.g., DNN1 602 and DNN2 606A) can be easily replaced by amplitude or time domain models as well as more advanced DNN architectures. Fig. 8A , Figure 8B , Figure 8C and Fig.8D One such model is further explained.

[0225] Fig. 8A , Figure 8B , Figure 8C and Fig.8D A schematic diagram depicting a network architecture 800 for dereverberation of a speech signal according to some other embodiments of the present disclosure is shown. The network architecture 800 corresponds to a DNN, such as DNN1 206A and DNN2 206B.

[0226] The network architecture 800 is a temporal convolutional network (TCN) 806. The TCN 806 includes four layers, each layer having six dilated convolution blocks, such as dilated convolution block 802A, dilated convolution block 802B, dilated convolution block 802C, dilated convolution block 802D, dilated convolution block 802E, and dilated convolution block 802F (hereinafter referred to as dilated convolution blocks 802A-802F). In each dilated convolution block 802A-802F, a one-dimensional (1D) depthwise separable convolution 804 is used to reduce the number of parameters. For example, each of the dilated convolution blocks 802A-802F may include approximately 6.9 million parameters for dereverberation of a speech signal. These large number of parameters may be reduced by the 1D depthwise separable convolution 804.

[0227] In addition, TCN 806 is sandwiched between U-Net including encoder 808 and decoder 810. In each of encoder 808 and decoder 810, dense network blocks are inserted at multiple frequency scales. Dense network blocks are architectures that use shorter connections between layers of DNN to train DNNs (such as DNN1 602 and DNN2 606A). For example, dense network blocks 808A, dense network blocks 808B, dense network blocks 808C, dense network blocks 808D, and dense network blocks 808E (hereinafter referred to as dense network blocks 808A-808E) at multiple frequency scales are included in encoder 808. Similarly, the decoder 810 of U-Net includes dense network blocks 810A, dense network blocks 810B, dense network blocks 810C, dense network blocks 810D, and dense network blocks 810E (hereinafter referred to as dense network blocks 810A-810E) at multiple frequency scales. The U-Net can maintain local fine-grained structures via skip connections and model contextual information along frequency by downsampling and upsampling. The TCN 806 utilizes mixed long-range information of the received acoustic signal by using dilated convolution along the time domain. The dense network blocks 808A-808E enable feature reuse in the speaker separation task and improve the discriminability of the speech signals of multiple speakers 102A and 102B.

[0228] The encoder 808 includes a two-dimensional (2D) convolution 812 and seven convolution blocks, such as convolution block 814A, convolution block 814B, convolution block 814C, convolution block 814D, convolution block 814E, convolution block 814F, and convolution block 814G (hereinafter referred to as convolution blocks 814A-814G). Each of the convolution blocks 814A-814G includes a 2D convolution, an exponential linear unit (ELU) nonlinearity, and instance normalization (IN) for downsampling, i.e., reducing the sampling rate or sample size (bits per sample) of the input signal (e.g., the mixture of the acoustic signal 704). The 2D convolution forms the basic component of feature extraction corresponding to the estimate of the target direct path signal. ELU is the activation function of the DNN (e.g., DNN1602 and DNN2 606A), and IN is a normalization layer used to stabilize the hidden state dynamics in DNN1602 and DNN2 606A.

[0229] The decoder 810 includes seven 2D deconvolution blocks (such as deconvolution 816A, deconvolution 816B, deconvolution 816C, deconvolution 816D, deconvolution 816E, deconvolution 816F and deconvolution 816G with ELU (hereinafter referred to as deconvolution 816A-816G)) and IN and one 2D deconvolution 820 for upsampling by adding zero-value samples between original samples to increase the sampling rate.

[0230] As previously described, the reverberant-reduced mixtures of multiple speakers 102A and 102B, such as the reverberant-reduced mixture 510A and the reverberant-reduced mixture 510B, are represented as tensors. The format of the tensor is featureMapstimeStepsfrequencyChannels. Each of the convolution blocks 814A-814G (i.e., Conv2D+ELU+IN) and deconvolution blocks 816A-816G (i.e., Deconv2D+ELU+IN) is specified in the following format: kernelSizeTimekernelSizeFreq, (stridesTime, stridesFreq), (paddingsTime, paddingsFreq, and featureMaps.

[0231] Each of the dense network blocks 808A-808E, such as dense blocks (g1, g2), contains five Conv2D+ELU+IN blocks with a growth rate g1 for the first four layers of the dense network blocks 808A-808E and a growth rate g2 for the last layer. The format of the tensor shape after each TCN block 806 is: featureMapstimeStep. Each IN+ELU+Conv1D box is specified in the following format: kernelSizeTime, stridesTime, paddingsTime, dilationTime, featureMaps

[0232] Fig.9A A flow chart of a method 900a for estimating a RIR for reverberation modeling of a speech signal according to an embodiment of the present disclosure is shown. The method 900a is performed by the system 200. At operation 902, the method 900a includes receiving a mixture of acoustic signals (e.g., the mixture of acoustic signals 302) including a target direct path signal (e.g., the direct path signal 106A) and a reverberation of the target direct path signal via a communication channel. The mixture of acoustic signals may include at least one of a single channel signal or a multi-channel signal received from a single microphone or a microphone array connected to an input interface connected to the communication channel.

[0233] At operation 904, the received mixture of acoustic signals is submitted to a first DNN (such as DNN1 206) to generate a first estimate (e.g., first estimate 408) of the target direct path signal 106A. In a multi-speaker scenario, the first DNN determines a corresponding first estimate for each of the plurality of speakers. The corresponding first estimate may be determined individually for each of the plurality of speakers or simultaneously for the plurality of speakers. In some embodiments, the first DNN may be pre-trained to generate the first estimate from the observed mixture of acoustic signals based on a training data set of the mixture of acoustic signals and a corresponding reference target direct path signal in the training data set. Pre-training of the first DNN may be performed by minimizing a loss function.

[0234] At operation 906, a filter (e.g., filter 306) that models a room impulse response (RIR) (e.g., RIR model 308) is estimated for the first estimate 408 of the target direct path signal 106A such that a result of applying the filter to the first estimate of the target direct path signal is closest to a residual between a mixture of the acoustic signal and the first estimate of the target direct path signal according to a distance function (e.g., a least square distance function). In some embodiments, the filter corresponds to a linear filter structure based on a convolutional prediction estimate. The linear filter using convolutional prediction (in Figure 3A , Figure 3B , Figure 4 , Figure 5 and Figure 6 ) forward filters the first estimate at each frequency in the time-frequency domain. In some example embodiments, the mixture of the received acoustic signals includes speech signals from multiple speakers. The first DNN generates multiple outputs, and each output includes a first estimate of a target direct path signal from a speaker of the multiple speakers. In some embodiments, early reflections (e.g., early reflections 320B) and late reverberations (e.g., late reverberations 320C) of the first estimate can be identified based on the RIRs modeled by the filter. The identified early reflections and late reverberations can be removed from the first estimate to estimate a mixture with reduced reverberation.

[0235] At operation 908, the estimated filter modeling the RIR is transmitted through a communication channel. The communication channel is a combination of any wired or wireless communication channels, as required by the application of the filter after transmission. To this end, method 900a can be enhanced with additional processing steps, such as the following Fig. 9B As described in.

[0236] Fig. 9B A flow chart of another method 900 b for dereverberation of a speech signal according to an embodiment of the present disclosure is shown. The method 900 b is executed by the system 200 .

[0237] At operation 910, the method 900b includes receiving, via an input interface, a mixture of acoustic signals (e.g., the mixture of acoustic signals 302) including a target direct path signal (e.g., the target direct path signal 106A) and a reverberation of the target direct path signal. The mixture of acoustic signals may include at least one of a single channel signal or a multi-channel signal that may be received from a single microphone or a microphone array connected to the input interface.

[0238] At operation 912, the mixture of received acoustic signals is submitted to a first DNN (such as DNN1 206) to generate a first estimate (e.g., first estimate 408) of the target direct path signal 106A. In a multi-speaker scenario, the first DNN determines a corresponding first estimate for each of the plurality of speakers. The corresponding first estimates may be determined individually for each of the plurality of speakers or simultaneously for the plurality of speakers. In some embodiments, the first DNN may be pre-trained to generate the first estimate based on at least one of an observed mixture of acoustic signals or a training data set of the mixture of acoustic signals and a corresponding reference target direct path signal in the training data set. Pre-training of the first DNN may be performed by minimizing a loss function.

[0239] In some embodiments, a first estimate of the target direct path signal is updated based on application of a filter that models the RIR. A mixture of the updated first estimate and the acoustic signal is then used to update training data of a training data set. The updated first estimate of the target direct path signal is used as a pseudo-label to identify the target direct path signal in the updated training data set. Furthermore, the updated training data set is then used to retrain the first DNN.

[0240] At operation 914, a filter (e.g., filter 306) that models a room impulse response (RIR) (e.g., RIR model 308) is estimated for the first estimate 408 of the target direct path signal 106A such that the result of applying the filter that models the RIR to the first estimate of the target direct path signal is closest to the residual between the mixture of the acoustic signal and the first estimate of the target direct path signal according to a distance function (e.g., a least square distance function). In some embodiments, the filter corresponds to a linear filter structure based on a convolutional prediction estimate. Using (in Figure 3A , Figure 3B , Figure 4 , Figure 5 and Figure 6The first estimate is forward filtered at each frequency in the time-frequency domain using a linear filter that is convolutionally predicted by a (described in ) convolutional method. In some example embodiments, the mixture of the received acoustic signals includes speech signals from multiple speakers. The first DNN generates multiple outputs, and each output includes a first estimate of a target direct path signal from a speaker of the multiple speakers. In some embodiments, early reflections (e.g., early reflections 320B) and late reverberations (e.g., late reverberations 320C) of the first estimate can be identified based on the RIRs modeled by the filter. The identified early reflections and late reverberations can be removed from the first estimate to estimate the mixture of the acoustic signals.

[0241] At operation 916, a reverberation-reduced mixture of the target direct path signal 106A is obtained by removing the result of applying the filter to the first estimate 408 of the target direct path signal 106A from the received mixture. In some embodiments, the second DNN can be trained based on a training data set created from enhanced data obtained by creating a reverberant mixture using a set of estimated filters and a set of estimated target direct path signals. For example, the enhanced data includes one or more of the following: external data measured outside the room or another mixture of acoustic signals in the training data set. The enhanced data is then dereverberated based on the above-described techniques. In addition, the result of applying the filter and dereverberation includes enhanced data, which is then used to further train the second DNN.

[0242] At operation 918, the reverberation-reduced mixture is submitted to a second DNN (e.g., DNN2 206B) to produce a second estimate of the target direct path signal. In some example embodiments, one or a combination of the mixture of the received acoustic signal and the first estimate of the target direct path signal is submitted to the second DNN to produce the second estimate of the target direct path signal. In some other example embodiments, the mixture of the received acoustic signal, the first estimate of the target direct path signal, and the reverberation-reduced mixture are submitted to the second DNN to produce the second estimate of the target direct path signal. In still other example embodiments, the first estimate of the target direct path signal and the reverberation-reduced mixture are submitted to the second DNN to produce the second estimate of the target direct path signal. In some embodiments, the second DNN can be trained based on a training data set created from enhanced data obtained by creating a reverberant mixture using a set of estimated filters and a set of estimated target direct path signals.

[0243] At operation 920, a second estimate of the target direct path signal is output via an output interface, such as output interface 210. The steps of estimating the filter, obtaining a reverberation-reduced mix, and submitting the reverberation-reduced mix may be repeated for each of the plurality of outputs of the first DNN to further improve dereverberation of the speech signal. The output interface may also be configured to output the RIR modeled by the filter. The output RIR may perform audio analysis for one or a combination of room acoustic parameter analysis, room geometry reconstruction, speech enhancement, and dereverberation of the speech signal.

[0244] In some example embodiments, dereverberation of a speech signal using estimates of a target direct path signal (i.e., a first estimate and a second estimate) and a filter of the target direct path signal is evaluated for three tasks: 1) speech dereverberation with weak stationary noise, 2) two speaker separation under reverberation conditions with white noise, and 3) two speaker separation under reverberation conditions with challenging non-stationary noise. The evaluation results are shown in FIG. Fig.10 , Fig.11 and Fig.12 shown.

[0245] Fig.10 A tabular representation 1000 corresponding to a simulated test set for dereverberation of speech signals according to an embodiment of the present disclosure is shown. The tabular representation 1000 depicts the datasets for dereverberation, reverberant speaker separation, and speech enhancement tasks, hyperparameter settings, and a baseline system for dereverberation of speech signals. The tabular representation 1000 also depicts the results of the ASR task on the REVERB corpus.

[0246] For de-reverberation of speech signals, a simulated reverberation dataset with weak conditioning noise can be used to train DNNs, such as DNN1 206A and DNN2 206B. In addition to evaluating the trained DNNs on the simulated test set, the DNNs are directly applied to the Reverberant Voice Enhancement and Recognition Benchmark (REVERB) corpus to show their effectiveness in processing real recorded noisy reverberant utterances. The REVERB corpus is a benchmark for evaluating automatic speech recognition technology. The data set also includes clean signals obtained from the WSJCAM0 corpus for simulation. The WSJCAM0 corpus contains 7861, 742, and 1088 utterances in its training, validation, and test sets, respectively. These utterances in the WSJCAM0 corpus are used to simulate 39305 (7861×5), 2968 (742×4), and 3264 (1088×3) noisy reverberation mixtures as training, validation, and test sets, respectively. This is followed by a data spatialization process where, for each utterance, the room is randomly sampled with random room characteristics as well as speaker and microphone positions using the RIR estimated for the dereverberation of the speech signal. The speaker-to-microphone distance is sampled in the range [0.75, 2.5] m. The reverberation time (T60) is taken from the range [0.2, 1.3] s. For each utterance, diffuse conditioning noise is sampled from the REVERB corpus and added to the speaker's reverberant speech. The signal-to-noise ratio between anechoic speech and noise is sampled in the range [5, 25] dB. The sampling rate is 16 kHz.

[0247] The trained model is applied to real reverberation recordings without retraining on the ASR task of REVERB. The test mix is ​​obtained from a real recording recorded in a room (e.g., environment 100) where the reverberation time T60 is around 0.7s and the speaker-microphone distance is around 1m in the near-field case and 2.5m in the far-field case. The recorded noise is diffuse air-conditioning noise and is weak.

[0248] The official REVERB corpus is used in software such as Kaldi to build a backend for ASR trained with noisy reverberant speech and clean source signals using REVERB. In an example implementation, a plug-and-play approach is followed for ASR, where the enhanced time domain signal is directly input to the backend for decoding.

[0249] For the reverberant speaker separation task, a six-channel spatialized multi-speaker Wall Street Journal (Spatialized Multi-Speaker Wall Street Journal, SMS-WSJ) data set is used. The SMS-WSJ data set includes simulated two-speaker mixtures under reverberant conditions. Clean speech is sampled from WSJ0 and WSJ1 data sets. The corpus contains 33561, 982 and 1332 two-speaker mixtures for training, verification and testing, respectively. The speaker-to-array distance is sampled in the range [1.0, 2.0] m, and T60 is extracted in the range [0.2, 0.5] s. Weak white noise is added to simulate microphone noise. The energy level between the sum of the reverberant target speech signal and the noise is sampled in the range [20, 30] dB. The sampling rate is 8kHz. The first channel of the six-channel SMS-WSJ data set is used for training and evaluation. In addition, direct sound is used as the training target, and both the dereverberation task and the separation task are performed.

[0250] For ASR, the default Kaldi-based backend acoustic model provided in the SMS-WSJ dataset was used and trained using single-speaker noisy reverberant speech as input and the state alignment of its corresponding direct path signal as labels. Signals at the first, third, and fifth channels (i.e., more than one microphone) were used to train the acoustic model. The task standard tri-gram language model was used for decoding.

[0251] The noisy reverberation WSJ0 hipster ambient mixture (WHAMR!) dataset is used to evaluate the noisy reverberation speaker separation task. WHAMR! pairs the two-speaker mixture in the WSJ0-2 mixture dataset with a noisy background scene for noisy reverberation binaural two-speaker separation. In this evaluation, the clean two-speaker mixture is reused in the WSJ0-2 mixture dataset, which reverberates each clean signal and adds the non-steady-state ambient noise documented in WHAM!. The reverberation time T60 is randomly sampled in the range [0.2, 1.0] s. The signal-to-noise ratio between the louder speaker and the noise is extracted in the range [-6, 3] dB. The energy level between the two speakers in each mixture is sampled in the range [-5, 5] dB. The speaker-to-array distance is sampled in the range [0.66, 2.0] m. There are 20,000, 5,000, and 3,000 binaural mixtures in the training, validation, and test sets, respectively. The corpora used are the minimal and 8kHz versions.

[0252] For STFT, the window length is 32ms, the hop size is 8ms, and the analysis window is the square root of the Hann window. For 16kHz sampling rate, a 512-point FFT is applied to extract 257-dimensional STFT features, while for 8kHz sampling rate, a 256-point FFT is used to extract 129-dimensional features. No sentence or global level average variance normalization is performed on the input features. For each mixture, we normalize its sample variance to unity before any processing. During training, the target signal needs to be scaled by the same factor used to scale the mixture.

[0253] For WPE and DNN-WPE, the number of filter taps K is set to 37 and the filter delay Δ is set to 3. The number of iterations in WPE is set to 3. No PSD context is used. Based on the validation set, K and Δ are tuned to 40 and 0, 39 and 1, 38 and 2, 37 and 3, and 36 and 4, where setting the filter taps and filter delay to 37 and 3 works best on the dataset. For convolutional prediction, K is set to 40, resulting in the same amount of context as in WPE. This is equivalent to a 344 (= (40-1) × 8 + 32) ms filter length in the time domain. The filter taps K are increased up to 125, which corresponds to up to 1.0 s RIR length. This results in an increase in the amount of computation spent in the linear regression step without a significant difference in the evaluation scores. The energy of the RIR is mainly in the 0.35 s range after the peak pulse. The lower limit value ε for calculating the dereverberation result is set to 1.0 (indicating that no weights are used) or 0.001. The PSD at each TF unit may be -30dB lower than the TF unit with the highest energy.

[0254] For all tasks, the main evaluation metric is the scale-invariant signal-to-distortion ratio (SI-SDR). SI-SDR measures the quality of sample-level predictions in the time domain. Extended short-time objective intelligibility (eSTOI) and perceptual evaluation of speech quality (PESQ) scores are measured. For PESQ, narrowband MOS-LQO scores are reported using the ITU P.862.1 standard based on the python-pesq toolkit. A reference is calculated using the metric based on a target direct path signal obtained by setting the reverberation time RT60 parameter to zero in the RIR. The word error rate (WER) of the ASR is also shown in the tabular representation 1000.

[0255] In tabular representation 1000 , the target direct path signal is denoted “d,” the target direct path signal with early reflections is denoted “d+e,” and the target direct path signal with early reflections and noise is denoted “d+e+v.”

[0256] As shown in tabular representation 1000, if the first estimate of the first DNN (DNN1) is considered as the final prediction, the training objective of DNN1 shows better performance than the other two (i.e., "d+e" and "d+e+v"). Comparing the different training objectives using DNN1, there is no significant difference in DNN1-WPE, which applies the DNN1 output to improve WPE. However, it can be seen that using the target direct path signal to train DNN1 shows improved performance in DNN1+DNN2, which stacks two DNNs by using a mixture of the acoustic signal and the output of DNN1, i.e., the first estimate of the target direct path signal is used to train the second DNN2.

[0257] The table representation 1000 also includes a comparison of methods using inverse convolutive prediction (ICP), forward convolutive prediction (FCP), or weighted prediction error (WPE) between two DNNs (i.e., DNN1 and DNN2), and DNN1+FCP+DNN2 (where the lower limit value ε is set to 0.001) shows better performance than DNN1+WPE+DNN2 and DNN1+ICP+DNN2. As shown in the table representation 1000, by performing one or more iterations of linear or convolutional prediction and DNN2 at runtime, DNN1+(WPE+DNN2)×2 and DNN1+(ICP+DNN2)×2 show slight improvements in SI-SDR and PESQ and slight decreases in word error rate (WER), while DNN1+(CCP+DNN2)×2 shows improvements in all metrics. These results indicate that the DNN1+FCP+DNN2 approach is more effective than WPE and DNN1+WPE+DNN2.

[0258] In DNN1+ICP+DNN2, the SI-SDR and PESQ scores are improved by setting the lower limit value ε to 1.0. When the lower limit value is set to 0.001, the SI-SDR and PESQ scores in DNN1+FCP+DNN2 are further improved. For example, when the lower limit value is 1.0, the SI-SDR score is 11.9 and the PESQ score is 3.15. When the lower limit value is 0.001, the SI-SDR score is 12.3, and the PESQ score is 3.18. The lower limits at 1.0 and 0.001 are also used to evaluate the trained DNN1 using ICP and FCP. As shown in the tabular representation 1000, for DNN1+ICP at a lower limit of 1.0, the SI-SDR score is 3.2 and the PESQ score is 1.78, for DNN1+ICP at a lower limit of 0.001, the SI-SDR score is 0.7 and the PESQ score is 1.77, for DNN1+FCP at a lower limit of 1.0, the SI-SDR score is 3.6 and the PESQ score is 1.82, and for DNN1+FCP at a lower limit of 0.001, the SI-SDR score is 3.0 and the PESQ score is 1.82. Therefore, DNN1+FCP+DNN2 shows better scores than the methods ICP and FCP used to train DNN1.

[0259] In general, for speech dereverberation, the hybrid SI-SDR and PESQ improved from -3.6 dB and 1.64 to 8.2 dB and 2.65 using one DNN (i.e., DNN1), to 9.1 dB and 2.82 using two DNNs (i.e., DNN1+DNN2), to 12.3 dB and 3.18 by adding an FCP module between the two DNNs (DNN1+FCP+DNN2), and to 12.8 dB and 3.24 by using one additional iteration of FCP and DNN2 (DNN1+(FCP+DNN2)×2).

[0260] Finally, the magnitude domain loss is added during the training of the second DNN 2. Improvements on word error rate (WER) and PESQ are obtained, while SI-SDR drops by about 0.5 dB.

[0261] Fig.11A tabular representation 1100 depicting evaluation results for dereverberation of a speech signal using a test dataset is shown in accordance with an embodiment of the present disclosure. The evaluation results show performance on the SMS-WSJ dataset as well as oracle results obtained by using a target direct path signal with or without early reflections, and oracle masks such as spectral magnitude mask (|S| / |Y|) and phase sensitive mask (|S| / |Y|cos(∠S-∠Y)). As shown in the tabular representation 1100, using the oracle target direct path signal for ASR achieves better WER (6.4% vs. 7.04%) than overusing the target direct path signal with early reflections, indicating the potential benefit of removing early reflections.

[0262] The table representation 1100 also shows the trained DNN1 and the loss function and As shown in tabular representation 1100, DNN1+DNN2 shows consistent improvement over DNN1.

[0263] For DNN-WPE, two variants for multi-speaker scenarios are used. The first variant uses the PSD of each estimated target speaker produced by DNN1 to compute a different WPE filter for each speaker. In the tabular representation 1100, DNN-WPE for the multi-speaker scenario is represented as DNN1+mfWPE+DNN2, where "mf" indicates multi-filter. The multi-filter sums all estimated target speakers provided by DNN1 and uses the PSD of the summed signal to compute a single WPE filter to dereverberate the mixture. The second variant is represented as DNN1+sfWPE+DNN2, where "sf" indicates a single filter.

[0264] As shown in tabular representation 1100, DNN1+sfWPE+DNN2 achieves slightly better performance than DNN1+mfWPE+DNN2, which suggests that computing separate filters for each target speaker may not work with WPE.

[0265] The scenario where all speakers are providing speech signals is represented as “allSpks” in the table representation 1100, where DNN2 is trained to enhance all target speakers simultaneously. As shown in the table representation 1100, DNN1+FCP+DNN2 shows better performance in all metrics compared to DNN1+sfWPE+DNN2 and DNN1+ICP+DNN2. This demonstrates that when there are competing speakers, convolutional predictions (such as Figure 5 and 6 The effectiveness of forward filtering on WPE in dereverberation is discussed in detail in

[14] .

[0266] If DNN2 is trained as Figure 6 Further improvements are achieved by enhancing the target speakers one by one as described in 1100 (denoted as “perSpk” in table representation 1100 ). This shows that dereverberation for each speaker individually can improve speech enhancement for that speaker. As shown in table representation 1100 , iterating convolutional prediction and DNN2 again can achieve consistent improvements. In addition, DNN2 trained by including a magnitude-level loss improves PESQ, eSTOI, and WER, while SI-SDR decreases.

[0267] In the tabular representation 1100, it is further shown that DNN1+(FCP+DNN2)x2 trained with the magnitude level loss function obtains scores of 12.2, 3.24, 89.0, and 12.77 for SI-SDR, PESQ, eSTOI, and WER, respectively. DNN1+(FCP+DNN2)x2 trained with the magnitude level loss function can perform better than DNN1+(FCP+DNN2)x2 trained with a spectrum mapping corresponding to a single microphone (such as a single-input, single-output microphone (SISO1)) for another complex spectrum mapping (12.5 vs. 5.1 dB SI-SDR). DNN1+(FCP+DNN2)x2 trained with the magnitude level loss function can perform better than DNN1+(FCP+DNN2)x2 trained with DPRNN-TasNet (12.5 vs. 6.5 dB SI-SDR).

[0268] Tabular representation 1100 also shows the performance of DNN1 and DNN2 trained based on spectral mapping corresponding to microphone arrays, such as a six-microphone, beamformed SISO with microphone arrays, (SISO1-BF-SISO2) that combines monaural complex spectral mapping with beamforming and post-filtering. These results suggest that combining an end-to-end DNN with convolutional prediction may be effective for reducing reverberation in a mixture of acoustic signals including speech signals of speakers (e.g., speakers 102A and 102B).

[0269] Fig.12 A tabular representation 1200 depicting evaluation results of dereverberation of speech signals using a test dataset according to some other embodiments of the present disclosure is shown. Tabular representation 1200 shows SI-SDR on the WHAMR! dataset. As shown in tabular representation 1200, DNN1+FCP+DNN2 produces better results than DNN1+mfWPE+DNN2 (7.4 vs. 6.8 dB SI-SDR). This suggests that DNN-FCP may be more robust than DNN-WPE in dereverberation when noise and competing speakers are present.

[0270] Tabular representation 1200 also shows a comparison with an end-to-end speech separation system such as Wavesplit. DNN1+(FCP+DNN2)x2 obtains a SI-SDR score of 7.5dB, which is higher than the SI-SDR score of Wavesplit, i.e., 5.9dB. Wavesplit may use speaker identity as auxiliary information during training for target speaker extraction. DNN1+(FCP+DNN2)x2 may not rely on the availability of information on speaker identity. In addition, dynamic mixing may be applied to data augmentation, resulting in a better SI-SDR (7.1dB). DNN1+(FCP+DNN2)x2 may be trained without data augmentation, and performs better than Wavesplit with dynamic mixing.

[0271] Fig.13 2 is a block diagram of an audio signal processing system 1300 according to an embodiment of the present disclosure. The audio signal processing system 1300 uses the system 200. In some example embodiments, the system 200 having a DNN (e.g., DNN1 206A and DNN2 206B) for dereverberation of a speech signal can be implemented on a remote server or in a cloud network. In some embodiments, the audio signal processing system 1300 (hereinafter referred to as the system 1300) can receive a RIR model (such as the RIR model 316A to the audio signal processing system 1300). The system 1300 can process the RIR model to perform audio analysis for at least one or a combination of dereverberation of a speech signal, room geometry reconstruction, and speech enhancement.

[0272] In some example implementations, system 1300 includes a sensor 1302 or multiple sensors, such as acoustic sensors, that collect data including acoustic signals 1204 from environment 1306. Environment 1306 corresponds to environment 100.

[0273] The acoustic signal 1304 may include one or more target direct path signals and their reverberations. For example, the acoustic signal 1304 may include multiple speakers with overlapping speech and their reverberations. In addition, the sensor 1302 may convert the acoustic input into the acoustic signal 1304.

[0274] The audio signal processing system 1300 includes a hardware processor 1308 in communication with a computer storage memory such as a memory 1310. The memory 1310 includes stored data including other data, algorithms and instructions that can be implemented by the hardware processor 1308. It is contemplated that the hardware processor 1308 may include two or more hardware processors depending on the requirements of a particular application. The two or more hardware processors may be internal or external. The audio signal processing 1300 may be combined with other components including output interfaces and transceivers in other devices.

[0275] In some alternative embodiments, the hardware processor 1308 can be connected to a network 1312, which communicates with one or more data sources 1314, a computer device 1316, a mobile phone device 1318, and a storage device 1320. As a non-limiting example, the network 1312 may include one or more local area networks (LANs) and / or wide area networks (WANs). The network 1312 may also include an enterprise-wide computer network, an intranet, and the Internet. The audio signal processing system 1300 may include one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources may include a single device or multiple devices that collaborate in a distributed environment of the network 1312.

[0276] In some other alternative implementations, the hardware processor 1308 may be connected to a network-enabled server 1322 connected to a client device 1324. The hardware processor 1308 may be connected to an external memory device 1326 and a transmitter 1328. In addition, the output for each target speaker may be output according to a specific user intended purpose 1330. For example, the specific user intended purpose 1330 may correspond to displaying speech (such as voice commands) as text on one or more display devices (such as a monitor or screen), or inputting the text for each target speaker into a computer-related device for further analysis, etc.

[0277] Data sources 1314 may include data resources for training DNNs, such as DNN1 206A and DNN2 206B for speech separation tasks. For example, in an embodiment, the training data may include acoustic signals of multiple speakers, such as speaker 102A and speaker 102B speaking at the same time. The training data may also include acoustic signals of a single speaker speaking alone, acoustic signals of a single or multiple speakers speaking in a noisy environment, and acoustic signals of a noisy environment (e.g., environment 100 with reverberant noise signal 110A).

[0278] The data source 1314 may also include data resources for training DNN1 206A and DNN2 206B for speech recognition tasks. The data provided by the data source 1314 may include labeled data and unlabeled data, such as transcribed data and untranscribed data. For example, in an embodiment, the data includes one or more sounds and may also include corresponding transcription information or labels, which may be used to initialize the speech recognition task.

[0279] In addition, the unlabeled data in the data source 1314 can be provided by one or more feedback loops. For example, usage data from a verbal search query performed on a search engine can be provided as untranscribed data. As an example and not limitation, other examples of data sources can include various spoken audio or image sources, including streaming sound or video, web page queries, mobile device camera or audio information, webcam video, smart glasses and smart watch data, customer service systems, security camera videos, web documents, directories, user information streams, SMS logs, instant messaging logs, spoken transcriptions, gaming system user interactions (such as voice commands or captured images (e.g., depth camera images)), tweets, chat or video call records, or social networking media. The specific data source 1314 used can be determined based on the application, including whether the data is essentially a specific category of data (e.g., data related only to a specific type of sound, including, for example, machine systems, entertainment systems) or general data (non-category specific).

[0280] The audio signal processing system 1300 may also include a third-party device, which may include any type of computing device, such as an automatic speech recognition (ASR) system on a computing device. For example, the third-party device may include a computer device or a mobile device 1318. The mobile device 1318 may include a personal data assistant (PDA), a smart phone, a smart watch, smart glasses (or other wearable smart devices), an augmented reality headset, a virtual reality headset, a laptop, a tablet computer, a remote control, an entertainment system, a vehicle computer system, an embedded system controller, an appliance, a home computer system, a security system, a consumer electronic device, or other similar electronic devices. The mobile device 1318 may also include a microphone or line input for receiving audio information, a camera for receiving video or image information, or a communication component (e.g., Wi-Fi functionality) for receiving such information from another source (such as the Internet or a data source 1314). In an example embodiment, the mobile device 1318 is capable of receiving input data such as audio and image information. For example, the input data may include a query of a speaker to a microphone of the mobile device 1318 when multiple speakers in the room are speaking. The input data can be processed by the ASR in the mobile device 1318 using the system 200 to determine the content of the query. The system 200 can enhance the input data by reducing noise in the speaker's environment, separating the speaker from other speakers, or boosting the audio signal of the query, and enables the ASR to output an accurate response to the query.

[0281] In some example embodiments, storage device 1320 may store information including data, computer instructions (e.g., software program instructions, routines, or services), and / or data related to a DNN, such as DNN1 206A and DNN2 206B of system 200. For example, storage device 1220 may store data from one or more data sources 1314, one or more deep neural network models, information for generating and training deep neural network models, and computer usable information output by one or more deep neural network models.

[0282] Fig.14A 14 is a block diagram of a system 1400A for dereverberation of a speech signal according to some example embodiments of the present disclosure. The system 1400 may be used to estimate a target audio signal from an input audio signal 1402 obtained from a sensor 1404 monitoring an environment 1406 .

[0283] The input audio signal 1402 includes a mixture of acoustic signals, which includes a target direct path signal (e.g., the target direct path signal 106A) and a corresponding reverberation (e.g., the reverberation 108A). The system 1400 processes the audio signal 1402 via a processor 1408 using a feature extraction module 1410. The feature extraction module 1410 calculates an audio feature sequence from the input audio signal 1402. A first target direct path signal estimation module 1412 processes the audio feature sequence to output a first estimate (e.g., the first estimate 408 of the target direct path signal 106A). The first estimate of the target direct path signal is processed by a filter estimation module 1414 to output a filter that models a room impulse response that affects the target direct path signal. For example, the target direct path signal may be affected to transform into a target reverberation signal. The filter is applied to the first estimate to output a reverberation-reduced mixture. The filter and the first estimate are further processed by a target direct path reverberation-reduced mixture estimation module 1416, which estimates a target direct path reverberation-reduced mixture. The mixture, first estimate, and features of the target direct path reverberation reduction are further processed by a second target direct path estimation module 1418 to calculate a signal estimate 1424 (e.g., second estimate 410) of the target direct path signal. The signal estimate 1424 is output via an output interface 1422. In some embodiments, the room impulse response modeled by the filter can be output via the output interface 1422. The output room impulse response can be used in an audio analysis application to perform one or a combination of room geometry reconstruction, speech enhancement, and dereverberation of a speech signal.

[0284] In some example implementations, the network parameters 1420 may be input to the first target direct path signal estimation module 1412, the filter estimation module 1412, the target direct path reverberation reduced mixture estimation module 1414, and the second target direct path estimation module 1418. The network parameters 1420 may include labeled data and unlabeled data, such as transcribed data and untranscribed data of different sounds or utterances that may be used to initialize a speech recognition task.

[0285] Fig. 14B A block diagram of a system 1400B for dereverberation of a speech signal according to some other example embodiments of the present disclosure is shown.

[0286] System 1400B includes a processor 1426 configured to execute stored instructions, and a memory 1428 storing instructions for a neural network 1430, which includes a speech separation network 1432 that allows speech separation and reverberation reduction. Processor 1426 can be a single-core processor, a multi-core processor, a graphics processing unit (GPU), a computing cluster, or any number of other configurations. Memory / storage 1428 can include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Memory 1328 can also include a hard drive, an optical drive, a thumb drive, a drive array, or any combination thereof. Processor 1426 is connected to one or more input and output interfaces / devices via bus 1434. In addition, system 1400B can include one or more microphones 1438 connected via bus 1434. The system 1400B is configured to receive / acquire a voice signal 1456 via one or more microphones 1438 or via a network interface 1452 and a network 1454 connected to a data source of the voice signal 1456 .

[0287] The memory 1428 stores a neural network 1430 that is trained to transform a mixture including a speech signal and a corresponding reverberated acoustic signal into a separated speech signal with reduced reverberation. The processor 1426 executing the stored instructions uses the neural network 1430 retrieved from the memory 1428 to perform speech separation. The neural network 1430 is trained to transform a mixed acoustic signal including a speech signal into a separated speech signal. The neural network 1430 may include a speech separation network 1432 that is trained to estimate the separated signal from the acoustic features of the acoustic signal.

[0288] Fig.15 A use case 1500 for dereverberation of a speech signal according to some example embodiments of the present disclosure is shown. Use case 1500 corresponds to a remote conference room including a group of speakers, such as speaker 1502A, speaker 1502B, speaker 1502C, speaker 1502D, speaker 1502E, and speaker 1502F (speaker groups 1502A-1502F). Speech signals of one or more speakers in speaker groups 1502A-1502F are received by an audio receiver 1506 of device 1504. Audio receiver 1506 is equipped with system 200 and receives acoustic speech signals from one speaker or one or more speakers in speaker groups 1502A-1502F.

[0289] Audio receiver 1506 may include a single microphone and / or a microphone array for receiving a mixture of acoustic signals from speaker groups 1502A-1502F and a noise signal in a remote conference room. These mixtures of acoustic signals from speaker groups 1502A-1502F may be processed using system 200. For example, system 200 may analyze a RIR model of a remote conference room. The RIR model may be used to generate a room geometry for a remote conference room. The room geometry may be used to locate reflective boundaries in a remote conference room. For example, the corresponding room geometry may be used to determine a location for placing a speaker, a seating arrangement for speaker groups 1502A-1502F, etc., to balance noise and other interferences in the remote conference room. In addition, the RIR model may be used to remove reflections and reverberation of speech signals of one or more speakers in speaker groups 1502A-1502F.

[0290] In an illustrative example scenario, multiple speakers in speaker group 1502A-1502F can output speech signals simultaneously. In this scenario, system 200 reduces reverberation in a remote conference room and individual speech signals of each of speakers 1502A-1506. System 200 can also perform beamforming of a mix of acoustic signals from a microphone array to enhance speech signals of corresponding speakers in speaker group 1502A-1502F. The enhanced speech signals can be used for transcription of the speaker's speech. For example, device 1504 can include an ASR module. The ASR module can receive the enhanced speech signal to output a transcription. The transcription can be displayed via a display screen of device 1504.

[0291] Fig.16 A use case 1600 for dereverberation of a speech signal according to some other example embodiments of the present disclosure is shown. Use case 1600 corresponds to a factory floor including one or more speakers, such as speaker 1602A and speaker 1602B. The factory floor may have highly reverberant signals and noise due to the operation of different industrial machines. The factory floor may also be equipped with an audio device 1604 for facilitating communication between a control operator (not shown) of the factory floor and one or more speakers 1602A and 1602B in the factory floor. The audio device 1604 may be equipped with system 200.

[0292] In an illustrative example scenario, audio device 1604 may send an audio command that may be addressed to person 1602A who manages a factory floor. The audio command may include "REPORT STATUS OF MACHINE 1". Speaker 1602A may utter "MACHINE 1 OPERATING". However, the speech signal of speaker 1602A's speech may be mixed with noise from the machine, noise from the background, and other speech from speaker 1602B in the background.

[0293] Such noise and reverberation signals can be mitigated by system 200. System 200 outputs the clean speech of speaker 1602A. The clean speech is input to audio device 1604. Audio device 1604 receives the clean speech and captures responses to audio commands from the clean speech corresponding to the utterance of speaker 1602A. System 200 enables audio devices to achieve enhanced communication with intended speakers, such as speaker 1602A.

[0294] Fig.17 A use case 1700 for dereverberation of a speech signal according to yet other example embodiments of the present disclosure is shown. Use case 1700 corresponds to a driver assistance system 1702. Driver assistance system 1702 is implemented in a vehicle such as a manually operated vehicle, an automatic vehicle, or a semi-automatic vehicle. The vehicle is occupied by one or more persons (such as person 1704A and person 1704B). Driver assistance system 1702 is equipped with system 200. For example, driver assistance system 1702 can be remotely connected to system 1702 via a network (such as network 1754). In some alternative example embodiments, system 200 can be embedded within driver assistance system 1702.

[0295] Driver assistance system 1702 may also include one or more microphones to receive a mixture of acoustic signals. The mixture of acoustic signals may include voice signals from people 1704A and 1704B and external noise signals (such as horns of other vehicles, etc.). In some cases, when person 1704A is sending a voice command to driver assistance system 1702, another person 1704B may be louder than person 1704A. The speech from person 1704B may interfere with the voice command of person 1704A. For example, the voice command of person 1704A may be "FIND THE NEAREST PARKING AREA", and the speech of person 1704B may be "LOOK FOR A SHOPPING MALL TO PARK". In this case, system 200 processes the speech of each of people 1704A and people 1704B simultaneously or individually. System 200 separates the speech of people 1704A and people 1704B. Driver assistance system 1702 uses separate utterances. Driver assistance system 1702 can process and execute the voice command of person 1704A and the utterance of person 1704B, and output a response to each utterance accordingly.

[0296] Fig.18 1800 is a use case for dereverberation of a speech signal according to yet other example embodiments of the present disclosure. In some example embodiments, the systems 200a, 200b, 200c (in Figure 2A , Figure 2B , Figure 2C 1804 ) can process pre-recorded data or live recorded sound to determine an estimate of the target direct path signal. Pre-recorded data of sound can be accessed from a database via network 1808. Network 1808 is an example of network 1312. Likewise, live recordings of sources can be streamed from corresponding sources at remote locations via network 1808.

[0297] The estimate of the target direct path signal can be filtered by the system 200 to determine the RIR model. The RIR model can be analyzed by an audio signal processing system (such as the audio signal processing system 1300 connected to the system 200). The audio signal processing system 1300 can process the RIR model for the room acoustic simulation 1802 of the environment (such as the concert hall 1806). The RIR model can be convolved with the recorded audio track source to imprint the acoustic effect of the concert hall 1806 based on the room acoustic simulation 1802. Using the room acoustic simulation 1802, an actual simulated environment or virtual reality environment of the concert hall 1806 can be created. The simulated environment of the concert hall 1806 can enable a music performer to rehearse before actually performing at the concert hall 1806.

[0298] In some cases, room acoustic simulation 1802 may be used to model room acoustic behavior for room geometry reconstruction 1804. Room geometry reconstruction 1804 may provide a number of architectural aspects for design and construction to maximize the listening experience of an audience in a concert hall, such as concert hall 1806.

[0299] In some embodiments, the room acoustic simulation 1802 may be used to model room acoustic parameters such as at least one of the following: direct to reverberation ratio, reverberation time (RT60), early decay time (EDT), center time, clarity C80, and resolution D50. These parameters are well known in the art.

[0300] By combining operations 902 to 920 in the above manner, methods 900a and 900b performed using a processor 208 arranged in the system 200 can allow for estimating filters including the amplitude and phase of reverberation, which can improve the dereverberation of speech signals. The filters are estimated based on a convolution prediction method, which enables the filters to reduce early reflections of the target direct path signal. In addition, the filter model is used for signal propagation in the room, i.e., RIR, which can improve the accuracy of reverberation estimation. The use of two DNNs in the system 200 can also improve the performance of dereverberation of speech signals and tasks such as speech enhancement and speaker separation. More specifically, the first DNN estimates a first estimate of the target direct path signal based on a mixture of acoustic signals including reverberation. The second DNN uses the first estimate and other data (such as filters and reduced reverberation estimated by the filters) to estimate a refined estimate of the target direct path signal. In this way, the two DNNs are able to identify and distinguish the target direct path signal from high reverberation and noise in an efficient and feasible manner.

[0301] In addition, various embodiments may be described as processes depicted as flow charts, process views, data flow diagrams, structure diagrams, or block diagrams. Although flow charts may describe operations as sequential processes, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. Processing may terminate when its operations are completed, but may have additional steps not discussed or included in the figure. In addition, not all operations occur in any particularly described processing in all embodiments. Processing may correspond to methods, functions, procedures, subroutines, subprograms, etc. When processing corresponds to a function, the termination of the function may correspond to the function returning to the calling function or main function.

[0302] In addition, embodiments of the disclosed subject matter may be implemented at least in part manually or automatically. Manual or automatic implementation may be performed or at least assisted by the use of a machine, hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks may be stored in a machine-readable medium. A processor may perform the necessary tasks.

[0303] The above-mentioned embodiments of the present disclosure can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or processor set provided in a single computer or distributed in multiple computers. Such a processor can be implemented as an integrated circuit, and one or more processors are in an integrated circuit assembly. However, the processor can be implemented using a circuit of any suitable format.

[0304] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, in various embodiments, the functions of program modules may be combined or distributed as desired.

[0305] In addition, embodiments of the present disclosure may be embodied as a method, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from that shown, which may include performing some actions simultaneously, even though the actions are shown as sequential actions in the illustrative embodiments. Therefore, it is the intent of the appended claims to cover all such variations and modifications that fall within the true spirit and scope of the present disclosure.

[0306] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. Therefore, the aspects of the appended claims cover all such changes and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. A method for estimating a room impulse response RIR, wherein: The method uses a processor coupled to stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the steps of: receiving a mixture of acoustic signals through a wired or wireless communication channel, the mixture of acoustic signals comprising a target direct path signal propagating in a room and a reverberation of the target direct path signal within the room; submitting the mixture of the received acoustic signals to a first deep neural network (DNN) to produce a first estimate of the target direct path signal; estimating a filter that models a room impulse response, RIR, representing a relationship between the target direct path signal and the reverberation of the target direct path signal, wherein the filter, when applied to the first estimate of the target direct path signal, generates a result that is closest to at least one of a mixture of the acoustic signal and a residual between the mixture of the acoustic signal and the first estimate of the target direct path signal according to a distance function; and The filter that models the RIR is transmitted.

2. The method according to claim 1, wherein: The first DNN is trained to estimate the amplitude and phase of the target direct path signal, wherein applying the filter that models the RIR to the first estimate of the target direct path signal includes calculation of both the estimated amplitude and phase, and wherein a distance function between a result of applying the filter that models the RIR to the first estimate of the target direct path signal and a mixture of the acoustic signal is measured in both the amplitude domain and the phase domain.

3. The method according to claim 1, wherein: The first DNN is pre-trained to obtain the first estimate of the target direct path signal from a mixture of observed acoustic signals.

4. The method according to claim 3, wherein: Pre-training of the first DNN is performed using a mixed training dataset of acoustic signals and corresponding reference direct path signals in the training dataset by minimizing a loss function, the loss function comprising one or a combination of: a distance function defined based on real and imaginary RI components of the first estimate of the target direct path signal in the first time-frequency domain and the RI component of the corresponding reference direct path signal in the first time-frequency domain, a distance function defined based on the amplitude and phase obtained from the RI component of the first estimate of the target direct path signal in the first time-frequency domain and the corresponding amplitude and phase of the reference direct path signal in the first time-frequency domain, a distance function defined based on a reconstructed waveform obtained by reconstruction in the time domain from the RI component of the first estimate of the target direct path signal in the first time-frequency domain and a waveform of the reference direct path signal, a distance function defined based on the first estimate in a second time-frequency domain by further transforming the reconstructed waveform in the second time-frequency domain and the RI component of the reference direct path signal in the second time-frequency domain, and A distance function defined based on a magnitude of an RI component obtained from the first estimate of the target direct path signal in the second time-frequency domain by further transforming the reconstructed waveform in the second time-frequency domain and a corresponding magnitude of the reference direct path signal in the second time-frequency domain.

5. The method according to claim 1, wherein: The coefficients of the filter that models the RIR are estimated in the time-frequency domain.

6. The method according to claim 1, wherein: The target direct path signal is a signal transmitted from a source of an original signal to a sensor, wherein the target direct path signal represents the original signal at the source that should be measured by the sensor in the absence of reverberation, and wherein reverberation of the target direct path signal represents one or more transmissions of the original signal from the source to the sensor over one or more paths longer than a shortest path, such that the RIR is modeled with respect to the target direct path signal.

7. The method according to claim 1, wherein: The filter that models the RIR is a linear filter estimated using a linear convolutional prediction module.

8. The method according to claim 7, wherein: The linear convolution prediction implements a linear structure of the filter that models the RIR to approximate linear reflections of the target direct path signal from surfaces within the room.

9. The method according to claim 1, wherein: The filter that models the RIR is applied to the first estimate of the target direct path signal in the time-frequency domain, and wherein the distance function is a weighted distance, and the weight at each time-frequency point in the time-frequency domain is determined by one or a combination of a mixture of the received acoustic signal and the first estimate of the target direct path signal, and wherein the distance function is based on a least square distance.

10. The method according to claim 1, further comprising the steps of: The filter modeling the RIR is applied to an external signal measured outside the room to synthesize a new acoustic mixture, wherein the new acoustic mixture provides the acoustic effect of the external signal propagating within the room.

11. The method according to claim 10, further comprising the steps of: The first DNN is updated using the new acoustic mixture.

12. The method according to claim 1, further comprising the steps of: obtaining a reduced-reverberation mixture of the target direct path signal by removing from a mixture of received acoustic signals a result of applying the filter modeling the RIR to the first estimate of the target direct path signal; submitting the reduced reverberation mixture to a second DNN to produce a second estimate of the target direct path signal; and The second estimate of the target direct path signal is output via an output interface.

13. The method according to claim 1, further comprising the steps of: updating the first estimate of the target direct path signal based on application of the filter that models the RIR; updating training data based on a mixture of the updated first estimate of the target direct path signal and the acoustic signal as a pseudo label; as well as The first DNN is retrained based on the updated training data.

14. The method according to claim 12, further comprising the steps of: generating enhancement data based on the filter that models the RIR, wherein the enhancement data is obtained by applying the filter that models the RIR to data including one or more of: external data measured outside the room, data obtained from one or more of a second estimate of the target direct path signal as another mixture of acoustic signals in training data, and the first estimate of the target direct path signal; and An enhanced training dataset is created based on the enhanced data.

15. The method according to claim 14, further comprising the steps of: The enhanced training data set is used to train a system for automatic speech recognition.

16. The method according to claim 14, further comprising the steps of: The enhanced training dataset is used to train a system for sound event detection.

17. The method according to claim 1, further comprising the steps of: The filter that models the RIR is processed to perform an audio analysis of at least one or a combination of room acoustic parameter analysis, room geometry reconstruction, speech enhancement, and dereverberation of a speech signal, wherein the room acoustic parameters include at least one of the following: direct to reverberation ratio, reverberation time RT60, early decay time EDT, center time, clarity C80, and resolution D50.

18. The method according to claim 1, wherein: The received mixture of acoustic signals includes speech signals from a plurality of speakers, and wherein the first DNN produces a plurality of outputs, each output including the first estimate of the target direct path signal from a speaker of the plurality of speakers.

19. The method according to claim 18, further comprising the steps of: A corresponding filter is estimated that models the corresponding RIR for each of the plurality of speakers.

20. The method according to claim 1, further comprising the steps of: The filter that models the RIR is processed with a downstream DNN trained to perform a task based on the RIR, wherein the task includes one or a combination of: Speaker recognition, emotion recognition, event detection, localization.

21. A system for estimating a room impulse response RIR, the system comprising: an input interface configured to receive a mixture of acoustic signals through a wired or wireless communication channel, the mixture of acoustic signals comprising a target direct path signal propagating in a room and a reverberation of the target direct path signal within the room; A memory storing a first deep neural network DNN; A processor, the processor being configured to: submitting a mixture of the received acoustic signals to the first DNN to produce a first estimate of the target direct path signal, estimating a filter that models the RIR, the RIR representing a relationship between the target direct path signal and the reverberation of the target direct path signal, wherein the filter, when applied to the first estimate of the target direct path signal, generates a result that is closest to a residual between a mixture of the acoustic signal and the first estimate of the target direct path signal according to a distance function; as well as An output interface configured to output the filter modeling the RIR through the communication channel.

Images