Method and system for dereverberating a speech signal - Patents.com

Abstract

A system and method for reverberation reduction is disclosed. A first deep neural network (DNN) generates a first estimate of a target direct path signal from an acoustic signal mixture including the target direct path signal and reverberation of the target direct path signal. A filter is estimated that models a room impulse response (RIR) of the first estimate. The filter, when applied to the first estimate of the target direct path signal, generates a result that is closest to a residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function. A reverberant-reduced mixture of the target direct path signal is obtained by removing a result of applying the filter to the first estimate of the target direct path signal from the received mixture. A second DNN generates a second estimate of the target direct path signal from the reverberant-reduced mixture.

Description

[Technical field]

[0001] The present disclosure relates generally to audio signal processing, and more particularly to methods and systems for dereverberation of speech signals. [Background technology]

[0002] Generally speaking, in a closed room, reverberation of an audio signal (e.g., speech) occurs during modern hands-free speech communication, such as remote conferencing and interaction with smart devices, such as a microphone on a smart speaker. In such a closed room, the speech signal propagates through the air and may be reflected by walls, floors, ceilings, and other objects in the room before being captured by the microphone. Reverberation is the multipath propagation of a speech signal from a source or speaker to a receiving end, such as a microphone. Such speech reverberation occurs when sound reflects off surfaces in the environment. Part of the sound may be absorbed by those surfaces, which causes multiple attenuations of the speech signal. The reflection and absorption of sound by those surfaces may generate multiple attenuated and delayed copies of the speech signal. These multiple attenuated and delayed copies may degrade the quality of the speech and hinder the performance of an Automatic Speech Recognition (ASR) system or any speech / audio processing system. For example, an ASR may generate inaccurate output due to a voice input with degraded speech quality.

[0003] Speech reverberation can be reduced by removing the effects of reverberation from the sound. Such removal of the effects of reverberation is known as dereverberation. Dereverberation may include identifying the direct path signal and distinguishing it from attenuated and delayed copies. The direct path signal corresponds to the signal that the sound follows when the source and microphone are in line of sight. However, it may be difficult to identify the direct path signal and distinguish it from the copies, especially when the reverberation is high and noise from non-stationary sources is present. For example, an environment such as an enclosed room with non-stationary sources such as an air conditioning system may have a large room reverberation. It may be difficult to reduce the reverberation due to noise from the air conditioning system or any multi-source environmental noise. Multi-source environmental noise may also correspond to a scenario where multiple people are talking in the environment. Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, there is a need to overcome the above problems, and more specifically, to develop a method and system for dereverberating a speech signal while overcoming reverberant conditions and non-stationary noise in a reverberant environment. [Means for solving the problem]

[0005] It is an object of some embodiments to develop a method and system for dereverberation of a speech signal. Another object of some embodiments is to perform dereverberation of a speech signal using deep learning techniques. Dereverberation of a speech signal can be extended to tasks such as reverberation reduction, speech enhancement, speaker separation, etc.

[0006] Some embodiments are based on the understanding that clean speech exhibits spectro-temporal patterns. Such spectro-temporal patterns are unique patterns exhibited in the time-frequency domain and can provide useful clues for reverberation reduction. Some of these patterns originate from the structure of the speech signal itself, but some patterns may also correspond to a linear filter structure of reverberation (i.e., reflections of sound waves) specific to the physical space in which the recording is made, including all objects, structures or entities present in that space, as well as the location of the source speech signal and the receiver, such as the microphone, that records the signal. This linear filter structure can be used to account for the signal originating from the source signal at the microphone location and reflections of that signal from walls and surfaces of objects or people in the space, which represents the effect of reverberation on the input signal as a linear convolution of the input signal and the Room Impulse Response (RIR). The input signal is the original source signal, also known as the dry source signal. The Room Impulse Response represents the effect of the space and everything in that space on the input signal. An estimate of the RIR between a source location and a receiver location can be recorded in a physical space such as a room, for example by firing an impulse sound (e.g., a blank gun or a balloon burst) at a source location in the room, which is a short-duration time-domain signal, and recording the resulting signal at the receiver location. The impulse excites the room and produces a reverberant impulse signal that can be used to estimate the RIR. The reverberation of a dry source sound signal that would be played at the same source location and recorded at the same receiver location can then be modeled by convolving the dry source signal and the estimated RIR. To that end, an objective of some embodiments is to estimate a basic filter to approximate or model the RIR. In some exemplary embodiments, the RIR can be estimated based on a linear regression problem solved for each frequency in the time-frequency domain.The filter estimates that model the RIRs can be used to identify delayed and attenuated copies of an input signal for dereverberation of a speech signal.

[0007] Moreover, such linear filters can be exploited as regularization to improve the dereverberation process. For example, linear filters as regularization prevent overfitting of the model of the dereverberation process to the training data. Some embodiments are based on the recognition that a linear filter structure can be utilized in combination with linear prediction and deep learning for single-channel and multi-channel reverberant speaker separation and dereverberation tasks. For that purpose, deep learning techniques supported by convolutional prediction can be used for dereverberation in environments with noise signals, reverberation of speech signals, etc. Convolutional prediction is a linear prediction method for speech dereverberation in reverberant situations that relies on source estimates obtained by a Deep Neural Network (DNN) and utilizes a linear filter structure between the source estimates and the reverberant version of the source signal in the observed input signal.

[0008] To obtain the source estimate, a DNN is trained in the time-frequency domain or the time domain to predict the target speech from the reverberant speech. The target speech corresponds to the target direct path signal between the source and a receiver such as a microphone. This approach can exploit prior knowledge of the speech patterns.

[0009] Previous studies have also attempted to utilize some form of linear filter structure to perform dereverberation. For example, weighted prediction error (WPE) may be used to dereverberate speech signals. WPE methods calculate an inverse linear filter based on variance normalized delayed linear prediction. The calculated linear filter is applied to past observations of a mixture input signal containing reverberation and possibly noise to estimate late reverberation of a target source signal in the mixture input signal from past observations of reverberation for dereverberation. The estimated late reverberation is subtracted from the acoustic signal mixture received from various sources to estimate a target speech signal in the acoustic signal mixture. In some embodiments, the filter may also be estimated using a time-varying power spectral density (PSD) of the target speech signal. The PSD is the distribution of the power of a signal over the frequency domain of the signal. Such a linear filter may be iteratively estimated using WPE in an unsupervised manner. However, the iterative procedure of WPE for filter estimation may lead to suboptimal results and may be computationally expensive.

[0010] To overcome the above deficiencies of WPE, the iterative procedure for filter estimation can be replaced with a DNN-based WPE (DNN-WPE) approach. DNN-WPE uses the amplitude estimated by the DNN as the PSD of the target speech signal for filter estimation. However, DNN-WPE cannot reduce early reflections because it requires a strict non-zero frame delay to avoid trivial solutions and cannot have a mechanism to utilize the phase estimated by the DNN for filter estimation. Also, DNN-WPE may not be robust to interference caused by noise signals. For example, DNN-WPE may estimate a filter that relates a noisy past observation to a noisy current observation, thereby limiting the filter estimation accuracy. Also, DNN-WPE directly uses a linear prediction result as its output, which may result in partial or minimal reverberation reduction.

[0011] To that end, another objective of some embodiments is to remove both early and late reflections for dereverberation. Early reflections and late reverberations can be removed using a convolutional prediction approach, which leverages both the amplitude and phase estimated by the DNN for filter estimation. The convolutional prediction approach also provides closed-form solutions for linear filters (similar to the DNN-WPE approach above), which are suitable for online real-time processing applications and can be jointly trained with other DNN modules, such as acoustic models.

[0012] In some embodiments, two DNNs are trained based on a convolutional prediction approach for dereverberation of speech signals. Initially, the first of the two DNNs outputs a first estimate of a direct path signal of a target source (a person speaking, hereafter referred to as a speaker) from an input, i.e., an acoustic signal mixture including the speaker's speech. The direct path signal of the target source is hereafter referred to as a target direct path signal. The first estimate of the target direct path signal is used to determine a filter using the convolutional prediction approach. The filter is such that, under some weighted distance function, the application of the filter to the target direct path estimate is as close as possible to the residual obtained by subtracting the target direct path estimate from the mixture. Furthermore, the filter is applied to the first estimate of the target direct path signal in the time-frequency domain. When the filter is applied to the first estimate of the target direct path signal, a result is obtained that identifies delayed and attenuated copies of the estimated target direct path signal from the acoustic signal mixture. These delayed and attenuated copies are, in this specification, derivative signals of the target direct path signal that are reflected in multiple paths due to reverberation. For example, a target direct path signal is reflected in different directions by different objects in an environment such as a room. Such identified delayed and attenuated copies are removed from the acoustic signal mixture for dereverberation. Removal of the delayed and attenuated copies produces a mixture with reduced reverberation.

[0013] The result obtained when the filter is applied to the first estimate of the target direct path signal is by the above configuration closest to the residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function. The distance function is a weighted distance between the filtered target direct path signal and the residual obtained by subtracting the target direct path estimate from the mixture, the weight at each time-frequency point in the time-frequency domain being determined by one or a combination of the acoustic signal mixture and the first estimate of the target direct path signal. In some embodiments, the distance function is based on a least square distance. Furthermore, the result of applying the filter to the first estimate of the target direct path signal is removed from the acoustic signal mixture to obtain a reverberant reduced mixture of the target direct path signal. In some embodiments, the reverberant reduced mixture is input to a second DNN of the two DNNs. The second DNN outputs a second estimate of the target direct path signal, which may be an improved estimate of the target direct path signal compared to the first estimate of the target direct path signal. The second DNN may also perform steps similar to those of the first DNN, but in some embodiments, the second DNN may take as input a different set of signals, such as one or a combination of the acoustic signal mixture, the reverberant reduced mixture, and the first estimate of the target direct path signal.

[0014] In some embodiments, the first DNN may be trained for speaker separation purposes. To that end, the first DNN generates a plurality of outputs corresponding to a first estimate of a target direct path signal for a speaker from a plurality of speakers. Further, the estimation of the filter and obtaining the reduced reverberation mixture are repeated for each of the plurality of speakers to generate a corresponding filter and a corresponding reduced reverberation mixture for each of the plurality of speakers. The corresponding reduced reverberation mixtures for each of the plurality of speakers are then combined, and the combined reduced reverberation mixtures for each of the plurality of speakers are fed into a second DNN. The second DNN then generates a second estimate of the target direct path signal for each of the plurality of speakers.

[0015] Additionally or alternatively, the dereverberated mixture, i.e., the delayed and attenuated copies, may be utilized as additional features for the second DNN to determine a second estimate of the target direct path signal, which improves dereverberation. Additionally or alternatively, features corresponding to the delayed and attenuated copies may also be used for the speaker separation task. In some exemplary embodiments, the delayed and attenuated copies may be identified based on a linear regression problem. In some embodiments, one or a combination of the acoustic signal mixture and the first estimate of the target direct path signal may be provided as inputs to the second DNN to generate a second estimate of the target direct path signal. In some embodiments, the acoustic signal mixture, the first estimate, and the dereverberated mixture are provided as inputs to the second DNN to determine a second estimate of the target direct path signal.

[0016] Also, some embodiments are based on the understanding that individual or each speaker of multiple speakers is convolved with a different RIR. The WPE method estimates a single filter to reduce the reverberation of all sources. However, computing a single filter for dereverberation of a mixture may be infeasible when noise or competing speakers are louder than the target source. The filter computed in this way is biased towards suppressing the reverberation of higher energy sources. To that end, a dereverberation filter needs to be estimated for each source, because each source is convolved with a different RIR. The DNN-WPE method can compute different filters for each source, but only by using the estimated PSD of each source as a weight in the distance function that DNN-WPE uses to estimate the linear prediction filter, which may limit the accuracy and variety of those different filters.

[0017] Some embodiments are based on the recognition that in the case of multiple speakers in a room, a corresponding filter is estimated for each individual speaker for dereverberation. In the case of multiple speakers, the acoustic signal mixture includes speech signals from the multiple speakers. In such a case, the first DNN generates a corresponding first estimate of the target direct path signal for each of the multiple speakers. To generate a reverberation-reduced mixture for each of the multiple speakers, the steps for determining the first estimate for each speaker, determining the filter for each speaker, and feeding one or a combination of the first estimate and the reverberation-reduced mixture for each speaker may be combined and fed into a second DNN to generate a second estimate of the target direct path signal for each of the multiple speakers.

[0018] In some cases, the acoustic signal mixture may be received from a single channel, such as a single microphone, or from multiple channels, such as an array of microphones. Each different channel measures a different version of the acoustic signal mixture. The DNN may be trained to estimate the target direct path signal in the reference channel or each channel. The training may be based on a complex spectral mapping in one or more channels. The DNN is trained to output an estimate in the time-frequency domain of the target direct path signal in one or more channels such that a distance between the estimate in the time-frequency domain of the target direct path signal in one or more channels and the reference is minimized. In the case of an array of microphones, a beamforming output may be obtained. The beamforming output may be obtained based on statistics calculated from one or a combination of the first estimate of the target direct path signal in each microphone of the array of microphones and the reverberant reduced mixture of the target direct path signal. The beamforming output may be input to a second DNN to generate a second estimate of the target direct path signal for each of the multiple speakers. Additionally or alternatively, the beamforming output and dereverberation results may be used as additional features for a second DNN to better perform separation and dereverberation tasks.

[0019] In some embodiments, the first DNN may be pre-trained to obtain a first estimate of a target direct path signal from an observed acoustic signal mixture. The pre-training of the first DNN may be performed using a training dataset of acoustic signal mixtures and a corresponding reference target direct path signal in the training dataset. In particular, the pre-training of the first DNN may be performed by minimizing a loss function. The loss function may include one or a combination of distance functions defined based on the real and imaginary (RI) components of the first estimate of the target direct path signal in the complex time-frequency domain and the RI components of the corresponding reference target direct path signal. Also, the distance function may be defined based on a magnitude obtained from the RI components of the first estimate of the target direct path signal in the complex time-frequency domain and the corresponding magnitude of the reference target direct path signal.

[0020] Additionally or alternatively, the distance function may be defined based on a reconstructed waveform obtained from the RI component of the first estimate of the target direct path signal by reconstruction in the time domain and a corresponding waveform of the reference target direct path signal.

[0021] In some alternative embodiments, the distance function may be defined based on the RI components of the first estimate in a second complex time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and the corresponding RI components of the reference target direct path signal in the second time-frequency domain.

[0022] In some alternative embodiments, the distance function may be defined based on a magnitude obtained from the RI component of the first estimate in a second complex time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and a corresponding magnitude of the reference target direct path signal in the second time-frequency domain.

[0023] In some exemplary embodiments, the first estimate of the target direct path signal may be replaced with the second estimate of the target direct path signal to obtain an updated first estimate of the target direct signal. The steps of obtaining the first estimate, obtaining the filter, and injecting the first estimate and the reverberant reduced mixture may be repeated on the updated first estimate of the target direct signal to obtain an updated second estimate of the target direct signal.

[0024] In some examples, in a multi-speaker scenario, the above steps are repeated for each of the multiple speakers to generate a corresponding filter for each of the multiple speakers. Furthermore, a portion of the received acoustic signal mixture corresponding to a speaker of the multiple speakers may be extracted by removing reverberant speech of the other speaker of the multiple speakers from the acoustic signal mixture. An estimate of the reverberant speech of the other speaker of the multiple speakers is obtained by adding the first estimate of the target direct path signal for the other speaker to a result of applying the corresponding filter for the other speaker to the first estimate of the target direct path signal for the other speaker. After extraction, a filter for estimating a reduced reverberant mixture for each speaker of the multiple speakers may be estimated based on the portion of the received mixture.

[0025] Some embodiments provide evaluation results on speech dereverberation and speaker separation that show the effectiveness of speech signal dereverberation based on a convolutional prediction approach.

[0026] Accordingly, one embodiment of the present disclosure discloses a computer-implemented method for dereverberation of a speech signal. The method includes receiving an acoustic signal mixture including a target direct path signal and a plurality of reverberations of the target direct path signal via an input interface. The method includes feeding the received acoustic signal mixture into a first DNN to generate a first estimate of the target direct path signal. The method includes estimating a filter that models a room impulse response (RIR) of the first estimate of the target direct path signal, the filter, when applied to the first estimate of the target direct path signal, generates a result that is closest to a residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function. The method includes obtaining a reduced reverberation mixture of the target direct path signal by removing the result of applying the filter to the first estimate of the target direct path signal from the acoustic signal mixture. The method includes feeding the reduced reverberation mixture into a second DNN to generate a second estimate of the target direct path signal. The method further includes outputting the second estimate of the target direct path signal via an output interface.

[0027] Accordingly, another embodiment of the present disclosure discloses a system for dereverberation of a speech signal. The system includes an input interface configured to receive an acoustic signal mixture including a target direct path signal and a reverberation of the target direct path signal. The system includes a memory storing a first deep neural network (DNN) and a second DNN. The system includes a processor configured to input the received mixture into the first DNN to generate a first estimate of the target direct path signal and to estimate a filter that models a room impulse response (RIR) of the first estimate of the target direct path signal. The filter, when applied to the first estimate of the target direct path signal, produces a result that is closest to a residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function. The processor is further configured to remove the result of applying the filter to the first estimate of the target direct path signal from the received mixture to obtain a reduced reverberation mixture of the target direct path signal and to input the reduced reverberation mixture to a second DNN to generate a second estimate of the target direct path signal. The system further includes an output interface configured to output the second estimate of the target direct path signal.

[0028] Further features and advantages will become more readily apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0029] The present disclosure is further described in the following detailed description with reference to a number of drawings as non-limiting examples of exemplary embodiments of the present disclosure, in which like reference numerals represent like parts throughout the several views of the drawings. The drawings shown are not necessarily drawn to scale, with emphasis instead generally being placed on illustrating the principles of the embodiments disclosed herein. [Brief description of the drawings]

[0030] [Figure 1A] FIG. 2 illustrates an exemplary representation for dereverberation of a speech signal according to an embodiment of the present disclosure. [Figure 1B] FIG. 2 illustrates an exemplary representation for dereverberation of a speech signal according to another embodiment of the present disclosure. [Diagram 2] 1 is a schematic block diagram of a system for dereverberation of a speech signal according to an embodiment of the present disclosure. [Figure 3A] FIG. 1 is a schematic block diagram of a process for dereverberation of a speech signal according to an embodiment of the present disclosure. [Figure 3B] FIG. 2 illustrates a representation of a Room Impulse Response (RIR) in the time domain according to an embodiment of the present disclosure. [Figure 3C] FIG. 1 illustrates a representation of the application of a filter that models the RIR in frequency bins, according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating an architecture for dereverberation of a speech signal according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a schematic diagram illustrating an architecture for dereverberation of a speech signal for a multi-speaker case, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram illustrating an architecture for dereverberation of a speech signal for a multi-speaker case, according to some other embodiments of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram illustrating an architecture representation for improving dereverberation of a speech signal according to some embodiments of the present disclosure. [Figure 8A] FIG. 2 is a schematic diagram illustrating a network architecture for dereverberation of a speech signal, according to some other embodiments of the present disclosure. [Figure 8B] FIG. 2 is a schematic diagram illustrating a network architecture for dereverberation of a speech signal, according to some other embodiments of the present disclosure. [Figure 8C]FIG. 2 is a schematic diagram illustrating a network architecture for dereverberation of a speech signal, according to some other embodiments of the present disclosure. [Figure 8D] FIG. 2 is a schematic diagram illustrating a network architecture for dereverberation of a speech signal, according to some other embodiments of the present disclosure. [Figure 9] FIG. 2 is a flow diagram of a method for dereverberation of a speech signal according to an embodiment of the present disclosure. [Figure 10] FIG. 2 illustrates a tabular representation corresponding to simulated tests for dereverberation of a speech signal, according to an embodiment of the present disclosure. [Figure 11] FIG. 1 illustrates a tabular representation of evaluation results for dereverberation of speech signals using a test dataset, according to an embodiment of the present disclosure. [Figure 12] FIG. 11 illustrates a tabular representation of evaluation results for dereverberation of speech signals using a test data set, according to some other embodiments of the present disclosure. [Figure 13] FIG. 1 is a block diagram of an audio processing system according to an embodiment of the present disclosure. [Figure 14A] FIG. 1 is a block diagram of a system for dereverberation of a speech signal, according to some exemplary embodiments of the present disclosure. [Figure 14B] FIG. 1 is a block diagram of a system for dereverberation of a speech signal according to some other exemplary embodiments of the present disclosure. [Figure 15] FIG. 2 illustrates a use case for dereverberation of a speech signal, according to some exemplary embodiments of the present disclosure. [Figure 16] FIG. 10 illustrates a use case for dereverberation of a speech signal, according to some other exemplary embodiments of the present disclosure. [Figure 17] FIG. 1 illustrates a use case for dereverberation of a speech signal, according to some further exemplary embodiments of the present disclosure. [Figure 18] FIG. 1 illustrates a use case for dereverberation of a speech signal, according to some further exemplary embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] While the above drawings set forth embodiments disclosed herein, other embodiments are contemplated, as noted in the description. The present disclosure presents exemplary embodiments by way of representation and not by way of limitation. Numerous other modifications and embodiments may be devised by those skilled in the art that fall within the scope and spirit of the principles of the embodiments disclosed herein.

[0032] In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, the apparatus and methods are shown only in block diagram form to avoid obscuring the present disclosure. It is contemplated that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0033] As used in the specification and claims, the terms "for example," "for instance," and "such as," as well as the verbs "comprising," "having," "including," and other forms of these verbs, when used in conjunction with a list of one or more components or other items, should be construed as open-ended, meaning that the list should not be considered to exclude further components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology used herein are for purposes of description and should not be considered limiting. Any headings used in this description are for convenience only and do not have any legal or limiting effect.

[0034] In the following description, specific details are set forth to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments can be practiced without these specific details. For example, systems, processes and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments. Additionally, like reference numbers and names in the various drawings refer to like elements.

[0035] Although most of the description has been given using speech as the target source, the same methods can be applied to other types of audio signals.

[0036] 1A is a diagram illustrating a representation of an environment 100A for dereverberation of a speech signal, according to an embodiment of the present disclosure. The environment 100A may correspond to an enclosed environment with a speaker 102A. Also shown in FIG. 1A is a device 104 that includes at least a microphone or an array of microphones. In some exemplary embodiments, the device 104 may correspond to an Automatic Speech Recognition (ASR) system, an audio signal processing system, or any speech processing system.

[0037] In the illustrated exemplary scenario, when speaker 102A outputs speech, the corresponding acoustic speech signals may travel towards device 104 by different paths. These acoustic speech signals may be linearly distorted by object reflections such as wall reflections, ceiling reflections, etc., as shown in Fig. 1A. In particular, the acoustic speech signal of speaker 102A is distorted in multipath directions before reaching device 104, resulting in reverberation of the acoustic speech signal.

[0038] Thus, the device 104 receives such acoustic speech signal of the speaker 102A as an acoustic signal mixture. The acoustic signal mixture includes an anechoic speech signal and a reverberant speech signal. The anechoic speech signal is the target direct path signal 106A. The reverberant speech signal, hereinafter collectively referred to as reverberation 108A, includes non-direct path or multipath signals. In some cases, there may be multiple speakers in the environment 100A, such as a speaker 102A with another speaker 102B. In such a case, the acoustic signal mixture includes the target direct path signal 106B and a reverberant speech signal, hereinafter collectively referred to as reverberation 108B, corresponding to the speaker 102B. The acoustic signal mixture may also include a reverberant noise signal 110A of a non-target source, such as an air conditioner 110 in the environment 100A.

[0039] In some cases, the speech signals of speaker 102A and / or speaker 102B may be obstructed before reaching device 104, which is illustrated in FIG. 1B.

[0040] FIG. 1B illustrates an exemplary representation for dereverberation of a speech signal according to another embodiment of the present disclosure. As shown in the environment 100B of FIG. 1B, the speech signal of the speaker 102A or speaker 102B is blocked by a block 114 before reaching the device 104. The block 114 may cause the speech signal of the corresponding speaker (such as the speaker 102A or speaker 102B) to reflect in different directions. Such reflections may increase the attenuated and delayed copies (not shown in FIG. 1B) of the speech signal of the speaker 102A or speaker 102B. When the device 104 is blocked by the block 114, the speech signal of the speaker 102A or speaker 102B may not have a corresponding target direct path signal. Instead, the speech signal may include shortest paths, such as the shortest path 106C of the speech signal corresponding to the speaker 102A and / or the shortest path 106D of the speech signal corresponding to the speaker 102B. In such a situation, for purposes of explanation, the shortest path signal is considered to be the target direct path signal, and signals corresponding to paths longer than the shortest path are considered to be reverberation.

[0041] The device 104 can reduce reverberations, such as reverberations 108A and 108B, using a system 112 that can be integrated or incorporated into the device 104. The system 112 is further described with reference to FIG.

[0042] 2 is a schematic block diagram of a system 200 for dereverberation of a speech signal according to an embodiment of the present disclosure. System 200 corresponds to system 112 of FIGS. 1A and 1B.

[0043] In some demonstrative embodiments, the system 200 includes an input interface 202, a memory 204 that stores a first deep neural network (DNN1) (e.g., DNN1 206A) and a second deep neural network (DNN2) (e.g., DNN2 206B), a processor 208, and an output interface 210.

[0044] The input interface 202 is configured to receive an acoustic signal mixture including a target direct path signal (e.g., the target direct path signal 106A or the target direct path signal 106B) and a reverberation of the target direct path signal (e.g., the reverberation 108A and / or the reverberation 108B). In some exemplary embodiments, the input interface 202 may be configured to connect to at least a microphone of the device 104 or an array of microphones of the device 104.

[0045] The processor 208 inputs the acoustic signal mixture including the target direct path signal 106A and the reverberation 108A into the DNN 1206A. The DNN 1206A outputs a first estimate of the target direct path signal 106A. In a multi-speaker scenario including a speaker 102A and a speaker 102B generating sound signals in the environment 100A or the environment 100B, the target direct path signals corresponding to each of the speakers 102A and 102B are estimated by the DNN 1206A. The DNN 1206A may determine the corresponding estimates of the target direct path signal for each of the speakers 102A and 102B one by one or simultaneously. For example, the DNN 1206A may determine the first estimate of the target direct path signal 106A for the speaker 102A and the first estimate of the target direct path signal 106B for the speaker 102B simultaneously.

[0046] The first estimate of the target direct path signal 106A is used with the received acoustic signal mixture to estimate a filter that models a room impulse response (RIR) of the first estimate of the target direct path signal 106A. The RIR is the impulse response of a room, such as environment 100A or environment 100B, between the sound source (e.g., speaker 102A and speaker 102B) and the microphones in the device 104. The filter that models the RIR may be output via the output interface 210.

[0047] In some embodiments, a filter that models the RIR of the first estimate of the target direct path signal is estimated such that when the filter is applied to the first estimate of the target direct path signal 106A, the corresponding result is closest to the residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function. In some embodiments, the distance function may correspond to a weighted distance with a weight at each time-frequency point in the time-frequency domain. The weight may be determined by one or a combination of the received acoustic signal mixture and the first estimate of the target direct path signal. In an exemplary embodiment, the distance function may be based on a least squares distance.

[0048] When the result of applying the filter to the first estimate of the target direct path signal 106A is removed from the acoustic signal mixture, a reduced reverberant mixture of the target direct path signal 106A is obtained. The reduced reverberant mixture of the target direct path signal 106A is provided as an input to the DNN2 206B. The DNN2 206B generates a second estimate of the target direct path signal 106A. The second estimate of the target direct path signal 106A is output via the output interface 210.

[0049] Similarly, for speaker 102B, the second estimate of the target direct path signal 106B is used with the received acoustic signal mixture to estimate a filter that models the RIR of the first estimate of the target direct path signal 106B. This filter is applied to the first estimate of the target direct path signal 106B to obtain a corresponding result. This result is removed from the acoustic signal mixture to obtain a reduced reverberation mixture of the target direct path signal 106B. The reduced reverberation mixture of the target direct path signal 106B is input to the DNN2 206B, which generates a second estimate of the target direct path signal 106B.

[0050] A second estimate of the target direct path signal (e.g., a second estimate of the target direct path signal 106A or a second estimate of the target direct path signal 106B) is obtained as the dereverberated speech signal of the corresponding speaker (e.g., speaker 102A or speaker 102B). Dereverberation of the speech signal by the system 200 is described in further detail with reference to FIG.

[0051] 3A is a schematic block diagram of a process 300 for dereverberation of a speech signal according to an embodiment of the present disclosure. The process 300 is performed by the system 200. In an exemplary embodiment, an acoustic signal mixture 302(Y) is received via the input interface 202 of the system 200. The acoustic signal mixture includes a target direct path signal, such as a target direct path signal 106A of a speaker 102A, and reverberation, such as reverberation 108A of the target direct path signal 106A, or a target direct path signal 106B of a speaker 102B, and reverberation 108B of the target direct path signal 106B, along with reverberation of other sources, such as a noise signal 110A of the device 110. The received acoustic signal mixture 302 is input to the DNN 1206A.

[0052] The DNN 1206A determines a first estimate 304 of a target direct path signal, such as the target direct path signal 106A or the target direct path signal 106B. Furthermore, a filter estimate 306 (hereinafter synonymously referred to as filter 306) is determined to model a room impulse response (RIR) 308 of the first estimate 304 of the target direct path signal 106A. The RIR model 308, hereinafter referred to as RIR 308, may correspond to an impulse response of an environment, such as environment 100A or environment 100B, between a source, such as speaker 102A and / or speaker 102B, and a receiver, such as device 104. For that purpose, the absolute delay and absolute attenuation due to the propagation from the source to the microphone are not modeled, but only the relative delay and relative attenuation using the direct path signal as a reference. The impulse response is not considered for the actual dry source signal at the source position, but for the received direct path signal in the mixture as a reference. For simplicity of explanation, the filter estimate 306 is such that application of the filter estimate 306 to the direct path signal includes only the early reflections and late reverberations of the direct path signal, not the direct path signal. By modifying the filter estimate 306 to further include the direct path signal, an associated full filter estimate 307 is equivalently obtained. These two filter estimates are equivalent and one can be obtained directly from the other.

[0053] In some demonstrative embodiments, the acoustic signal mixture 302 may correspond to a mono signal recorded in a noisy reverberant environment, such as environment 100A or environment 100B. Such a mono signal may be constructed into a physical model in a time domain. This physical model represents the relationship between the acoustic signal mixture 302(y), the reverberant target speech signal (x) (including both a target direct path signal, such as target direct path signal 106A, and reverberation, such as reverberation 108A), and non-target sources (v) (e.g., device 110), including a reverberant noise signal (e.g., reverberant noise signal 110A) and a reverberant competing speaker (e.g., speaker 102B).

[0054] The constructed physical model of the acoustic signal mixture 302 can be expressed in the time domain as follows, considering the target direct path signal 106A as the unique target for ease of notation:

number

[0055] The term “r d ", "r e " and "r l " respectively represent the direct, early and late portions of the RIR 308 of the environment 100. The term "s" represents a target direct path signal (such as the target direct path signal 106A), which is expressed as s=a*r d The term “h” represents a non-direct path signal (e.g., reverberation 108A), which is the sum of the early reflections a*r e and late reverberation a*r l and the sum, that is, h=a*r e +a*r l =a*r e+l The portion of RIR308 that corresponds to both the direct path and early reflections, r d+e can be defined as the set of impulses up to 50 ms after the direct path peak, r, where r is the initial reflection component of the RIR. e is r e =r d+e -r d The filter that models the RIR in this application can be defined as r d That is, the time origin of the filter is r d is implicitly taken to be the time of the impulse of rd The height of the impulse can be considered as the standard.

[0056] The above physical model of the acoustic signal mixture 302 can be constructed with a Short-Time Fourier Transform (STFT) to evaluate the time series, i.e., the Fourier transform of the acoustic signal mixture 302 over a short-term window. The physical model of the acoustic signal mixture 302 in the STFT domain is constructed as follows:

number

[0057] The target direct path signal 106A, represented as S(t,f) in equation (2), is estimated using a DNN from the STFT coefficients (Y(t,f)) of the acoustic signal mixture 302. The recovered target direct path signal 106A (S(t,f)) can be used as a first estimate 304 of the target direct path signal 106A.

[0058]

number

[0059] To that end, in some embodiments, the filter 306 is estimated based on convolutional prediction, which corresponds to forward convolutive prediction (FCP) that forward filters the first estimate 304 of the target direct path signal 106A obtained by the DNN 1206A. The forward filtering may estimate the filter 306 by solving the following minimization problem:

number

[0060]

number

[0061]

number

[0062] Using equation (4), the first estimate 304 is filtered to approximate the reverberant target speech X. The dereverberation result is obtained as follows:

number

[0063]

number

[0064] Solving equation (6) also leads to good filter estimates even in the presence of interference from a corresponding target speaker, such as speaker 102A or speaker 102B. Equation (6) can be reconstructed in terms of the reverberant target speech (X) of speaker 102A. For example, the acoustic signal mixture (Y) 302 including the reverberant target speech and interference is defined as follows:

number

[0065]

number

[0066] The estimate of the filter 306 can be improved by calculating a full filter estimate 307 using equation (7) if an estimate of the reverberant target speech X can be obtained. In some embodiments, estimates of the reverberant speech of each speaker are iteratively removed from the acoustic signal mixture 302 to refine the reverberant target signal used to estimate the filter 306.

[0067] In this embodiment, the FCP equation (4) can remove the reverberation associated with the target speaker 102A. The ability to obtain the reverberation of the target speaker 102A can be particularly useful in multi-speaker separation tasks, since each target speaker is convolved with a different RIR. To that end, in some embodiments, a different filter can be calculated to dereverberate each speaker (as illustrated in FIG. 6). The estimated filter, such as filter 306, can focus on reducing the reverberation of the target speaker 102A, rather than the reverberation of another speaker (e.g., speaker 102B) combined with a non-target source (e.g., air conditioner 110). To dereverberate the speech signal even in the presence of non-target sources, the output of the DNN 1206A, such as the first estimate 304 of the target direct path signal 106A, and the reverberant-reduced mixture obtained using filter 306 can be utilized to dereverberate the speech signal. For that purpose, the first estimate 304 and the reverberant-reduced mixture obtained using the filter 306 may be input to the DNN2206B to output a second estimate 314 of the target direct path signal 106A (or the target direct path signal 106B). The output generated by the DNN2206B, such as the second estimate 314, will be better than the output of the DNN1206A because the input to the DNN2206B (i.e., the first estimate 304 and the reverberant-reduced mixture obtained using the filter 306) is more refined than the input of the DNN1206A. For example, there will be less interference between the first estimate 304 and the reverberant-reduced mixture obtained using the filter 306 output by the DNN1206A. When these less-interfering first estimates 304 and the reverberant-reduced mixture obtained using the filter 306 are processed by DNN2 206B, the corresponding output (i.e., second estimate 314) may be better than the output of DNN1 206A (i.e., first estimate 304).Thus, using the second estimate generated by DNN2206B, another iteration of convolutional prediction can be performed to obtain a second filter and a second reverberant-reduced mixture, which can be input to DNN2206B along with the second estimate to generate a refined output.

[0068] In some exemplary embodiments, the corresponding RIR of each speaker, such as RIR 308, may be estimated by solving a linear regression problem for each frequency in the time-frequency domain or the time domain. A filter 306 that models the RIR 308 may be used to identify delayed and attenuated copies of the target direct path signal of speaker 102A and / or speaker 102B. The delayed and attenuated copies, which are repetitive patterns caused by reverberation, may be removed from the received acoustic signal mixture 302. To that end, the filter 306 is applied to the first estimate 304 to output a result 310. The result 310 may be closest to the residual between the acoustic signal mixture 302 and the first estimate 304 of the target direct path signal based on a distance function, such as a weighted least squares distance function. When the result 310 is removed from the acoustic signal mixture 302, a reverberantly reduced mixture 312 is obtained.

[0069]

number

[0070] The removal of the result 310 reduces the delayed and attenuated copies from the reverberant-reduced mixture 312. The delayed and attenuated copies may correspond to late and early reflections of the target direct path signal. These early and late reflections may be identified from the RIR 308 modeled by the filter estimate 306. The RIR 308 with early reflections and late reverberation is shown in FIG. 3B.

[0071] 3B is a diagram illustrating a representation 316 of a room impulse response (RIR) model 316A for an original source of a signal from a speaker, such as speaker 102A, showing impulses corresponding to a target direct path signal 320A, impulses corresponding to early reflections 320B, and impulses corresponding to late reverberations 320C. In this application, instead of the original source of the signal from the speaker, the target direct path signal is taken as a reference. In other words, application of the RIR to the target direct path signal results in a reverberant signal for the speaker that is the sum of the target direct path signal and the early reflections and late reverberations of the target direct path signal.

[0072] 3C is a diagram illustrating application 326 of a filter 316B that models an RIR 316A at frequency bin f, according to an embodiment of the present disclosure. The RIR model 316A corresponds to the RIR model 308, and the first estimate 316B corresponds to the full filter estimate 307 related to the filter estimate 306.

[0073] The RIR model 316A has a structure that can be represented as a sequence of impulses in the time domain. For example, the RIR model 316A is represented as a graph with an amplitude axis 318A and a tap number axis 318B representing time delay. The structure of the RIR model 316A is such that the target direct path signal 320A (r d ) and the late reverberation 320C (r l ) followed by a target direct path signal 320A(r d ) discrete early reflections 320B(r e ) and a number of impulses corresponding to the target direct path signal 320A. The target direct path signal 320A may correspond to the target direct path signal 106A or the target direct path signal 106B.

[0074] In some exemplary embodiments, the early reflections 320B and late reverberations 320C are identified from the RIR model 316A. Given that the filter is modeled using K coefficients at each frequency f, the coefficients of the filter estimate 307 at frequency f are obtained such that application 326 of the filter to the first estimate 304 of the target direct path signal by summing the results of multiplying the time-frequency bins of the first estimate at the same frequency f at time t-k+1 (for all k=1,...,K) by the kth coefficient can best approximate the reverberant mixture 322 at the same frequency f at the current time t.

[0075] As shown in graph 316B, graph 316B represents approximating the acoustic signal mixture 322(Y) by application 326 of a K-tap filter 324 to the first estimate 304 of the target direct path signal. The filter 324 is estimated by optimizing forward filtering of the first estimate 304 of the target direct path signal 302A. The filter 324 is an example of a filter 307. For example, the number of taps K of the filter 324 may be set to 40, which may mean that the filter length in the time domain is ((40-1)x8+32) milliseconds.

[0076] There may be various scenarios for dereverberation of speech signals by the DNN 1206A and the DNN 2206B. For example, the acoustic signal mixture 302 may be received by a single microphone of the device 104 or by an array of microphones of the device 104, from a single speaker (e.g., speaker 102A) or from multiple speakers (e.g., speakers 102A and 102B). In the case of multiple speakers, the first DNN 1206A estimates different first estimates of the target direct path signal for each of the multiple speakers. Dereverberation of speech signals in various scenarios is further described with reference to FIG. 4, FIG. 5 and FIG. 6.

[0077] 4 is a schematic diagram of an architecture representation 400 for dereverberation of a speech signal according to an embodiment of the present disclosure. As shown in FIG. 4, the architecture representation 400 includes a DNN1402, a DNN2406, and a convolution prediction module 404 between the DNN1402 and the DNN2406. The DNN1402 corresponds to the DNN1206A, and the DNN2406 corresponds to the DNN2206B.

[0078]

number

[0079]

number

[0080]

number

[0081] Some embodiments are based on the recognition that the second estimate 410 is better than the first estimate 408 because the DNN 2406 processes a refined acoustic signal mixture, which is a reverberation-reduced mixture 412. The second estimate 410 may be further refined to perform better than the first estimate 408. To that end, the DNN 2406 may be input one or a combination of the acoustic signal mixture 302 and the first estimate 408 to generate the second estimate 410. In some cases, the acoustic signal mixture 302, the first estimate 408 and the reverberation-reduced mixture 412 may be input to the DNN 2406 to generate the second estimate 410. In some other cases, the first estimate 408 and the reverberation-reduced mixture 412 may be input to the DNN 2406 to generate the second estimate 410. Further, the estimation of the filter, obtaining the reverberation-reduced mixture 412, and injecting the reverberation-reduced mixture 412 may be repeated to gradually refine the second estimate 410 of the target direct path signal 106A to improve the dereverberation of the speech signal of the speaker 102A. The iteration may be terminated when a termination condition is met. The termination condition may correspond to a user-defined condition. In this manner, since the second estimate 410 is refined with the reverberation-reduced mixture 412, the second estimate 410 will be superior to the first estimate 408. In some embodiments, the DNN2406 may be trained using the acoustic signal mixture 302, the reverberation-reduced mixture 412, and the first estimate 408 to output the second estimate 410 that improves the dereverberation of the speech signal.

[0082] In some cases of multiple speakers, the received acoustic signal mixture 302 may include speech signals from multiple speakers, such as speaker 102A and speaker 102B. In such cases, the DNN 1402 may generate multiple outputs, such as various first estimates of the target direct path signal, from which various filters that model the corresponding RIRs for the multiple speakers can be derived, as further described with reference to FIG. 5.

[0083] 5 is a schematic diagram illustrating an architecture representation 500 for dereverberation of a speech signal in the case of multiple speakers (e.g., speakers 102A and 102B) according to some embodiments of the present disclosure. As shown in FIG. 5, the architecture representation 500 corresponds to a multi-speaker scenario and includes multiple instances of convolution prediction modules, such as DNN1502, DNN2506, and convolution prediction modules 504A and 504B between DNN1502 and DNN2506. DNN1502 corresponds to DNN1206A, and DNN2506 corresponds to DNN2206B.

[0084]

number

[0085]

number

[0086] The reverberant-reduced mixture 510A and the reverberant-reduced mixture 510B are concatenated and provided as inputs to the DNN2506 to output corresponding second estimates 512A and 512B for the speakers 102A and 102B. In some exemplary embodiments, the DNN2506 may receive the first estimates 508A along with the reverberant-reduced mixture 510A, the first estimates 508B along with the reverberant-reduced mixture 510B, and the acoustic signal mixture 302 to output the second estimates 512A and 512B.

[0087] In some exemplary embodiments, the filters and the reverberant-reduced mixtures 510A and 510B for the first estimates 508A and 508B may be iterated by replacing the first estimates 508A and 508B with the second estimates 512A and 512B to generate corresponding filters and corresponding reverberant-reduced mixtures for each of the multiple speakers 102A and 102B. The iterations are terminated when a user-defined termination condition is met. The termination condition may include a user-defined termination condition, such as, for example, terminating after three iterations.

[0088] In some exemplary embodiments, the reverberant-reduced mixture 510A and the reverberant-reduced mixture 510B may be combined into a tensor. The tensor is a -dimensional data structure that represents all the reverberant-reduced mixtures of the multiple speakers 102A and 102B. The tensor is fed into the DNN2 506 to output corresponding second estimates 512A and 512B for each of the multiple speakers 102A and 102B.

[0089] In some cases, the corresponding second estimates for each of the multiple speakers 102A and 102B may be estimated one by one, which will now be described in FIG.

[0090] 6 is a schematic diagram illustrating an architecture representation 600 for dereverberation of a speech signal in the case of multiple speakers 102A and 102B according to some other embodiments of the present disclosure. As shown in FIG. 6, the architecture representation 600 corresponds to a multi-speaker scenario and includes a DNN1602, multiple instances of a second DNN, such as DNN2606A and DNN2606B, and multiple instances of a convolution prediction module, such as a convolution prediction module 604A and a convolution prediction module 604B, between the DNN1602 and multiple instances of the second DNN, such as DNN2606A and DNN2606B. The DNN1602 corresponds to the DNN1206A, and each of the DNN2606A and DNN2606B corresponds to the DNN2206B.

[0091] The DNN 1602 receives the acoustic signal mixture 302 and estimates a corresponding target direct path signal for each of the multiple speakers 102A and 102B. For example, the DNN 1602 estimates a first estimate 608A of the target direct path signal 106A for the speaker 102A. The DNN 1602 estimates a first estimate 608B of the target direct path signal 106B for the speaker 102B. The first estimate 608A is input to a convolutional prediction module 604A, and the first estimate 608B is input to a convolutional prediction module 604B.

[0092] The convolution prediction module 604A estimates a filter that models the RIR of the first estimate 608A. This filter is applied to the first estimate 608A to obtain a reverberation-reduced mixture 610A of the target direct path signal 106A. Similarly, the convolution prediction module 604B estimates a filter that models the RIR of the first estimate 608B. The estimated filter output by the convolution prediction module 604B is applied to the first estimate 608B to obtain a reverberation-reduced mixture 610B of the target direct path signal 106B.

[0093] In some embodiments, each of the reverberation-reduced mixtures 610A and 610B may be obtained by removing from the received acoustic signal mixture the sum of the results of applying each filter to each of the first estimates of the target direct path signals of each of the multiple speakers 102A and 102B, instead of simply the results of applying the filter to the first estimates of the target direct path signals of the corresponding speaker. Thus, for every speaker c, the reverberation-reduced mixtures may be obtained as follows:

number

[0094] Further, each of the reduced reverberation mixtures is fed into an instance of DNN2. Each of the reduced reverberation mixtures 610A and 610B is fed into a corresponding instance DNN2 606A and DNN2 606B (essentially the same DNN2 but applied to different inputs), respectively. DNN2 606A outputs a second estimate 612A of the target direct path signal 106A for speaker 102A. DNN2 606B outputs a second estimate 612B of the target direct path signal 106B for speaker 102B. Multiple instances of the second DNN, such as DNN2 606A and DNN2 606B, outputting the second estimates 612A and 612B for the corresponding speakers 102A and 102B, respectively, can be used to obtain clear speech of individual speakers from multiple speakers.

[0095] To improve the second estimate 612A and the second estimate 612B, the DNN2 606A and the DNN2 606B may be input with one or a combination of the acoustic signal mixture, the first estimate 608A and the first estimate 608B, and the reverberant reduced mixtures 610A and 610B.

[0096] In some demonstrative embodiments, the first estimate 608A may be replaced with the second estimate 612A to generate an updated first estimate 608A of the target direct path signal 106A. Similarly, the first estimate 608B may be replaced with the second estimate 612B to generate an updated first estimate 608B of the target direct path signal 106B. Further, the estimation of the filters by the DNN 1602, the estimation of the reverberant reduced mixtures 610A and 610B, and the injection of the reverberant reduced mixtures 610A and 610B may be repeated to output an updated second estimate of the target direct path signal for each of the multiple speakers 102A and 102B.

[0097] In some other exemplary embodiments, a portion of the acoustic signal mixture corresponding to a speaker (e.g., speaker 102A) may be extracted. The portion is extracted by removing reverberant speech of other speakers, such as speaker 102B, from the acoustic signal mixture. An estimate of the reverberant speech of the other speaker of the plurality of speakers is obtained by adding a first estimate of the target direct path signal of the other speaker to a result of applying a corresponding filter of the other speaker to the first estimate of the target direct path signal of the other speaker. After extraction of the portion of the acoustic signal corresponding to speaker 102A, a filter for the first estimate of the extracted portion is estimated. The filter is used to estimate a reduced reverberant mixture for speaker 102A based on the portion. Processing of the portion can improve the quality of the estimated filter for the speaker and the quality of the corresponding second estimate.

[0098] In some exemplary embodiments, the acoustic signal mixture of a single speaker 102A and / or multiple speakers 102A and 102B may be received from a single microphone or from an array of microphones. To that end, DNNs such as DNN1602 and DNN2606A and DNN2606B may be trained based on a spectral mapping corresponding to a single microphone and an array of microphones. This spectral mapping trains DNN1602 to predict real and imaginary (RI) components (i.e., frequencies) of an estimate, such as the first estimate 608A of the target direct path signal 106A, from the RI components of the acoustic signal mixture 302. The RI components of the acoustic signal mixture 302 and the RI components of the first estimate 608A may be input to DNN2606A to predict a second estimate of the target direct path signal 106A. The DNN1602 may be pre-trained using a training dataset of acoustic signal mixtures and corresponding reference target direct path signals in the training dataset.

[0099] In some embodiments, the pre-training of the DNN 1602 may be performed by minimizing a loss function. The loss function may include one or a combination of distance functions defined based on the RI components of the target direct path signal 106A in the first time-frequency domain and the RI components of the reference target direct path signal in the first time-frequency domain. The reference target direct path signal may be obtained from a training dataset of speech, and the corresponding reverberant mixture may be obtained by convolving the reference target direct path signal with the recorded or synthetic RIR and summing it with other interference signals. The distance function may be defined based on the magnitude obtained from the RI components of the estimated target direct path signal in the first time-frequency domain and the corresponding magnitude of the reference target direct path signal.

[0100] In an alternative embodiment, the distance function may be defined based on a reconstructed waveform obtained from the RI component of the estimated target direct path signal in the first time-frequency domain by reconstruction in the time domain and the waveform of the reference target direct path signal. The distance function may also be defined based on the RI component in the complex time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and the RI component of the reference target direct path signal in the second time-frequency domain. The distance function may also be defined based on a magnitude obtained from the RI component in the second time-frequency domain obtained by transforming the reconstructed waveform in the second time-frequency domain and the corresponding magnitude of the reference target direct path signal in the second time-frequency domain.

[0101] The loss function for the predicted RI components can be defined as follows:

number

[0102]

number

[0103]

number

[0104] For speaker separation, a loss based on the predicted RI components can be calculated using Permutation-Invariant Training (PIT) on the utterance. A speaker index c ∈ {1, ..., C} of multiple speakers, such as speakers 102A and 102B, is used to distinguish between C speakers S(1), ..., S(C). The loss function is defined as follows:

number

[0105]

number

[0106] In some exemplary embodiments, the acoustic signal mixture 302 may correspond to a multi-channel signal that may be received from an array of microphones. Beamforming is performed on such a multi-channel signal, as will be further described with reference to FIG.

[0107] 7 is a schematic diagram illustrating an architecture representation 700 for improving dereverberation of a speech signal according to some embodiments of the present disclosure. The architecture representation 700 is similar to the architecture representation of FIG. 5, but further includes several instances of a Minimum Variance Distortionless Response (MVDR) beamforming module 704. In some exemplary embodiments, each instance of the MVDR module may output a beamforming output for a multi-channel signal. The beamforming filter may be obtained based on statistics calculated from one or a combination of a first estimate, such as a first estimate 508A (and / or a first estimate 508B) output by a first DNN, such as DNN1 502, a reverberantly reduced mixture 510A (and / or a reverberantly reduced mixture 510B), and a second estimate, such as a second estimate 512A (and / or a second estimate 512B) output by a second DNN, such as DNN2 506, which would have been obtained using the architecture representation of FIG. 5 including only a convolutional prediction module between the two DNNs, or a previous iteration of the architecture representation of FIG. 7 including MVDR beamforming. A beamforming output for a speaker may be obtained by applying a beamforming filter to the reverberantly reduced mixture 510A or the mixture 510B. The MVDR beamforming module may be used between two DNNs, such as DNN1 502 and DNN2 506. The output of the MVDR beamforming module, such as beamforming output 514A (and / or beamforming output 514B), may be used as an input to a second DNN, such as DNN2 506. In some exemplary embodiments, the output of the MVDR beamforming module, such as beamforming output 514A, may be combined with one or a combination of a first estimate, such as first estimate 508A, a reverberation-reduced mixture, such as reverberation-reduced mixture 510A, and mixture 510B.In some exemplary embodiments, the beamforming output for all speakers is combined with the reverberant reduced mixture for all speakers, the first estimates for all speakers, and the mixture and used as input to DNN2506. In some exemplary embodiments, the MVDR beamforming module may output beamforming using MVDR techniques such that signals from multiple channels can be combined to derive a good estimate of the target direct path signal.

[0108] To that end, MVDR beamforming can be applied to the reverberantly reduced mixture to further enhance the dereverberation and separation tasks.

[0109] The MVDR beamformer for obtaining beamforming of a multi-channel signal at a reference microphone q can be calculated in a frequency bin f (omitted here) as follows:

number

[0110]

number

[0111] Furthermore, DNNs such as DNN1602 and DNN2606A can be easily replaced with magnitude or time domain models, and can be easily replaced with more advanced DNN architectures, one such model being further described with reference to Figures 8A, 8B, 8C, and 8D.

[0112] 8A, 8B, 8C, and 8D are schematic diagrams illustrating a network architecture 800 for dereverberation of a speech signal according to some other embodiments of the present disclosure. The network architecture 800 corresponds to a DNN such as DNN1 206A and DNN2 206B.

[0113] The network architecture 800 is a Temporal Convolutional Network (TCN) 806. The TCN 806 includes four layers, each of which has six extended convolution blocks, such as extended convolution block 802A, extended convolution block 802B, extended convolution block 802C, extended convolution block 802D, extended convolution block 802E, and extended convolution block 802F (hereinafter referred to as extended convolution blocks 802A-802F). In each of the extended convolution blocks 802A-802F, one one-dimensional (1D) depthwise separable convolution 804 is used to reduce the number of parameters. For example, each of the extended convolution blocks 802A-802F may include approximately 6.9 million parameters for dereverberation of a speech signal. These large number of parameters can be reduced by the 1D depthwise separable convolution 804.

[0114] Further, the TCN 806 is sandwiched by a U-Net including an encoder 808 and a decoder 810. In each of the encoder 808 and the decoder 810, a DenseNet block is inserted at multiple frequency scales. The DenseNet block is an architecture that trains DNNs such as DNN1602 and DNN2606A using shorter connections between layers of the DNN. For example, the encoder 808 includes a DenseNet block 808A, a DenseNet block 808B, a DenseNet block 808C, a DenseNet block 808D, and a DenseNet block 808E (hereinafter simply referred to as DenseNet blocks 808A-808E) at multiple frequency scales. Similarly, the decoder 810 of U-Net includes DenseNet block 810A, DenseNet block 810B, DenseNet block 810C, DenseNet block 810D, and DenseNet block 810E (hereinafter simply referred to as DenseNet blocks 810A-810E) at multiple frequency scales. U-Net can maintain fine-grained local structure by skip connections and model context information along frequency through downsampling and upsampling. TCN 806 exploits long-term information of the received acoustic signal mixture by using dilated convolution along the time domain. DenseNet blocks 808A-808E enable feature reuse and improve the discriminability of speech signals of multiple speakers 102A and 102B in speaker separation tasks.

[0115] The encoder 808 includes one two-dimensional (2D) convolution 812 and seven convolution blocks, such as convolution block 814A, convolution block 814B, convolution block 814C, convolution block 814D, convolution block 814E, convolution block 814F, and convolution block 814G (hereinafter referred to as convolution blocks 814A-814G). Each of the convolution blocks 814A-814G includes a 2D convolution, an Exponential Linear Unit (ELU) nonlinearity, and an Instance Normalization (IN) for downsampling, i.e., lowering the sampling rate or sample size (bits per sample) of an input signal, such as the acoustic signal mixture 704. The 2D convolution forms an essential component of feature extraction corresponding to an estimate of the target direct path signal. ELU is the activation function for DNNs (e.g., DNN1602 and DNN2606A), and IN is a normalization layer to stabilize hidden state dynamics in DNN1602 and DNN2606A.

[0116] The decoder 810 includes seven blocks of 2D deconvolution, such as Deconvolution 816A, Deconvolution 816B, Deconvolution 816C, Deconvolution 816D, Deconvolution 816E, Deconvolution 816F and Deconvolution 816G (hereinafter referred to as Deconvolutions 816A-816G), along with ELU and IN and one 2D deconvolution 820 for upsampling by adding zero-valued samples between the original samples to increase the sampling rate.

[0117] As described above, the de-reverberated mixtures for multiple speakers 102A and 102B (e.g., de-reverberated mixture 510A and de-reverberated mixture 510B) are represented by tensors. The tensors are of the form featureMapstimeStepsfrequencyChannels. Each of the convolution blocks 814A-814G (i.e., Conv2D+ELU+IN) and de-convolution blocks 816A-816G (i.e., Deconv2D+ELU+IN) are specified in the form kernelSizeTimekernelSizeFreq, (stridesTime,stridesFreq), (paddingsTime,paddingsFreq) and featureMaps.

[0118] Each of the DenseNet blocks 808A-808E, such as DenseBlock(g1,g2), includes five Conv2D+ELU+IN blocks with growth rate g1 for the first four layers and growth rate g2 for the last layer of the DenseNet blocks 808A-808E. The tensor shape after each TCN block 806 is of the form featureMapstimeSteps. Each IN+ELU+Conv1D block is specified in the form kernelSizeTime, stridesTime, paddingsTime, dilationTime, featureMaps.

[0119] 9 is a flow diagram of a method 900 for dereverberation of a speech signal, according to an embodiment of the present disclosure. Method 900 is performed by system 200. At operation 902, method 900 includes receiving, via an input interface, an acoustic signal mixture (e.g., acoustic signal mixture 302) including a target direct path signal (e.g., target direct path signal 106A) and reverberation of the target direct path signal. The acoustic signal mixture may include at least one of a single channel signal or a multi-channel signal, which may be received from a single microphone or an array of microphones connected to the input interface.

[0120] In operation 904, the received acoustic signal mixture is fed into a first DNN, such as DNN 1206, to generate a first estimate (e.g., first estimate 408) of the target direct path signal 106A. In a multi-speaker scenario, the first DNN determines a corresponding first estimate for each of the multiple speakers. The corresponding first estimates may be determined one by one for each of the multiple speakers, or may be determined simultaneously for the multiple speakers. In some embodiments, the first DNN may be pre-trained to generate the first estimate based on at least one of the observed acoustic signal mixtures or a training data set of acoustic signal mixtures and a corresponding reference target direct path signal in the training data set. The pre-training of the first DNN may be performed by minimizing a loss function.

[0121] In operation 906, a filter (e.g., filter 306) that models a room impulse response (RIR) (e.g., RIR model 308) is estimated for the first estimate 408 of the target direct path signal 106A, such that a result of applying the filter to the first estimate of the target direct path signal is closest to a residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function (e.g., a least squares distance function). In some embodiments, the filter corresponds to a linear filter structure estimated based on convolutional prediction. The first estimate is forward filtered by frequency in the time-frequency domain using a linear filter of convolutional prediction (described in Figures 3A, 3B, 4, 5 and 6). In some exemplary embodiments, the received acoustic signal mixture includes speech signals from multiple speakers. The first DNN generates multiple outputs, each output including a first estimate of the target direct path signal for a speaker from the multiple speakers. In some embodiments, early reflections (e.g., early reflections 320B) and late reverberations (e.g., late reverberations 320C) of the first estimate may be identified based on the RIR modeled by the filter. The identified early reflections and late reverberations may be removed from the first estimate to estimate the acoustic signal mixture.

[0122] At operation 908, a reverberant reduced mixture of the target direct path signal 106A is obtained by removing from the received mixture the result of applying the filter to the first estimate 408 of the target direct path signal 106A. In some embodiments, a second DNN may be trained based on a training data set created from augmented data obtained using the set of estimated filters and the set of estimated target direct path signals to create the reverberant mixture.

[0123] In operation 910, the reverberant-reduced mixture is fed into a second DNN (e.g., DNN2206B) to generate a second estimate of the target direct path signal. In some exemplary embodiments, one or a combination of the received acoustic signal mixture and the first estimate of the target direct path signal is fed into the second DNN to generate a second estimate of the target direct path signal. In some other exemplary embodiments, the received acoustic signal mixture, the first estimate of the target direct path signal, and the reverberant-reduced mixture are fed into the second DNN to generate a second estimate of the target direct path signal. In some still other exemplary embodiments, the first estimate of the target direct path signal and the reverberant-reduced mixture are fed into the second DNN to generate a second estimate of the target direct path signal. In some embodiments, the second DNN may be trained based on a training data set created from augmented data obtained using the set of estimated filters and the set of estimated target direct path signals to create the reverberant mixture.

[0124] In operation 912, the second estimate of the target direct path signal is output via an output interface, such as the output interface 210. To further improve the dereverberation of the speech signal, the steps of estimating the filter, obtaining the reverberant-reduced mixture, and injecting the reverberant-reduced mixture may be repeated for each of the multiple outputs of the first DNN. Also, the output interface may be configured to output the RIR modeled by the filter. The output RIR may be used to perform speech analysis for one or a combination of room acoustic parameter analysis, room geometry reconstruction, speech enhancement, and dereverberation of the speech signal.

[0125] In some exemplary embodiments, the dereverberation of a speech signal using the estimates, i.e., the first and second estimates of the target direct path signal and the filter of the target direct path signal, is evaluated for three tasks: 1) speech dereverberation with weak stationary noise, 2) two-speaker separation in reverberant conditions with white noise, and 3) two-speaker separation in reverberant conditions with challenging non-stationary noise. The evaluation results are shown in Figures 10, 11 and 12.

[0126] 10 illustrates a tabular representation 1000 corresponding to a simulated test set for dereverberation of speech signals, according to an embodiment of the present disclosure. The tabular representation 1000 shows the dataset used for dereverberation, reverberant speaker separation and speech enhancement tasks, hyper-parameter settings, and a baseline system for dereverberation of speech signals. The tabular representation 1000 also shows results for an ASR task on the REVERB corpus.

[0127] For dereverberation of speech signals, DNNs, such as DNN1206A and DNN2206B, can be trained using simulated reverberant datasets with weak air-conditioning noise. In addition to evaluating the trained DNNs on simulated test sets, the DNNs are directly applied to the Reverberant Voice Enhancement and Recognition Benchmark (REVERB) corpus to show their effectiveness in processing real recorded noisy reverberant speech. The REVERB corpus is a benchmark for the evaluation of automatic speech recognition techniques. The dataset also includes clean signals for the simulations taken from the WSJCAM0 corpus. The WSJCAM0 corpus contains 7,861 utterances, 742 utterances, and 1,088 utterances in its training, validation, and test sets, respectively. Using these utterances in the WSJCAM0 corpus, we simulate 39,305 (7,861 x 5) noisy reverberant mixtures, 2,968 (742 x 4) noisy reverberant mixtures, and 3,264 (1,088 x 3) noisy reverberant mixtures as the training, validation, and test sets, respectively. A data spatialization process is then performed to randomly sample rooms for each utterance with random room features as well as speaker and microphone positions using the estimated RIR for dereverberation of the speech signal. The distance between the speaker and the microphone is sampled from the range [0.75, 2.5] m. The reverberation time (T60) is derived from the range [0.2, 1.3] seconds. For each utterance, diffuse air-conditioned noise is sampled from the REVERB corpus and added to the speaker's reverberant speech. The signal-to-noise ratio between anechoic speech and noise is sampled from the range [5, 25] dB. The sampling rate is 16kHz.

[0128] The trained model is applied to realistic reverberation recordings without retraining and to the ASR task of REVERB. The test mixtures are taken from real recordings recorded in a room (e.g., environment 100) with a reverberation time T60 of approximately 0.7 seconds and a distance between the speaker and the microphone of approximately 1 m for the near-field case and 2.5 m for the far-field case. The recorded noise is diffuse air-conditioning noise and weak.

[0129] The official REVERB corpus is used in software such as Kaldi to build a backend for ASR that is trained using noisy reverberant speech and REVERB clean source signals. In an exemplary embodiment, a plug-and-play approach is then implemented for ASR, where the enhanced time-domain signal is directly input to the backend for decoding.

[0130] For the reverberant speaker separation task, the 6-channel Spatialized Multi-Speaker Wall Street Journal (SMS-WSJ) dataset is used. The SMS-WSJ dataset contains simulated two-speaker mixtures in reverberant conditions. Clean speech is sampled from the WSJ0 and WSJ1 datasets. The corpus contains 33,561 two-speaker mixtures, 982 two-speaker mixtures, and 1,332 two-speaker mixtures for training, validation, and testing, respectively. The distance between the speaker and the array is sampled from the range [1.0, 2.0] m, and T60 is derived from the range [0.2, 0.5] s. Weak white noise is added to simulate microphone noise. The energy level between the sum of the reverberant target speech signal and the noise is sampled from the range [20, 30] dB. The sampling rate is 8 kHz. The first channel of the 6-channel SMS-WSJ dataset is used for training and evaluation. Furthermore, the direct sound as a training target is used to perform both the dereverberation and separation tasks.

[0131] For ASR, the default Kaldi-based backend acoustic model defined in the SMS-WSJ dataset is used, and the model is trained using single speaker noisy reverberant speech as input and the state alignment of its corresponding direct path signal as labels. Signals in the first, third and fifth channels (i.e. more than microphones) are used to train the acoustic model. Task-standard trigram language models are used for decoding.

[0132] The noisy reverberant speaker separation task is evaluated using the Noisy Reverberant WSJ0 Hipster Ambient Mixture (WHAMR!) dataset. WHAMR! pairs the two-speaker mixtures in the wsj0-2mix dataset with the noisy background scenes used for noisy reverberant binaural two-speaker separation. In this evaluation, the clean two-speaker mixtures are reused in the WSJ0-2mix dataset and each clean signal is reverberated and supplemented with non-stationary environmental noise recorded in WHAM!. The reverberation time T60 is randomly sampled from the range [0.2, 1.0] seconds. The signal-to-noise ratio between the louder speaker and the noise is derived from the range [-6, 3] dB. The energy levels between the two speakers in each mixture are sampled from the range [-5, 5] dB. The distance between the speaker and the array is sampled from the range [0.66, 2.0] m. The training, validation and test sets contain 20,000, 5,000 and 3,000 binaural mixtures, respectively. The corpus used is the 1 min and 8 kHz version.

[0133] For the STFT, the window length is 32 ms, the hop size is 8 ms, and the analysis window is the square root of the Hann window. When the sampling rate is 16 kHz, a 512-point FFT is applied to extract 257-dimensional STFT features, and when the sampling rate is 8 kHz, a 256-point FFT is used to extract 129-dimensional features. No sentence-level or global-level mean-variance normalization is performed on the input features. For each mixture, its sample variance is normalized to 1 before any processing. During training, the target signal needs to be scaled by a factor identical to the factor used to scale the mixtures.

[0134] For WPE and DNN-WPE, the number of filter taps K is set to 37 and the filter delay Δ is set to 3. The number of iterations in WPE is set to 3. No PSD context is used. Based on the validation set, K and Δ are adjusted to 40 and 0, 39 and 1, 38 and 2, 37 and 3, and 36 and 4, of which setting the filter taps and filter delay to 37 and 3 worked best across the entire dataset. For convolutional prediction, K is set to 40, which results in an amount of context identical to that in WPE. This means that the filter length in the time domain is 344 (=(40-1)×8+32) ms. The filter taps K are increased to 125, which corresponds to an RIR length of up to 1.0 s. This incurs an increase in the amount of computation spent on the linear regression step, but does not make a significant difference in terms of the evaluation score. The RIR has energy mostly within 0.35 s after the peak impulse. The floor value ε used in computing the dereverberation results is set to 1.0, indicating that no weighting is used, or to 0.001. The PSD at each TF unit will be −30 dB lower than the TF unit with the highest energy.

[0135] For all tasks, the main evaluation metric is the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR). SI-SDR measures the quality of the time-domain sample-level prediction. The extended Short-Time Objective Intelligibility (eSTOI) and Perceptual Evaluation of Speech Quality (PESQ) scores are measured. For PESQ, narrowband MOS-LQO scores are reported based on the ITU P.862.1 standard using the python-pesq toolkit. The basis for the metric calculation is used from the target direct path signal, which is obtained by setting the reverberation time T60 parameter to zero in the RIR. The ASR Word Error Rate (WER) is also shown in the tabular representation 1000.

[0136] In the tabular representation 1000, the target direct path signal is represented by "d", the target direct path signal with early reflections is represented by "d+e", and the target direct path signal with early reflections and noise is represented by "d+e+v".

[0137] As shown in the tabular representation 1000, when the first estimate of the first DNN (DNN1) is considered to be the final prediction, the training target of DNN1 performs better than the other two (i.e., "d+e" and "d+e+v"). Compared to using various ones as the training target of DNN1, there is no significant difference in DNN1-WPE, which applies the DNN1 output to improve the WPE. However, it can be seen that training DNN1 using the target direct path signal shows an improvement in performance in DNN1+DNN2, which stacks two DNNs using the acoustic signal mixture and the output of DNN1, i.e., DNN1+DNN2, which trains the second DNN2 using the first estimate of the target direct path signal.

[0138] Also, the tabular representation 1000 includes a comparison of using the Inverse Convolutive Prediction (ICP), Forward Convolutive Prediction (FCP) or Weighted Prediction Error (WPE) methods between two DNNs, namely DNN1 and DNN2, where DNN1+FCP+DNN2 with a floor value ε set to 0.001 performs better than DNN1+WPE+DNN2 and DNN1+ICP+DNN2. As shown in the tabular representation 1000, by performing linear or convolutional prediction and DNN2 for one or more iterations at run time, DNN1+(WPE+DNN2)×2 and DNN1+(ICP+DNN2)×2 show slight improvements in terms of SI-SDR and PESQ, and slight degradation in terms of word error rate (WER). Meanwhile, DNN1+(FCP+DNN2)×2 shows improvements in terms of all metrics. These results indicate that the DNN1+FCP+DNN2 approach is more effective than WPE and DNN1+WPE+DNN2.

[0139] In DNN1+ICP+DNN2, the SI-SDR and PESQ scores were improved by setting the floor value ε to 1.0. When the floor value is set to 0.001, the SI-SDR and PESQ scores in DNN1+FCP+DNN2 are further improved. For example, when the floor value is 1.0, the SI-SDR score is 11.9 and the PESQ score is 3.15. When the floor value is 0.001, the SI-SDR score is 12.3 and the PESQ score is 3.18. The floor values ​​of 1.0 and 0.001 are also used to evaluate the trained DNN1 using ICP and FCP. As shown in the tabular representation 1000, for DNN1+ICP with a floor value of 1.0, the SI-SDR score is 3.2 and the PESQ score is 1.78, for DNN1+ICP with a floor value of 0.001, the SI-SDR score is 0.7 and the PESQ score is 1.77, for DNN1+FCP with a floor value of 1.0, the SI-SDR score is 3.6 and the PESQ score is 1.82, and for DNN1+FCP with a floor value of 0.001, the SI-SDR score is 3.0 and the PESQ score is 1.82. Thus, DNN1+FCP+DNN2 shows a better score than training DNN1 using the ICP and FCP methods.

[0140] Overall, for speech dereverberation, the mixture SI-SDR and PESQ improve from -3.6 dB and 1.64 to 8.2 dB and 2.65 by using one DNN (i.e., DNN1), to 9.1 dB and 2.82 by using two DNNs (i.e., DNN1+DNN2), to 12.3 dB and 3.18 by adding an FCP module between the two DNNs (DNN1+FCP+DNN2), and to 12.8 dB and 3.24 by using one additional iteration for FCP and DNN2 (DNN1+(FCP+DNN2)×2).

[0141] Finally, a magnitude-domain loss is added during training of the second DNN2, which gives improvements in terms of word error rate (WER) and PESQ, but degrades the SI-SDR by approximately 0.5 dB.

[0142] 11 illustrates a tabular representation 1100 of evaluation results for dereverberation of speech signals using a test dataset, according to an embodiment of the present disclosure. The evaluation results show performance on the SMS-WSJ dataset and oracle results obtained by using a target direct path signal with or without early reflections and oracle masks such as a spectral magnitude mask (|S| / |Y|) and a phase sensitive mask (|S| / |Y|cos(∠S-∠Y)). As shown in the tabular representation 1100, using the oracle target direct path signal for ASR results in a better WER than using a target direct path signal with early reflections (6.4% vs. 7.04%), indicating the potential benefit of removing early reflections.

[0143]

number

[0144] In DNN-WPE, two variants for multi-speaker scenarios are used. The first variant uses the PSD of each estimated target speaker generated by DNN1 to calculate a different WPE for each speaker. In the tabular representation 1100, DNN-WPE in a multi-speaker scenario is represented as DNN1+mfWPE+DNN2, where "mf" indicates multi-filter. Multi-filter sums all estimated target speakers provided by DNN1 and uses the PSD of the summed signal to calculate a single WPE filter to dereverberate the mixture. The second variant is represented as DNN1+sfWPE+DNN2, where "sf" indicates single filter.

[0145] As shown in the tabular representation 1100, DNN1+sfWPE+DNN2 performs slightly better than DNN1+mfWPE+DNN2, suggesting that it is not effective for WPE to compute separate filters for each target speaker.

[0146] The scenario where all speakers are contributing speech signals is represented as "allSpks" in the tabular representation 1100, and DNN2 is trained to simultaneously emphasize all target speakers. As shown in the tabular representation 1100, compared with DNN1+sfWPE+DNN2 and DNN1+ICP+DNN2, DNN1+FCP+DNN2 shows better performance in all metrics. This proves that forward filtering of convolutional prediction (as described in Figures 5 and 6) is more effective than WPE in dereverberation in the presence of competing speakers.

[0147] Further improvement is realized when DNN2 is trained to enhance the target speakers one by one as described in FIG. 6 (represented by “perSpk” in tabular representation 1100). This suggests that dereverberating each speaker individually can improve the speaker's speech enhancement. As shown in tabular representation 1100, steady improvement can be achieved by repeating the convolutional prediction and DNN2 for one or more iterations. Also, DNN2 trained by including the magnitude level loss improves PESQ, eSTOI and WER, but SI-SDR degrades.

[0148] In the tabular representation 1100, it is further shown that the SI-SDR score, PESQ score, eSTOI score and WER score are 12.2, 3.24, 89.0 and 12.77 for DNN1+(FCP+DNN2)×2 trained with the magnitude-level loss function. The DNN1+(FCP+DNN2)×2 trained with the magnitude-level loss function can perform better than the DNN1+(FCP+DNN2)×2 trained with another complex spectrum mapping, a spectral mapping corresponding to a single microphone, such as single-input single-output microphone (SISO1) (SI-SDR 12.5 dB vs. 5.1 dB). The DNN1+(FCP+DNN2)×2 trained with the magnitude-level loss function can perform better than the DNN1+(FCP+DNN2)×2 trained with DPRNN-TasNet (SI-SDR 12.5 dB vs. 6.5 dB).

[0149] Tabular representation 1100 also shows the performance of DNN1 and DNN2 trained based on a spectral mapping corresponding to an array of microphones, such as a 6-microphone SISO with beamforming of the array of microphones (SISO1-BF-SISO2), which combines mono complex spectral mapping with beamforming and post-filtering. These results suggest that combining end-to-end DNNs with convolutional prediction can be effective in reducing reverberation in an acoustic signal mixture that includes the speech signals of a speaker (e.g., speakers 102A and 102B).

[0150] FIG. 12 illustrates a tabular representation 1200 showing evaluation results for dereverberation of speech signals using a test dataset according to some other embodiments of the present disclosure. The tabular representation 1200 shows the SI-SDR for the WHAMR! dataset. As shown in the tabular representation 1200, DNN1+FCP+DNN2 produces better results than DNN1+mfWPE+DNN2 (SI-SDR 7.4 dB vs. 6.8 dB). This indicates that DNN-FCP may be more robust than DNN-WPE in dereverberation in the presence of noise and competing speakers.

[0151] The tabular representation 1200 also shows a comparison with end-to-end speech separation systems such as Wavesplit. DNN1+(FCP+DNN2)×2 has a SI-SDR score of 7.5 dB, which is higher than that of Wavesplit, i.e., 5.9 dB. Wavesplit may use speaker identity as side information during training for target speaker extraction. DNN1+(FCP+DNN2)×2 does not rely on the availability of speaker identity information. Dynamic mixing may also be applied for data augmentation, which results in a better SI-SDR (7.1 dB). DNN1+(FCP+DNN2)×2 may be trained without data augmentation, which performs better than Wavesplit with dynamic mixing.

[0152] 13 is a block diagram of an audio signal processing system 1300 according to an embodiment of the present disclosure. The audio signal processing system 1300 uses the system 200. In some exemplary embodiments, the system 200 having a DNN for dereverberation of a speech signal, such as the DNN 1206A and the DNN 2206B, may be implemented on a remote server or in a cloud network. In some embodiments, the audio signal processing system 1300 (hereinafter referred to as the system 1300) may receive an RIR model, such as the RIR model 316A, to the audio signal processing system 1300. The system 1300 may process the RIR model to perform audio analysis for at least one or a combination of room geometry reconstruction, speech enhancement, and dereverberation of a speech signal.

[0153] In some demonstrative embodiments, system 1300 includes one or more sensors 1302, such as acoustic sensors, that collect data including an acoustic signal 1304 from an environment 1306. Environment 1306 corresponds to environment 100.

[0154] The acoustic signal 1304 may include one or more target direct path signals and their reverberations. For example, the acoustic signal 1304 may include multiple speakers with overlapping speech and their reverberations. Additionally, the sensor 1302 may convert an acoustic input into the acoustic signal 1304.

[0155] The audio signal processing system 1300 includes a hardware processor 1308 in communication with a computer storage memory, such as memory 1310. The memory 1310 includes stored data, including algorithms, instructions, and other data that may be executed by the hardware processor 1308. It is contemplated that the hardware processor 1308 may include two or more hardware processors depending on the requirements of a particular application. These two or more hardware processors may be either internal or external. The audio signal processing system 1300 may be incorporated into other components, including output interfaces and transceivers, among other devices.

[0156] In some alternative embodiments, the hardware processor 1308 may be connected to a network 1312 that communicates with one or more data sources 1314, a computer device 1316, a mobile phone device 1318, and a storage device 1320. The network 1312 may include, by way of non-limiting example, one or more local area networks (LANs) and / or wide area networks (WANs). The network 1312 may also include an enterprise-wide computer network, an intranet, and the Internet. The audio signal processing system 1300 may include one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources may include a single device or may include multiple devices that cooperate in a distributed environment of the network 1312.

[0157] In some other alternative embodiments, the hardware processor 1308 may be connected to a network-enabled server 1322 that is connected to the client device 1324. The hardware processor 1308 may be connected to an external memory device 1326 and a transmitter 1328. Additionally, an output may be output for each target speaker according to a particular user's intended use 1330. For example, a particular user's intended use 1330 may correspond to displaying the speech as text (e.g., speech commands) on one or more display devices such as a monitor or screen, or inputting the text for each target speaker into a computer-related device for further analysis, or the like.

[0158] The data source 1314 may include data resources for training DNNs, such as DNN 1206A and DNN 2206B, for a speech separation task. For example, in one embodiment, the training data may include acoustic signals of multiple speakers, such as speaker 102A and speaker 102B, speaking simultaneously. The training data may also include acoustic signals of a single speaker speaking alone, acoustic signals of single or multiple speakers speaking in a noisy environment, and acoustic signals of a noisy environment (e.g., environment 100 with reverberant noise signal 110A).

[0159] The data source 1314 may also include data resources for training the DNN 1206A and the DNN 2206B for a speech recognition task. The data provided by the data source 1314 may include labeled and unlabeled data, such as transcribed and non-transcribed data. For example, in one embodiment, the data may include one or more sounds and may also include corresponding transcription information or labels that may be used to initialize a speech recognition task.

[0160] Additionally, the unlabeled data in the data sources 1314 may be provided by one or more feedback loops. For example, usage data from spoken search queries executed on a search engine may be provided as non-transcribed data. Other examples of data sources may include, by way of example and not limitation, various oral language audio or image sources including streaming sound or video, web queries, mobile device camera or audio information, webcam feeds, smart glasses and smartwatch feeds, customer care systems, security camera feeds, web documents, catalogs, user feeds, SMS logs, instant messaging logs, spoken word transcripts, gaming system user interactions such as voice commands or captured images (e.g., depth camera images), tweets, chat or video call recordings, or social networking media. The particular data source 1314 used may be determined based on the application, including whether the data is a particular class of data (e.g., data related only to a particular type of sound, including machine systems, entertainment systems) or general (non-class specific) in nature.

[0161] The audio signal processing system 1300 may also include a third party device, which may be comprised of any type of computing device, such as an automatic speech recognition (ASR) system on a computing device. For example, the third party device may include a computer device or a mobile device 1318. The mobile device 1318 may include a personal data assistant (PDA), a smartphone, a smart watch, smart glasses (or other wearable smart devices), an augmented reality headset, a virtual reality headset, a laptop, a tablet, a remote control device, an entertainment system, a vehicle computer system, an embedded system controller, an appliance, a home computer system, a security system, a consumer electronic device, or other similar electronic device. The mobile device 1318 may also include a microphone or line input terminal for receiving audio information, a camera for receiving video or image information, or a communication component (e.g., Wi-Fi capability) for receiving such information from another source, such as the Internet or a data source 1314. In one exemplary embodiment, the mobile device 1318 may be capable of receiving input data, such as audio information and image information. For example, the input data may include a speaker's query into a microphone of the mobile device 1318 while multiple speakers in a room are speaking. To determine the query content, the input data may be processed by the ASR in the mobile device 1318 using the system 200. The system 200 enhances the input data by reducing noise in the speaker's environment, separating the speaker from other speakers, or enhancing the audio signal of the query to enable the ASR to output an accurate response to the query.

[0162] In some demonstrative embodiments, storage 1320 may store data, computer instructions (e.g., software program instructions, routines or services), and / or information including data related to DNNs, such as DNN 1206A and DNN 2206B of system 200. For example, storage 1320 may store data from one or more data sources 1314, one or more deep neural network models, information for generating and training the deep neural network models, and computer usable information output by one or more deep neural network models.

[0163] 14A is a block diagram of a system 1400A for dereverberation of a speech signal, according to some exemplary embodiments of the present disclosure. The system 1400 can be used to estimate a target speech signal from an input speech signal 1402 obtained from a sensor 1404 monitoring an environment 1406.

[0164] The input audio signal 1402 includes an acoustic signal mixture including a target direct path signal (e.g., target direct path signal 106A) and a corresponding reverberation (e.g., reverberation 108A). The system 1400A processes the audio signal 1402 via the processor 1408 using a feature extraction module 1410. The feature extraction module 1410 calculates an audio feature sequence from the input audio signal 1402. The first target direct path signal estimation module 1412 processes the audio feature sequence to output a first estimate (e.g., first estimate 408 of the target direct path signal 106A). The first estimate of the target direct path signal is processed by a filter estimation module 1414 to output a filter that models the room impulse response that affects the target direct path signal. For example, the target direct path signal may be influenced to change into a target reverberation signal. A filter is applied to the first estimate to output a reverberation-reduced mixture. The filter and the first estimate are further processed by a target direct path reverberation reduced mixture estimation module 1416, which estimates a target direct path reverberation reduced mixture. The target direct path reverberation reduced mixture, the first estimate and the features are further processed by a second target direct path estimation module 1418 to calculate a signal estimate 1424 (e.g., the second estimate 410) of the target direct path signal. The signal estimate 1424 is output via an output interface 1422. In some embodiments, a room impulse response modeled by the filter may be output via the output interface 1422. The output room impulse response can be used in a speech analysis application to perform one or a combination of room geometry reconstruction, speech enhancement, and dereverberation of a speech signal.

[0165] In some demonstrative embodiments, network parameters 1420 may be input to the first target direct path signal estimation module 1412, the filter estimation module 1414, the target direct path reverberation reduction mixture estimation module 1416 and the second target direct path estimation module 1418. The network parameters 1420 may include labeled and unlabeled data, such as transcribed and non-transcribed data for various sounds or utterances that may be used to initialize a speech recognition task.

[0166] FIG. 14B is a block diagram of a system 1400B for dereverberation of a speech signal according to some other exemplary embodiments of the present disclosure.

[0167] The system 1400B includes a processor 1426 configured to execute stored instructions and a memory 1428 storing instructions for a neural network 1430 including a speech separation with reverberation reduction network 1432 that enables speech separation and reverberation reduction to be achieved. The processor 1426 may be a single-core processor, a multi-core processor, a graphic processing unit (GPU), a computing cluster, or any number of other configurations. The memory / storage 1428 may include a random access memory (RAM), a read only memory (ROM), a flash memory, or other suitable memory system. Also, the memory 1428 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. The processor 1426 is connected to one or more input and output interfaces / devices via a bus 1434. Additionally, the system 1400B may include one or more microphones 1438 connected via the bus 1434. The system 1400B is configured to receive / acquire a speech signal 1456 via one or more microphones 1438 or via a network interface 1452 and a network 1454 connected to a data source of the speech signal 1456.

[0168] The memory 1428 stores a neural network 1430 trained to convert an acoustic signal mixture including a speech signal mixture and corresponding reverberation into a separated speech signal with reduced reverberation. The processor 1426 executing the stored instructions performs speech separation using the neural network 1430 retrieved from the memory 1428. The neural network 1430 is trained to convert an acoustic signal including a speech signal mixture into a separated speech signal. The neural network 1430 may include a speech separation network 1432 trained to estimate the separated signal from acoustic features of the acoustic signal.

[0169] 15 illustrates a use case 1500 for dereverberation of a speech signal, according to some exemplary embodiments of the present disclosure. The use case 1500 corresponds to a teleconference room including a group of speakers (group of speakers 1502A-1502F), such as speaker 1502A, speaker 1502B, speaker 1502C, speaker 1502D, speaker 1502E, and speaker 1502F. Speech signals of one or more speakers of the group of speakers 1502A-1502F are received by an audio receiver 1506 of a device 1504. The audio receiver 1506 comprises the system 200 and receives acoustic speech signals of a speaker or one or more speakers from the group of speakers 1502A-1502F.

[0170] The audio receiver 1506 may include a single microphone and / or an array of microphones for receiving acoustic signal mixtures and noise signals from the group of speakers 1502A-1502F in the teleconference room. These acoustic signal mixtures from the group of speakers 1502A-1502F may be processed using the system 200. For example, the system 200 may analyze an RIR model of the teleconference room. The RIR model may be used to generate a room geometry structure of the teleconference room. The room geometry structure may be used to position reflection boundaries in the teleconference room. For example, the corresponding room geometry structure may be used to determine the placement of speakers, seating arrangements of the group of speakers 1502A-1502F, etc., to cancel noise and other disturbances in the teleconference room. Furthermore, the RIR model may be used to remove reflections and reverberation of speech signals of one or more speakers of the group of speakers 1502A-1502F.

[0171] In the illustrated exemplary scenario, multiple speakers in the group of speakers 1502A-1502F may output speech signals simultaneously. In such a scenario, the system 200 reduces reverberation in the teleconference room and separates the speech signals of each of the speakers 1502A-1502F. The system 200 may also perform beamforming of the acoustic signal mixture from the array of microphones to enhance the speech signal of a corresponding speaker in the group of speakers 1502A-1502F. The enhanced speech signal may be used for transcription of the speaker's utterance. For example, the device 1504 may include an ASR module. The ASR module may receive the enhanced speech signal and output a transcription. The transcription may be displayed by a display screen of the device 1504.

[0172] 16 illustrates a use case 1600 for dereverberation of a speech signal, according to some other exemplary embodiments of the present disclosure. Use case 1600 corresponds to a factory floor including one or more speakers, such as speaker 1602A and speaker 1602B. The factory floor may have high reverberation signals and noise due to the operation of various industrial machines. The factory floor may also include audio devices 1604 to facilitate communication between a factory floor control operator (not shown) and the one or more speakers 1602A and 1602B within the factory floor. The audio devices 1604 may include system 200.

[0173] In the illustrated example scenario, the audio device 1604 may be transmitting a voice command that may be addressed to a person 1602A managing a factory floor. The voice command may include, "Report the status of machine 1." The speaker 1602A may utter, "Machine 1 running." However, the speech signal of the speaker's 1602A utterance may be mixed with noise from the machine, noise from the background, and other utterances from speaker 1602B in the background.

[0174] Such noise and reverberation signals can be mitigated by the system 200. The system 200 outputs clean speech of the speaker 1602A. The clean speech is input to the audio device 1604. The audio device 1604 receives the clean speech and captures responses to voice commands from the clean speech corresponding to the utterances of the speaker 1602A. The system 200 enables the audio device to achieve improved communication with an intended speaker, such as the speaker 1602A.

[0175] 17 illustrates a use case 1700 for dereverberation of a speech signal, according to some further exemplary embodiments of the present disclosure. The use case 1700 corresponds to a driver assistance system 1702. The driver assistance system 1702 is implemented in a vehicle, such as a manually operated vehicle, an automated vehicle, or a semi-automated vehicle. The vehicle is occupied by one or more people, such as person 1704A and person 1704B. The driver assistance system 1702 comprises the system 200. For example, the driver assistance system 1702 may be remotely connected to the system 1702 via a network, such as the network 1454. In some alternative exemplary embodiments, the system 200 may be incorporated within the driver assistance system 1702.

[0176] Also, the driver assistance system 1702 may include one microphone or multiple microphones to receive an audio signal mixture. This audio signal mixture may include speech signals from the persons 1704A and 1704B and external noise signals, such as the horns of other vehicles. In some cases, when the person 1704A is sending a speech command to the driver assistance system 1702, the other person 1704B may speak louder than the person 1704A. The speech from the person 1704B may interfere with the speech command of the person 1704A. For example, the speech command of the person 1704A may be "find the nearest parking lot" and the speech of the person 1704B may be "find a shopping mall to park in." In such a case, the system 200 processes the speech of each of the persons 1704A and 1704B simultaneously or separately. The system 200 separates the speech of the person 1704A from the speech of the person 1704B. The separated utterances are used by the driver assistance system 1702. The driver assistance system 1702 may process and execute the speech commands of person 1704A and the utterances of person 1704B, and output responses to each utterance accordingly.

[0177] 18 illustrates a use case 1800 for dereverberation of a speech signal, according to some further exemplary embodiments of the present disclosure. In some exemplary embodiments, the system 200 (shown in FIG. 2) may process pre-recorded data or live recordings of sounds to determine an estimate of a target direct path signal. The pre-recorded data of sounds may be accessed from a database via a network 1808. The network 1808 is an example of the network 1312. Similarly, live recordings of sources may be streamed from corresponding sources at remote locations via the network 1808.

[0178] The estimate of the target direct path signal may be filtered by the system 200 to determine a RIR model. The RIR model may be analyzed by an audio signal processing system, such as the audio signal processing system 1300, connected to the system 200. The audio signal processing system 1300 may process the RIR model for a room acoustics simulation 1802 of an environment, such as a music concert hall 1806. The RIR model may be convolved with a recorded soundtrack source to imprint the acoustics of the music concert hall 1806 based on the room acoustics simulation 1802. The room acoustics simulation 1802 may be used to create a simulated or virtual reality environment of the actual music concert hall 1806. The simulated environment of the music concert hall 1806 may allow musicians to rehearse before actually performing in the music concert hall 1806.

[0179] In some cases, the room acoustic simulation 1802 can be used to model room acoustic behavior for room geometry reconstruction 1804. The room geometry reconstruction 1804 can provide architectural aspects to the design and construction to maximize the listening experience of an audience in a music concert hall, such as music concert hall 1806.

[0180] By incorporating operations 902-912 in the manner described above, the method 900 performed using the processor 208 located in the system 200 can improve dereverberation of speech signals, since it can enable estimating a filter that includes both the magnitude and phase of the reverberation. The filter is estimated based on a convolutional prediction approach, which allows the filter to reduce early reflections of the target direct path signal. Furthermore, the filter can improve the accuracy of the estimation of the reverberation, since it models the signal propagation in the room, i.e., the RIR. The use of two DNNs in the system 200 can also improve the performance of dereverberation of speech signals, as well as tasks such as speech enhancement and speaker separation. More specifically, the first DNN estimates a first estimate of the target direct path signal from the acoustic signal mixture including the reverberation. The second DNN estimates a refined estimate of the target direct path signal using the first estimate together with other data, such as the filter and the reduction in reverberation estimated by the filter. In this way, the two DNNs allow for identifying and distinguishing the target direct path signal from high reverberation and noise in an efficient and feasible manner.

[0181] Also, particular embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. Also, the order of these operations may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in the diagram. Moreover, not all operations in any process specifically described are performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to a return of the function to a calling function or to a main function.

[0182] Furthermore, embodiments of the disclosed subject matter may be implemented at least partially manually or automatically. The manual or automatic implementation may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0183] The above-described embodiments of the present disclosure can be implemented in any of a number of ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether these processors are located in one computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit having one or more processors as integrated circuit components. However, the processor may be implemented using any suitable form of circuitry.

[0184] Also, the various methods or processes outlined herein may be coded as software executable on one or more processors utilizing any one of a variety of operating systems or platforms. Moreover, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0185] Also, embodiments of the present disclosure may be embodied as a method, an example of which is provided. The operations performed as part of this method may be ordered in any suitable manner. Thus, embodiments may be constructed in which operations are performed in an order different from that shown, which may include performing some operations simultaneously even though they are shown as sequential operations in the example embodiment. It is therefore the object of the appended claims to cover all such variations and modifications that come within the true spirit and scope of the present disclosure.

[0186] Although the disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the disclosure. It is therefore the aspect of the appended claims to cover all such variations and modifications that come within the true spirit and scope of the disclosure.

Claims

1. 1. A computer-implemented method for dereverberating a speech signal, comprising: receiving via an input interface an acoustic signal mixture comprising a target direct path signal and a reverberation of the target direct path signal; feeding the received acoustic signal mixture to a first deep neural network (DNN) to generate a first estimate of the target direct path signal; and estimating a filter that models a Room Impulse Response (RIR) of the first estimate of the target direct path signal, the filter, when applied to the first estimate of the target direct path signal, produces a result that is closest to a residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function, the method further comprising: removing the result of applying the filter to the first estimate of the target direct path signal from the received acoustic signal mixture to obtain a reverberant-reduced mixture of the target direct path signal; feeding the dereverberated mixture to a second DNN to generate a second estimate of the target direct path signal; and outputting the second estimate of the target direct path signal via an output interface.

2. The method of claim 1 , wherein estimating the filter comprises estimating a linear filter based on convolutional prediction.

3. 2. The method of claim 1, further comprising feeding one or a combination of the received acoustic signal mixture and the first estimate of the target direct path signal into the second DNN to generate the second estimate of the target direct path signal.

4. 2. The method of claim 1 , wherein the received acoustic signal mixture includes speech signals from a plurality of speakers, and the first DNN generates a plurality of outputs, each output of the plurality of outputs including the first estimate of the target direct path signal for a speaker from the plurality of speakers.

5. 5. The method of claim 4, further comprising repeating the steps of estimating the filter, obtaining the reverberant reduced mixture, and injecting the reverberant reduced mixture for each of the multiple outputs of the first DNN.

6. repeating the steps of estimating the filter and obtaining the reverberant reduced mixture for each of the plurality of speakers to generate a corresponding filter and a corresponding reverberant reduced mixture for each of the plurality of speakers; combining the corresponding de-reverberated mixtures for each of the plurality of speakers; and feeding the combined de-reverberated mixture for each of the plurality of speakers into the second DNN to generate a second estimate of the target direct path signal for each of the plurality of speakers.

7. estimating a corresponding filter for each of the plurality of speakers, wherein the dereverberated mixture is obtained by removing from the received acoustic signal mixture a corresponding result of applying each of the filters to each of the first estimates of the target direct path signal for each of the plurality of speakers; 5. The method of claim 4, further comprising feeding the dereverberated mixtures of the multiple speakers into the second DNN to generate the second estimates of the target direct path signal for each of the multiple speakers.

8. replacing the first estimate of the target direct path signal with the second estimate of the target direct path signal to generate an updated first estimate of the target direct path signal; repeating the steps of estimating the filter, obtaining the reverberant reduced mixture, and injecting the reverberant reduced mixture on the updated first estimate of the target direct path signal until a termination condition is met; and obtaining an updated second estimate of the target direct path signal.

9. repeating the filter estimating step for each of the plurality of speakers to generate a corresponding filter for each of the plurality of speakers; extracting a portion of the received acoustic signal mixture corresponding to a speaker of the multiple speakers by removing from the received acoustic signal mixture a corresponding estimate of reverberant speech of the other speaker of the multiple speakers, the estimate of reverberant speech of the other speaker of the multiple speakers being obtained by adding the first estimate of the target direct path signal for the other speaker to the result of applying the corresponding filter for the other speaker to the first estimate of the target direct path signal for the other speaker; 5. The method of claim 4, further comprising estimating the filter for estimating the reverberant reduced mixture for each speaker of the plurality of speakers based on the portion of the received acoustic signal mixture.

10. The step of receiving the mixture of acoustic signals comprises: receiving a single channel signal from a single microphone connected to the input interface; and 2. The method of claim 1, comprising at least one of the steps of: receiving a multi-channel signal from an array of microphones connected to the input interface.

11. The step of receiving the multi-channel signal from the array of microphones comprises: obtaining a beamforming output based on statistics calculated from one or a combination of the first estimate of the target direct path signal and the reverberant reduced mixture of the target direct path signal at each microphone of the array of microphones; and feeding the beamforming output into the second DNN to generate the second estimate of the target direct path signal.

12. 2. The method of claim 1 , wherein the filter is applied to the first estimate of the target direct path signal in a time-frequency domain, the distance function is a weighted distance with a weight at each time-frequency point in the time-frequency domain, the weighted distance being determined by one or a combination of the received acoustic signal mixture and the first estimate of the target direct path signal, and the distance function is based on a least squares distance.

13. The method of claim 1 , wherein the first DNN is pre-trained to obtain the first estimate of the target direct path signal from an observed acoustic signal mixture.

14. The pre-training of the first DNN is performed using a training dataset of acoustic signal mixtures and corresponding reference target direct path signals in the training dataset by minimizing a loss function, the loss function being: a distance function defined based on real and imaginary (RI) components of the first estimate of the target direct path signal in a first time-frequency domain and RI components of the corresponding reference target direct path signal in the first time-frequency domain; a distance function defined based on a magnitude obtained from the RI component of the first estimate of the target direct path signal in the first time-frequency domain and a corresponding magnitude of the reference target direct path signal in the first time-frequency domain; a distance function defined based on a reconstructed waveform obtained from the RI component of the first estimate of the target direct path signal in the first time-frequency domain by reconstruction in a time domain and a waveform of the reference target direct path signal; a distance function defined based on the RI component of the first estimate in the time-frequency domain obtained by further transforming the reconstructed waveform in a second time-frequency domain and the RI component of the reference target direct path signal in the second time-frequency domain; a distance function defined based on the magnitude obtained from the RI component of the first estimate of the target direct path signal in the second time-frequency domain obtained by further transforming the reconstructed waveform in the time-frequency domain and the corresponding magnitude of the reference target direct path signal in the second time-frequency domain.

15. 1. A system for dereverberation of a speech signal, comprising: an input interface configured to receive an acoustic signal mixture including a target direct path signal and a reverberation of the target direct path signal; a memory storing a first deep neural network (DNN) and a second DNN for dereverberation of the speech signal; a processor, the processor comprising: feeding the received acoustic signal mixture into the first DNN to generate a first estimate of the target direct path signal; and configured to estimate a filter that models a Room Impulse Response (RIR) of the first estimate of the target direct path signal, the filter producing a result that, when applied to the first estimate of the target direct path signal, is closest to a residual between the acoustic signal mixture and the first estimate of the target direct path signal according to a distance function, the processor further comprising: removing the result of applying the filter to the first estimate of the target direct path signal from the received acoustic signal mixture to obtain a reverberant-reduced mixture of the target direct path signal; configured to feed the dereverberated mixture to a second DNN to generate a second estimate of the target direct path signal; The system further comprises: The system comprises an output interface configured to output the second estimate of the target direct path signal.

16. The system of claim 15 , wherein to estimate the filter, the processor is configured to estimate a linear filter.

17. 16. The system of claim 15, wherein the processor is further configured to feed one or a combination of the received acoustic signal mixture and the first estimate of the target direct path signal into the second DNN to generate the second estimate of the target direct path signal.

18. 16. The system of claim 15, wherein the received acoustic signal mixture includes speech signals from a plurality of speakers, and the first DNN generates a plurality of outputs, each output of the plurality of outputs including the first estimate of the target direct path signal for a speaker from the plurality of speakers.

19. The processor further comprises:

20. The system of claim 18, configured to repeat estimating the filter, obtaining the reverberant reduced mixture, and injecting the reverberant reduced mixture for each of the multiple outputs of the first DNN.

20. The processor further comprises: repeating the estimating of the filter and the obtaining of the reverberant reduced mixture for each of the plurality of speakers to generate a corresponding filter and a corresponding reverberant reduced mixture for each of the plurality of speakers; combining the corresponding dereverberated mixtures for each of the plurality of speakers into a tensor; 20. The system of claim 18, configured to feed the tensors into the second DNN to generate second estimates of the target direct path signal for each of the multiple speakers.

Images